Understanding Enterprise SOA: Book Review

I was handed this book by one of my programmers. He lent it to me because he knew I was doing some reading on the subject and trying to work out how to bring my software developers up to speed with SOA. I didn't get the impression that he had got terribly much from the book himself. After reading it I can see there is benefit for managers but I think there was not enough technical meat in the book for programmers and analysts.

This book is not technical. It can be read by managers, salesmen or students and I think these were the intended audience. The explanations are easy to understand although I found them long-winded and overly simplistic on many occasions.

The authors attempted to explain procedural code as if it was museum artifact. This made me feel very old. It begs the questions whether the developers brought up on object-oriented development and SOA really need a detailed explanation of the concept of procedural code.

On the positive side this books tackles issues that others do not. It has a good non-technical discussion about security issues and there is a lot of discussion about human issues such as training, reorganization and corporate culture.

Oddly for a book written in 2006 I did not find any mention of Enterprise Service Bus (ESB) although it was clear the author was encouraging looking at off-the-shelf SOA management solutions as well as SOA security and routing software.

Eric Pulier in his preface is unbridled in his enthusiasm for SOA standards:

In a spectacular cycle of enlightened self-interest, every major company in the world has build on top of standards which have ushered a potentially exponential release of value from investments in information technology.

Elsewhere in the book there are sufficient cautions to temper this enthusiasm. In particular the authors advise that much of SOA is still hard work and that there are no industry standard solutions to problems such as transaction processing and security.

There is a long and detailed case study running through the book. The authors do not hold back on the multitude of issues that confront an organization adoption SOA. Parts are realistic and even painful for those of us who have been through implementations of a difficult project or technology.

The book has a good history of technologies and standards. The authors make a good point about standards conforming to the Tipping Point model (see references below). This is when a standard reaches a certain level adoption that it suddenly becomes desirable for the majority of developers.

This is not a strong academic text. There are not many references to other information sources. The definitions are not clearly stated and often over-simplify the concepts. Concepts like Enterprise Architecture, Real-Time Computing and loose coupling are all not accurately defined.

"Loose-Coupling" is described but not adequately explained. The authors give the impression that loose-coupling is only about where you can locate services.

Web services, in contrast are loosely coupled. Once a piece of software has been exposed as a web service it is simple to move it to another computer.

This is an important aspect of coupling but is not the whole story as I have identified on previous postings.

The book focuses on synchronous services rather than asynchronous services. This is unfortunate as other texts such as Thomas Erl or Krafzig, Banke & Slama seem to advocate asynchronous services.

The book addresses the Subject of Enterprise Architecture again loosely defined. At one point the book states

There is no institute providing credentials for enterprise architects.

I'll give the authors the benefit of doubt. Perhaps in 2006 there were not the credentials around but these days there are a number of places to get recognized credentials as an IT Architect. The Microsoft Certified Architect programs spring to mind although I will not vouch for the content of these programs.

The authors see SOA as a method for replacing parts of the Data Warehouse. I was not particularly convinced by this. This was an interesting subject which I had not seen before. I think there is benefit in moving the content of SOA logging to the data warehouse but I was not convinced that SOA radically changes the typical data warehouse ETL process.

This is not a book with detail examples of SOAP and WSDL. It will not explain how to code or build services. You will not find much detail on WS-* here either.

Contracts and Service Level Agreements (SLAs) are described in the book. The authors also introduce the Business Level Agreement (BLA). This is agreement to monitor business conditions such as an inventory pile-up causing $100k per minute of cost.

The book is well illustrated with copious diagrams and notes attached to the diagrams. I found myself skipping past a lot of the diagrams because they did not seem to be explaining anything particularly special.

I congratulate the authors for including detail about setting up training programs in the book and explaining how the resources are brought in to deal with the adoption of SOA. This is missing from many other texts and is an important issue today.

The authors emphasize that SOA is an enterprise approach. Under SOA, IT departments put in place infrastructure to serve the enterprise rather than just serving particular applications.

I would recommend other books before this one for most audiences. The simplistic and at times pedantic prose would make it hard to recommend to an IT professional. Nevertheless there are some well-made observations in this book that you are not likely to find elsewhere.

References

Understanding Enterprise SOA, Eric Pulier and Hugh Taylor with forward by Paul Gaffney, Manning, 2006

The Tipping Point: How Little Things Can Make a Big Difference (ISBN 0-316-31696-2) is a book by Malcolm Gladwell, first published by Little Brown in 2000.

Laws of Software Development

I ran across an new eponymous law while reading "Understanding Enterprise SOA" by Eric Pulier and Hugh Taylor.
Pulier's rule of reuse:

A programmer will look at another programmer's output, no matter how brilliant and declare it garbage

The best law's and rules are named by others who admire the author's work, but this one seemed pretty worthy to me. This got me reminiscing about some laws of software development that perhaps had the same rigour as Newton's Laws, Boyle's Law or Hooke's Law.

The rules I have admired during my career are:

Brooks's Law:

adding manpower to a late software project makes it later

Hofstadter's Law:

It always takes longer than you expect, even when you take Hofstadter's Law into account.

Codd's Rules of relational database management systems which I won't quote here because there are 13 of them.

Moore's Law:

The number of transistors that can be inexpensively placed on an integrated circuit is increasing exponentially, doubling approximately every two years.

This has a number of applications outside of transistors such as processing speed, memory and disk capacity.

Clarke's 3rd Law:

Any sufficiently advanced technology is indistinguishable from magic.

The reason I class this as a software development Law is that it points out the futility of trying to explain the technical details of software to Business users. If a technical decision does not have any business impact then it may as well be magic.

I thought I would check what other software development Laws there are on the net. Here is a good sample from some useful sources listed in the Bibliography below:

Amdahl’s Law

The speedup gained from running a program on a parallel computer is greatly limited by the fraction of that program that can’t be parallelized.

Asimov's laws (Yes its SF but one day a software developer might have to follow them):

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Atwood’s Law:

Any application that can be written in JavaScript, will eventually be written in JavaScript

Bye's First Law of Model Railroading

Anytime you wish to demonstrate something, the number of faults is proportional to the number of viewers.

Clarke's 1st Law:

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

Clarke's 2nd Law:

The only way of discovering the limits of the possible is to venture a little way past them into the impossible.

Clarke's 4th Law (He's not often credited with this one):

For every expert there is an equal and opposite expert.

Conway's Law:

Any piece of software reflects the organizational structure that produced it.

Edwards' law

You cannot apply a technological solution to a sociological problem.

Greenspun's Tenth Rule of Programming (There actually are no rules 1 to 9)

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp

This can also be worded “Those who do not understand Lisp are doomed to reinvent it.”

Gustafson's Law (also known as Gustafson-Barsis' law)

Any sufficiently large problem can be efficiently parallelized

Kerckhoffs' law on secure cryptography

A cryptosystem should be secure even if everything about the system, except the key, is public knowledge

Lehmann's Laws

1. Systems that are used must change or automatically become less useful
2. Through changes the structure of a system becomes ever more complex and more resources are needed to simplify it

Linus's law — named for Linus Torvalds,

given enough eyeballs, all bugs are shallow

Lubarsky's law of Cybernetic Entomology:

There is always one more bug.

McLuhan's Law:

If it works it's obsolete.

Metcalfe's law

In communications and network theory, states that the value of a system grows as approximately the square of the number of users of the system.

Murphy's Law

If anything can go wrong, it will.

Occam's Razor:
There are many wordings for Occam's razor and debate about how it is interpreted but from software development I think the best wording is "The simplest solution is usually the best". This is very nearly the same as the KISS principal "Keep it simple stupid".

Putt’s Law:

Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand.

Sturgeon's Revelation (sometimes referred to as his second law):

Ninety percent of everything is crap

Weinberg’s Law:

If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

Wirth's law

Software gets slower faster than hardware gets faster.

Zawinski's law

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

Here are still more great laws from Joey DeVilla's Blog posting.

The Law	Who Said It	What it Says
Ellison’s Law of Cryptography and Usability	Carl Ellison	The user base for strong cryptography declines by half with every additional keystroke or mouse click required to make it work.
Ellison’s Law of Data	Larry Ellison	Once the business data have been centralized and integrated, the value of the database is greater than the sum of the preexisting parts.
Flon’s Axiom	Lawrence Flon	There does not now, nor will there ever, exist a programming language in which it is the least bit hard to write bad programs.
Gilder’s Law	George Gilder	Bandwidth grows at least three times faster than computer power.
Grosch’s Law	Herb Grosch	The cost of computing systems increases as the square root of the computational power of the systems.
Hartree’s Law	Douglas Hartree	Whatever the state of a project, the time a project-leader will estimate for completion is constant.
Heisenbug Uncertainty Principle (restated)	Jim Gray	Most production software bugs are soft: they go away when you look at them.
Hoare’s Law of Large Programs	C. A. R. Hoare	Inside every large problem is a small problem struggling to get out.
Jakob’s Law of the Internet User Experience	Jakob Nielsen	Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.
Joy’s Law	Bill Joy	smart(employees) = log(employees), or “No matter who you are, most of the smartest people work for someone else.”
Lister’s Law	Timothy Lister	People under time pressure don’t think faster.
Nathan’s First Law	Nathan Myhrvold	Software is a gas; it expands to fill its container.
Ninety-ninety Law	Tom Cargill	The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time.
Pesticide Paradox	Bruce Beizer	Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.
Reed’s Law	David P. Reed	The utility of large networks, particularly social networks, scales exponentially with the size of the network.
Sixty-sixty Rule	Robert Glass	Sixty percent of software’s dollar is spent on maintenance, and sixty percent of that maintenance is enhancement.
Spector’s Law	Lincoln Spector	The time it takes your favorite application to complete a given task doubles with each new revision.
Spafford’s Adoption Rule	George Spafford	For just about any technology, be it an operating system, application or network, when a sufficient level of adoption is reached, that technology then becomes a threat vector.

Some unattributed "laws" that are worth mentioning:

• Build a system that even a fool can use, and only a fool would want to use it.
• Any program over 100 instructions can be simplified by 3 instructions (without losing any functionality).
• Any idiot can learn to use computers, and many do
• There’s never time to do it right in the first place, but there’s always time to do it over when it doesn’t work.

This posting started with Pulier's Law: "A programmer will look at another programmer's output, no matter how brilliant and declare it garbage". At the risk of repeating Pulier's conceit of naming law about himself I propose:
Kimber's Corollary:

No amount of documentation is ever sufficient to completely understand a system

Bibliography
http://en.wikipedia.org/wiki/Adages_named_after_people

"Moore's Law is Dead", http://www.techworld.com/opsys/news/index.cfm?newsid=3477

http://en.wikipedia.org/wiki/Change_management_(engineering), This includes the reference to Lehmann's Laws siting Hinley, D.S. (1996). Software evolution management: a process-oriented perspective. Information and Software Technology, 38, 723-730

http://en.wikipedia.org/wiki/Moore

http://en.wikipedia.org/wiki/Occam

http://www.webopedia.com/TERM/C/Codds_Rules.html

Joey de Villa, http://globalnerdy.com/2007/07/18/laws-of-software-development/

Phil Haack, 19 Eponymous Laws Of Software Development, http://haacked.com/archive/2007/07/17/the-eponymous-laws-of-software-development.aspx

Eric Pulier and Hugh Taylor, "Understanding Enterprise SOA", Manning Publications, 2006

The Importance of Meta-Projects

The feature video on International Association of Software Architects (IASA) site last week provided. I wish I could reference the speaker and the name of the video but alas last weeks feature video has been replaced by this weeks feature video.

The talk was reasonably pragmatic discussion of Architecture by a staff member of delta airlines. What inspired me to write this posting was that a slide that depicted "Business Enablements" things, like agility and collaboration. It also depicted lists for Business strategy and IT strategy. This is nothing new but it reminded me I need to lift my eyes from the day to day IT projects and see what my organisation is doing for software development as a whole.

This is timely because its time for my branch to put together a plan for next year. Typically this has been very project-focussed. Of course I also hope to put together a good graphic although I doubt whether I can do as well as the delta-airlines chart.

Each business and each IT department has its projects to achieve some outcome for the business. My focus is on application development projects. Often these projects are part of a bigger business project and commonly involve business process changes and infrastructure changes. Another category of projects do not achieve direct outcomes for the business but allow the business or the IT department to undertake these projects in a more effective manner. For business projects these 'meta-projects' is what was referred to as "business enablements" by the IASA speaker. I will not be using the term further as my spell checker says "enablements" is not a real word, and I tend to agree.

To put these in context see the following table

	Business	IT
Goals/Objectives	Business Goals. Mission, Vision and Strategy	IT Strategy
Meta-Projects	Business meta-projects	IT meta-projects
Projects	Business projects	IT projects

There are strong relationships between each cell in the table and its neighbouring cells. The business goals drive the IT strategy and also the business meta-projects. Business meta-projects may drive business projects but also meta-projects may be driven by projects. Neither the meta-project or project is necessarily the driver. This is the same with IT meta-projects and IT projects. IT strategy should drive the IT meta-projects and IT projects.

The IT Meta-Projects that my IT department is engaged in are:

• SOA
• Enterprise Architecture
• Mobility (Supporting mobile computing devices)
• Business Intelligence (Data Warehouse and Reporting)
• Legacy Transition
• Spatial Systems (Geographic Information Systems)
• Online Project (Customer-facing IT systems)
• Identity Management (Authentication/Privileges/Access Control)
• Service Management (ITIL, Request Management, Change Management)
• Performance Management (Individual Performance)
• Cost Efficiency/Consolidation (ongoing)
• Business Alignment (ongoing)

Some of these meta-projects have explicit planning documents explaining the plan. Others are more just acknowledged as the agreed direction of the IT department. Cost efficiency and business alignment are not specific projects but an ongoing focus of attention as they would be in most IT departments.

Some of these IT meta-projects such as Online Project, Service Management, Performance Management and Costs Efficiency have analogous business meta-projects. Business Alignment parallels the business meta-project of customer alignment.

This list of IT meta-projects has not previously been written out. It is surprisingly long, especially given the much longer list of specific software development projects that are on our books. Ideally each of these meta-projects should be explicitly planned, prioritised and monitored.

Meta-projects provide a foundation to the work an IT department does. They could also be referred to as 'foundation projects' but they are not necessarily just internal projects as they may be programs of projects that directly affect the business user. The meta-projects help build the look and the feel of the IT department. They establish its competitive advantage and its identity. Accordingly, it is important to get these meta-projects right and to provide them with the attention they deserve over the more day to day concerns of getting software out the door.

Don't get the staff you deserve, get the staff you need

I am working on a resourcing plan for my branch. The move from legacy mainframe applications to SOA requires new skills and specializations, but we need to maintain some fairly rare legacy skills as well. Here I explore a couple of rational models and tear them down again in order to justify my own approach which I plan to provide in a later posting.

If you are managing a group responsible for software development there are a number of options to accessing personnel to get your software developed. These include:

• Permanent Employment (A salaried employee with can be Full-time and Part-time)
• Term Appointment (Contract to the organisation for a fixed term)
• Personnel Contract (Usually contracted through an agent rather than directly to the organisation)
• Service Level Contract (This is for a particular service such as "keep the database server running within service levels" rather than a chunk of work)
• Work Contract (Contract to deliver a piece of work. This contract might be on a 'fixed price' or 'time and materials' basis)
• Product Purchase (A vendor produces a product, often to specification which is then purchased)

This list is ordered from most tightly bound to the least tightly bound to the organisation. The expectation is that the salaried employee will stay for a number of years, stay after the current project is complete and usually paid for by recurrent funds. A contractor on the other hand is not tightly bond to the organisation. The contractor is expected to be employed for the length of the project and funded capital allocations.

Of course these are expectations. In practice we may have employees who leave after a few weeks and we may have contractors who are essential to the organisation and we keep extending as new projects become available. What is important is that we have the roles in our organisation that will allow it to continue for the long term but we also have sufficient staff to get our current projects done.

How do we decide which are the permanent roles, tightly bound to the organisation and which are the contract roles?

First I will look at some rational ways of doing human resource planning. Imagine we are able to predict our resources requirements for, say the next 10 years. That is stretching the imagination a long way but let us do it for the sake of argument. We would come up with a nice chart with peaks and troughs in our demand for people. There are a number of models we could use for our human resources:

• Staff to meet peak demand. This is more a model for a call centre than a software development shop.
• Use permanent staff to meet the trough of demand (ie steady-state demand) and contractors to meet the peaks.
• Minimize the cost of staffing based on the human resource demand (see footnote).

To minimize the cost of staff we must assume contractors cost more than permanent employees. The contractor should have compensation for not having the benefits of permanent employment. To minimize the cost we must fund a permanent staff level somewhere above the demand trough level. The implication is that it is economical to pay permanent staff to be idle in times of low demand rather than pay a premium to bring in a contractor in more busy times.

This is of course, never the way it works in practice. We can't predict our IT projects very far into the future. The concept of an idle IT professional is foreign to me. There is always maintenance work to do, more enterprise documentation or more skill development to be done. We do not count human resources as simple numbers. There are skills and specialties we need. The costs of our staff and contractors are not uniform and the formula for getting value for what you pay for can be very muddied when you are talking about people and what they can deliver to the organisation.

The big killer for getting in the way of a rational approach is that I have very little discretion over the number of my staff anyway. Governments and many large organisations have fixed FTE limits. Especially for Government it tends to a be vote-winner to promise reduction in the number of public servants and the folks sitting in the back office, like IT staff, not delivering anything directly to the public are often on the agenda for cuts and freezes. The outcome is that we may end up with less staff and more contractors than is rationally based on models like the ones above.

I have to ensure that I have the right permanent roles moving forward. By 'roles' I mean positions. Max Weber has the dubious distinction of inventing the bureaucracy where every person is employed to fill a position. The position has certain duties and requirements. It is these duties and requirements that we described in position document and we try to fit people into these roles. As much as I would just like to get a bunch of talented people together in the expectation they could come up with something wonderful, each position has to be described in detail so that we have all bases covered and this is what has to drive my planning.

Footnote:
I did maths a long time ago but occasionally I like using the formula editor. The total cost of staff for a period of time T is

We minimise TotalCost with respect to the permanent staff level P.

The Passion of Programming

Sometimes I think back to high school and University where programming was a passion. In my kit full of mediocre skills programming stood out as something I was good at and something I had a passion for. It's interesting to reflect on this today when I am asking programmers to be professional, to document their work properly and to follow the standards. There is passion amongst some of them. Sometimes this is detrimental and results in religious wars on technical ideologies but by and large there is not enough passion about the work. It's not like any other job. As a programmer you are building something unique every day.

I used to liken programming to building castles out of air. You create something useful from nothing except logic. It is not quite like that any more. Perhaps it never was. After all you usually needs a big computer, a powerful compiler, some noisy printers and dare I recall, stacks of punched cards and a card reader. This things faded into the environment and what really matter was logic. You seem to need much more to get started these days. Application servers, database servers, browsers, virtual machines and libraries of standard routines, classes and services. A good SOA approach is one that assembles and application from a catalogue of pre-fabricated parts. We do not building castle from air any more we are putting together with prefabricated walls, floors and ceilings, and then furnishing it using IKEA.

This is of course a good thing. We want to be able reuse what we develop and do not want to reinvent the wheel every time we build something new. The art has changed but the skill is still required, and so I hope the passion remains for many.

Like the components the human resource have also changed. There seems to be more specialties than before. I used to work for small organisations that did not really distinguish between programmers, designer and analysts. Now we can't get by without a DBA, a configuration manager, web designers, an application server specialist and a host of architects, test professionals and user support personnel.

Footnote:

Alas! "Building Castles out of Air" was not my original work. It has been bouncing around in my brain for many years first seeded there by a classic in the IT literature. After writing this blog I found the following quote from Fred Brooks' "The Mythical Man-Month". He writes it better than I.

"The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination."

http://www.wordyard.com/2006/10/02/mythical-man-month/

Code Monkeys

In preparing for this blog I looked the the infinite monkey theorem on wikepedia (http://en.wikipedia.org/wiki/Infinite_monkey_theorem). It’s the theorem that goes "a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will type one of Shakespeare's great works. Apparently this was subject to a real experiment. The result was they discovered the monkey's continually typed the letter "S", defecated on the typewriters and did not produce any text of which the bard would have approved.

The relevance of the infinite monkey theorem to blogging is of course that if you keep on writing long enough at some point you might come up with something that is worth reading. This is my hope. Of course like the monkeys (and some other writers I know) I could end up getting very repetitive and using the keyboard for purposes for which it was not designed.

The term "Code Monkeys" is pretty objectionable. Writing code is difficult and exacting. If Shakespeare had to conform his works to the requirements of a compiler I suspect he would have had difficulties. Writing code no longer seems to be about sticking to the one language and submitting it to one big box that follows the instrutions. Now an application may spread over several layers using different languages and running on different boxes. This actually could be progress if it done properly. It should allow more agile, more integrated systems with more flexibility. Sometimes it is hard to convinced.

(The image of the monkey above is borrowed from a Zapthink presentation by Jason Bloomberg)