Evolution: The Fittest and the Sexiest

The topic of this blog is "SOA Evolution" so it seems overdue for me to look at the Evolution process. I coined this title to chronicle my organisation's slow but deliberate adoption of SOA. 'Evolution' seemed to be an appropriate term because it was both a slow process but one that would improve our applications. In this posting I will look at evolution more literally and find more appropriateness in the term than I first imagined.

I am reading a book by Richard Dawkins (see references below) on a range of topics but primary about evolution in biology. There are two principal evolutionary processes. One is the survival of the fittest and the other is sexual selection which is the process of influencing the evolution process by the choice of a mate. The success of evolution is its ability to adapt to its environment and to succeed against other species in that environment.

Natural adaptations are gradual and based on the parent life form. The history of the life forms can be discerned in the DNA of the life form. We can determine that humans evolved from an ancestor we share with Chimpanzees from looking at this DNA.

The process of survival of the fittest can be applied to software development technologies. We see various versions of programming languages replaced by later more effective versions. We see whole languages disappear into obscurity (eg. Algol, APL, SmallTalk, ADA) or replaced by new languages. Relational databases took over from set-based or network databases. Software development approaches have evolved from structured programming, to object-oriented programming to SOA. It has been mentioned that SOA was forged from marriage of distributed computing and document-based integration.

Extinction of old approaches is often not as complete as we would like. The Cobol programs and the old database technologies are still around. The old dinosaurs are hard to kill off completely.

As with any evolutionary process the origins of software development are evident when we dig deep. Visual Basic .NET is still saddled with some of the terminology and constructs that were around in DOS Basic. The kernel of some operating systems still contains some anachronisms which would not be used if it were not for historical reasons. Many software systems have parts that were written long ago or copied from other systems that no-one can maintain or just do not make sense in the context of software written today. Some software maintenance tasks are like archaeological expeditions. Unfortunately it is not passive fossils that the software archaeologist finds amongst the code. Some of these historical curiosities actually support critical business functions.

There are some principles from long ago that we do not want to discard though. Just like some of the primitive instincts and emotions buried in the core of our brain there are principles from long ago that should remain in our software. Procedures and functions still have a role in our software as do objects. The development of services does not replace the need for these other means of encapsulation. The principles for strong typing and structured programming still have as much value today as they did thirty years ago.

The mechanism for natural selection process in software technologies is not that different from the animal kingdom. Programmers will rewrite software using new techniques if requirements change. New tools outsell older tools that do not provide the same benefits. New techniques that reduce development time or improve quality replace old techniques.

Just as species become extinct old software technique die from disuse or replacement. Old code does not die off as quickly as some of us would like. It can last forever in our systems if the requirements it fulfils and infrastructure it uses do not shift too markedly.

Charles Darwin had two theories that he espoused in his books "The Origin of the Species" and "The Descent of Man". One could be paraphrased as 'survival of the fittest' and the other once was 'sexual selection'. Sexual selection is where a mate is selected because he or she is bigger, is stronger, is darker, has more colourful tail feathers or has less hair and therefore individuals with these characteristics tend to propagate.

I have briefly explained how the fittest software and development techniques succeed at the cost of inferior versions but it is also true that the sexiest software can be propagated despite any objective superiority. This can happen through good marketing or the support of a major company. Software standards are often agreed upon, even though they may not be the best based on an objective measure of quality. Often the benefit to adopt a standard exceeds the cost of the standard being suboptimal.

'Survival of the fittest' and 'sexual selection' are alive and well in the software industry. I expect SOA to remain an important part for the genetic makeup of future software development even after the term has lost its currency.

References

Dawkins, Richard, A Devil's Chaplain: Selected Essays, Phoenix 2003

Darwin, Charles, The Descent of Man, 1871, http://www.infidels.org/library/historical/charles_darwin/descent_of_man/

Darwin, Charles, The Origin of the Species, 1859, http://www.infidels.org/library/historical/charles_darwin/origin_of_species/

SOA in a Virtual World

Wikipedia defines a virtual world as a computer-based simulated environment intended for its users to inhabit it. Often a computer application is a business process made virtual by the use of computer technology. We therefore have to be careful when discussing the virtualization of computer environments as this may imply a virtual virtual reality.

The only example of virtual virtual reality I have seen is in a Futurama episode. It displayed an arcade booth with the label 'Virtual Reality' complete with a player using stereoscopic goggles and wired gloves. It then displayed another arcade booth with man sitting in a simple chair imaging he was wearing stereoscopic goggles and wired gloves. The caption to this booth was "Virtual Virtual Reality".

If we talk about "Virtual SOA" we must be talking about something that is not an SOA at all. Judith Hurwitz comes close to this when she discusses a "Virtualized SOA" in her article "Is virtualization the foundation of SOA?"

Andrew Doble writes about building a virtual SOA without investing in in-house software like an ESB. I would argue that this is in fact legitimate SOA. The only thing "virtual" about Andrew's suggestion is the ownership of the software tools.

Virtualization and SOA however do go together. Judith Hurwitz defines Virtualization as

"a technique for abstracting the physical characteristics of computing resources
in order to more easily leverage hardware systems, applications, operating
systems, networks, graphics, data or storage so they can be repurposed based on
customer need."

I believe that loose-coupling is the foundation of SOA but virtualization in many cases allows the benefits from this loose-coupling to be realised. The basic examples of virtualization are virtual servers, storage area networks and thin client platforms such as Citrix. These provide infrastructure flexibility for the deployment of SOA software. The design of the software for an application written using SOA is not affected. The SOA discipline should make sure that flexible deployment options can be used.

Manufacturers such as IBM are building business around SOA infrastructure. One thing to remember is that you do not need virtualization to build SOA. An SOA application can be built on a single server if you required, although it should also give you the flexibility to distribute this application and take advantage of virtualization options.

I agree with Lorraine Lawson in her posting "SOA and Virtualization: Complex, Fuzzy Ideas That Go Great Together? ". These two concepts mix well. However SOA is from the software development world and virtualisation is from the infrastructure world. We should beware of talking in terms of virtualizing our SOA. Our CIOs may consider we are sitting in a corner thinking we have an SOA when this is just a virtual reality.

Fry from "Futurama" with a virtual Lucy Liu.

© 2001 Twentieth Century Fox. All Rights Reserved

The Importance of Meta-Projects

The feature video on International Association of Software Architects (IASA) site last week provided. I wish I could reference the speaker and the name of the video but alas last weeks feature video has been replaced by this weeks feature video.

The talk was reasonably pragmatic discussion of Architecture by a staff member of delta airlines. What inspired me to write this posting was that a slide that depicted "Business Enablements" things, like agility and collaboration. It also depicted lists for Business strategy and IT strategy. This is nothing new but it reminded me I need to lift my eyes from the day to day IT projects and see what my organisation is doing for software development as a whole.

This is timely because its time for my branch to put together a plan for next year. Typically this has been very project-focussed. Of course I also hope to put together a good graphic although I doubt whether I can do as well as the delta-airlines chart.

Each business and each IT department has its projects to achieve some outcome for the business. My focus is on application development projects. Often these projects are part of a bigger business project and commonly involve business process changes and infrastructure changes. Another category of projects do not achieve direct outcomes for the business but allow the business or the IT department to undertake these projects in a more effective manner. For business projects these 'meta-projects' is what was referred to as "business enablements" by the IASA speaker. I will not be using the term further as my spell checker says "enablements" is not a real word, and I tend to agree.

To put these in context see the following table

	Business	IT
Goals/Objectives	Business Goals. Mission, Vision and Strategy	IT Strategy
Meta-Projects	Business meta-projects	IT meta-projects
Projects	Business projects	IT projects

There are strong relationships between each cell in the table and its neighbouring cells. The business goals drive the IT strategy and also the business meta-projects. Business meta-projects may drive business projects but also meta-projects may be driven by projects. Neither the meta-project or project is necessarily the driver. This is the same with IT meta-projects and IT projects. IT strategy should drive the IT meta-projects and IT projects.

The IT Meta-Projects that my IT department is engaged in are:

• SOA
• Enterprise Architecture
• Mobility (Supporting mobile computing devices)
• Business Intelligence (Data Warehouse and Reporting)
• Legacy Transition
• Spatial Systems (Geographic Information Systems)
• Online Project (Customer-facing IT systems)
• Identity Management (Authentication/Privileges/Access Control)
• Service Management (ITIL, Request Management, Change Management)
• Performance Management (Individual Performance)
• Cost Efficiency/Consolidation (ongoing)
• Business Alignment (ongoing)

Some of these meta-projects have explicit planning documents explaining the plan. Others are more just acknowledged as the agreed direction of the IT department. Cost efficiency and business alignment are not specific projects but an ongoing focus of attention as they would be in most IT departments.

Some of these IT meta-projects such as Online Project, Service Management, Performance Management and Costs Efficiency have analogous business meta-projects. Business Alignment parallels the business meta-project of customer alignment.

This list of IT meta-projects has not previously been written out. It is surprisingly long, especially given the much longer list of specific software development projects that are on our books. Ideally each of these meta-projects should be explicitly planned, prioritised and monitored.

Meta-projects provide a foundation to the work an IT department does. They could also be referred to as 'foundation projects' but they are not necessarily just internal projects as they may be programs of projects that directly affect the business user. The meta-projects help build the look and the feel of the IT department. They establish its competitive advantage and its identity. Accordingly, it is important to get these meta-projects right and to provide them with the attention they deserve over the more day to day concerns of getting software out the door.

Architecture on Valentine's Day

I attended the Australian Computer Society (South Australian Branch) Architecture SIG this week. It happened that this was on Valentines Day. This drew a few comments about our dedication to the craft of IT Architecture, as we chose to spend the evening talking about architecture rather than with our chosen partners.

There were two presentations. Paul Turner introduced the International Association of Software Architects (IASA) to us. They have just set up chapter in Adelaide and will associate with our Architecture SIG.

IASA is an international professional society for Architects. It provides standards, training, certification and advocacy for Software Architects. Paul discussed some fundamental questions such as:

• What is an architect?
• What is software architecture?
• How can yo prove you are a software architect?

Paul explained how IASA could help answer some of these questions. The main item of interest for me is that one can now be certified as an Architect. You pay for training over a number of months and then get grilled by a board which determines whether you have made the grade. This is an interesting development and will provide some legitimacy to the role and credentials of a software architect.

Adam Davies then gave a demonstration of Ruby on Rails. I had previously heard much about this technology but had not seen it in action. It was a very competent demonstration and it was amazing how much Adam was able to put together in the course the demonstration without resorting to the "here is some code I prepared earlier" fallback. Rails is able to set up sensible defaults to field names, column names and object names and create mappings between these without much overhead. It seemed to be ideal for standalone applications but certainly had some features that would make it valuable in any software development environment.

Coupling Calculus

I have had thread going through this blog on Coupling. Coupling is important because good definitions of SOA are dependent on the concept of 'loose coupling' or more precisely, 'message coupling'. The thread so far has included the following postings:

Cohesion, Coupling and SOA
Bragging Rights on Coupling
AOP: The Worst Form of Coupling
Binding and Coupling
Subroutine Call Coupling
Seven Levels of Coupling or Seven Deadly Sins?
Coupling with Contracts

In the first posting of this series I went back to the origins of the coupling term that was coined by Edward Yourdan to explain good programming practice in terms of subroutines. I then tried to come up with a list of coupling types which would help explain what were good and bad form of coupling.

This served to clarify the concept in my mind but the list of coupling types is not exhaustive and term 'coupling' is still not clearly defined. Coupling is a measure, usually described as 'loose' and 'tight' which correspond to the value judgement of 'bad' and 'good' respectively.

A good measure has numbers that describe it, but this is not yet the case with coupling. I and other authors have attempted to rank coupling types in order of merit with the ranking an implicit measure of coupling.

There have been attempts to measure coupling by looking at program artefacts like variable names and measuring how many times they are referenced. These attempts seem to miss the point by measuring what can be measured rather measuring coupling properly. Some of these approaches are described in footnote 1.

In this posting I try to suggest some definitions relating to coupling. This will not come up with useful numbers but it will try to explore the concept of coupling and may be a basis on which more theory can be developed.

Coupling is about how easy it is to change applications. If one part of an application (module) has to be changed because another part of a program changes then these program parts a tightly coupled. If one module does not have to be changed for changes made to another module then these application modules are loosely coupled. There are a number of programming practices that can be put in place to loosen the coupling between modules. This is good because it reduces the effort required to maintain your code.

The above would seem to be a fairly elementary description. Definitions however can still be made more precise, so that the concept can be analysed in more detail.

Lets say we have two modules A and B in a system S. The number of changes that can be made to A can be defined as C(A).

We can make changes to A because we want system S to do something different, or we can make changes to A for other reasons. We might want S to be produce results more quickly or we just might want A or S to be reorganised to be more easily comprehended. A change to A might not mean any change the output of S.

In order for A to be changed and the result of S to be changed to a desirable result (which may be an unchanged outcome for S) then in some cases some module B would have to change as well.

The coupling between A and B is defined as the effort required to make changes to B as a result of these changes A. For change j then there is work (and cost) associated with this change for systems S say Wj(S). The work in module Wj(A) and module be Wj(B).

Coupling then can be defined in terms of an individual change
Coupling(j) =the work required in B to make change j as a result of the change in Aj.
Coupling(A,B) is the sum of all work require in B for such changes j.

This leaves the concept of 'Coupling Type' to be explained. A system can be organised to reduce coupling between A and B. A coupling type is a way of organising the system. Some ways of organising will mean that a change to A require changes to B. Some ways of organising will mean that B can remain independent of changes in A. Some ways may increase or decrease the work involved in making changes to B when A is changed.

In general a way of organising a system S is O(S) which is another system which produces the same results. This is a change to the structure of the system rather than a change to the outcomes of the system. If we wanted to measure the coupling of a way of organising O then 'all' we have to do is sum all the Coupling(A,B) over all such S, A and B. A coupling type describes a number of ways of organising a system. The coupling associated with a coupling type is therefore the sum of all the Coupling (A,B) over all these ways of organising S.

To express this succinctly:

Coupling between two modules (A and B) of a system (S) is the work required to make changes to a module (B) as result of changes to another module (A) and still produce the desired outcome of a system (S).

A coupling type is scheme for organising the system (S). A measure of the coupling of a coupling type is the sum over all A and B for systems S which conform to this coupling type.

These technical definitions of coupling describe big, practically incomputable numbers. The conclusions we can draw though are

• There is nothing really special about a coupling type. There could be a very large number of these and they are not confined to simple statements about how to pass parameters between modules. My earlier posting attempting to count these is inherently futile expect to provide common examples.
• Coupling is about work relating to a change. I have not defined what I mean by 'work' intentionally. We could count lines of code changed or look at whether logic or data is changing. We count the costs of making these changes (eg. Hours of labour). It is meant to be used in its common sense. If it is necessary to trawl through technical documentation to make a change to one line of code this is still consider then the trawling effort should be consider as part of the work expended for this change.
• Coupling is about systems in general. It is not just code instructions that are relevant. You might have to change constants, configurations, contracts or XML. These need to be considered in the concept of coupling.
• The concept of coupling is relevant to systems theory examples outside of the field of computer science. Systems can also refer to biological or sociological systems (See footnote 2).
• A change may not actually change the behaviour of the system at all but it is still relevant for coupling.
• Coupling at its most elementary level looks at two particular modules, a particular system and a particular change. Thus it sensible to say A is tightly coupled to B in a system S in relation to change j.
• Coupling can aggregated up to the module or level or to the Coupling Type level. Thus we can have A loosely coupled to B. We can talk about a Way of Organisation or a Coupling Type as facilitating loose coupling.
• Restriction have not been placed on A and B. Therefore if B=S we can have A tightly coupled with the rest of the system S. A and B could intersect or A could be a sub-module of B.

It is still important to describe different coupling types and to determine what types promote loose coupling and what types do not. Good practice will reduce coupling for common programming changes.

To determine whether one coupling type is better than another the above suggests we need to enumerate all possible changes and work out which type leaves you with less work involved in one module as result of changes in another. This is obviously impractical but the analyst can do a couple of things to determine a good coupling type. A range of typical changes can be investigation or the coupling type can be looked at in relation to a specific type of change. For instance the benefits of message queuing can be looked at in relation to changing the location or speed of hardware.

This posting has added some precision to the concept of coupling. Counting all the possible ways to change a system is futile, however good and bad coupling will be become evident by looking at coding examples and foreseeing the changes that will be required to any system.


Footnote 1:

The following web pages describe metrics for coupling. These can be based on a static analysis of code or a dynamic tracing of the execution paths. In both cases the emphasis is counting the reference to a variable or type. This has some value in the task of maintaining code but is not a measure of the work required to change one module based on the need to change another. In some cases there may be some correspondence between the amount of work in changing the use of the variable with the number of times it is reference but not generally.

Lethbridge, Timothy C. and Anquetil, Nicolas, Experiments with Coupling and Cohesion Metrics in a Large System, April 2000, https://www.site.uottawa.ca/~tcl/papers/metrics/ExpWithCouplingCohesion.html

Poshyvanyk, Denys and Marcus, Andrian, The Conceptual Coupling Metrics for Object-Oriented Systems, http://www.cs.wayne.edu/~vip/publications/poshyvanyk-ConceptualCoupling.pdf

Glover, Andrew, In pursuit of code quality: Code quality for software architects: Use coupling metrics to support your system architecture, April 2006, http://www.ibm.com/developerworks/java/library/j-cq04256/
FrontEndArt describes one of its products for montoring source code quality include various metrics for coupling, http://www.frontendart.com/monitor/help/node22.html
Diederik Krols provides a blog-site archive which includes the postion "OO-Metrics for Coupling". This provides a coupling metric based on types shared across a module boundary. http://dotbay.blogspot.com/2006_01_01_archive.html

Footnote 2:

The concept of "structural coupling" in systems theory is used to describe co-evolution in biological and social systems. This is where, if entity A evolves then entity B might need to evolve to compensate. There would seem to be more complicated processes at play here than in a computer application.

Kay, Robert and Goldspink, Chris, Towards a Complex Non-linear Systems Theory of Organisation, http://ftp.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-72/044%20Kay%20Complex.pdf

More on ESBs

One of my for more popular postings has been "ESB or not ESB". I suspect the only reason was the word "ESB" appeared twice in the title and the Google search engine gave it a high priority when people searched on this acronym.

This was cited in an Italian language blog posting by Andrea Gumina which also cited a good posting by Steve Vinoski . Steve's posting is perceptive and has many good comments on it.

Steve is commenting on an earlier posting by Patrick Logan. Patrick describes the concept of an ESB as a "mangy mutt" because it is a " grab-bag of mechanisms that form no coherent whole". I agree with this assessment but feel that the ESB concept will mature. The ESB may be a mechanism to provide some plumbing. Many organisations SOA efforts need that plumbing because the system delivery has a number of leaks.

My organization was galvanized into inaction and has not yet bought an ESB. However we are fairly typical I would guess because we want the vendors products to do 'everything' for us. IT is not our core business and we do not want to worry about location or services, security, integration, messaging etc. We want our developers to focus on business logic and data models specific to the business. If the vendor came up with an ESB that really did remove the need to spend time on software infrastructure (and did not charge a fortune for it), then we would be interested.

I would also like our developers to settle on a "one best way" to develop applications rather than the organization investing in training and support for a multitude of programming languages and technologies. Steve advocates using the right language for the job and not being confined to one language. This is a noble cause but it is hard work learning another language and becoming proficient in it. Most developers I know have the one language that they would prefer to write everything in, just as most people I know don't communicate in anything but English. If I weren't so lazy I'd teach myself Italian so I could find out what that Italian site was saying about my blog posting. Similarly I think many developers can't realistically be expected to be truly multilingual when it comes to programming languages.

The discussion of Steve's posting gets a hijacked by the REST vs Web Services debate. This is unfortunate because the ESB discussion should be a separate issue. Using SOAP and Web Services does not require an ESB. ESBs have been conceived to assist with REST such as open source ESBs WSO2 and Mule. The more popular arrangements seem to be that Web Services use an ESB and REST does not, but this need not be the case. The REST vs Web Services argument is taken up elsewhere and in some people's minds have already been settled. Ronald Schmelzer of Zapthink states:

In many ways, however, the debate about Web services and REST is as pointless as arguing whether a hammer or a screwdriver is a better tool.

In the discussion in Steve Vinoski's posting I found myself agreeing with Jonas Ekstrom and Paul H that the ESB market is not yet mature. Jonas put this eloquently:

Religions give easy answers, but there are no easy answers. Natural selection will strike again and again until we have reached the perfect SOA.

Paul H puts a similar slant on this:

I look forward to the day when we’re in a similar stage of maturity … and there exists generally accepted, standardized, manageable, and self-federating mediation technologies … until then, we’ll all have to suffer through endless arguments along the lines of 'should I use a BEA ESB or a Sonic ESB? is an ESB really necessary, or can I get away with REST + Dynamic Languages?'

I will continue to wait until the need for an ESB (or some part of one) becomes self-evident. While an implementation of SOA can function without and ESB there seems little point in making the investment.

Coupling with Contracts

I listened to a podcast by Thomas Erl yesterday which gave yet another slant to the subject of coupling in SOA. My previous posting about coupling was inspired by a Zapthink arcticle on "The Seven Levels of Loose Coupling" seven levels of coupling. The Thomas Erl podcast provides more information on at least three of the seven levels described by Ronald Schmelzer in the Zapthink article. These are

• the coupling between the service consumer and the services implementation,
• the coupling between the service consumer and the service contract, and
• the coupling between the service consumer and the business process

In his podcast Erl asks the listener to refer to a diagram on his "SOA Principles" web-site. The types of coupling he focuses on in the podcast and the diagram are:

Service consumers are coupled to the service contract

• The service contract can be coupled to a parent business process
• The service contract can be coupled to the service logic
• The service logic can be coupled to the service contract
• The service logic may be coupled to proprietary vendor technology
• The service logic may be coupled to multiple services it may need to compose
• The service logic may be coupled to various resources that are part of the overall implementation environment

These coupling relationships broaden the possibilities explained in the Zapthink article. Firstly all the coupling relations are not based around the service consumer and secondly the coupling is not a symmetric relationship. Erl explains that it is good to couple the logic to the contract as this is contract-driven development but poor practice to have the contract tied to the logic of the service.

The multitude of coupling relationships was referred to in my earlier posting. When discussing coupling, it is important to be clear on what is being coupled when explaining service coupling. Zapthink discusses implementation, contract, policy, business process, data schema, infrastructure and semantic layer. What Erl describes is the service logic is part of what Zapthink describes as the implementation. Erl does not describe policy coupling. The data schema is part of the implementation environment and the infrastructure could be vendor technology.

I would prefer to keep coupling as a symmetric relationship. In other words if logic is coupled to contract then contract is coupled to logic. Erl is trying to cover methodological issues when he talks about contract-driven development. This is an important 'pattern' but it is not about coupling. It is about what a developer does first. The fact is that the contract and the logic are coupled and if this coupling can be made loose then that will provided more flexibility.

The focus of the Zapthink article is on the coupling with the service consumer. Erl has a twin focus on the contract and the service logic. My previous discussions on coupling tended to focus on coupling between services (or program modules). I tend to think that most important coupling concern in SOA is between the consumer and the service implementation in SOA. The consumer may be a user interface or another service. Contract coupling is important but it is more of a vehicle for loosening the coupling between consumer and provider.

One strength of the Zapthink article is that it has enabled me to analyse the Erl model. As is usual for Thomas Erl, his illustrations provide some clarity to a complex interaction of concepts. In this case his diagram should be taken as a sample of the coupling issues rather than a representation of all of the coupling issues relevant to SOA. Also as discussed above I suggest we stick with a symmetric notion of coupling to keep the world of SOA a simpler place.