Coupling with Contracts

I listened to a podcast by Thomas Erl yesterday which gave yet another slant to the subject of coupling in SOA. My previous posting about coupling was inspired by a Zapthink arcticle on "The Seven Levels of Loose Coupling" seven levels of coupling. The Thomas Erl podcast provides more information on at least three of the seven levels described by Ronald Schmelzer in the Zapthink article. These are

• the coupling between the service consumer and the services implementation,
• the coupling between the service consumer and the service contract, and
• the coupling between the service consumer and the business process

In his podcast Erl asks the listener to refer to a diagram on his "SOA Principles" web-site. The types of coupling he focuses on in the podcast and the diagram are:

Service consumers are coupled to the service contract

• The service contract can be coupled to a parent business process
• The service contract can be coupled to the service logic
• The service logic can be coupled to the service contract
• The service logic may be coupled to proprietary vendor technology
• The service logic may be coupled to multiple services it may need to compose
• The service logic may be coupled to various resources that are part of the overall implementation environment

These coupling relationships broaden the possibilities explained in the Zapthink article. Firstly all the coupling relations are not based around the service consumer and secondly the coupling is not a symmetric relationship. Erl explains that it is good to couple the logic to the contract as this is contract-driven development but poor practice to have the contract tied to the logic of the service.

The multitude of coupling relationships was referred to in my earlier posting. When discussing coupling, it is important to be clear on what is being coupled when explaining service coupling. Zapthink discusses implementation, contract, policy, business process, data schema, infrastructure and semantic layer. What Erl describes is the service logic is part of what Zapthink describes as the implementation. Erl does not describe policy coupling. The data schema is part of the implementation environment and the infrastructure could be vendor technology.

I would prefer to keep coupling as a symmetric relationship. In other words if logic is coupled to contract then contract is coupled to logic. Erl is trying to cover methodological issues when he talks about contract-driven development. This is an important 'pattern' but it is not about coupling. It is about what a developer does first. The fact is that the contract and the logic are coupled and if this coupling can be made loose then that will provided more flexibility.

The focus of the Zapthink article is on the coupling with the service consumer. Erl has a twin focus on the contract and the service logic. My previous discussions on coupling tended to focus on coupling between services (or program modules). I tend to think that most important coupling concern in SOA is between the consumer and the service implementation in SOA. The consumer may be a user interface or another service. Contract coupling is important but it is more of a vehicle for loosening the coupling between consumer and provider.

One strength of the Zapthink article is that it has enabled me to analyse the Erl model. As is usual for Thomas Erl, his illustrations provide some clarity to a complex interaction of concepts. In this case his diagram should be taken as a sample of the coupling issues rather than a representation of all of the coupling issues relevant to SOA. Also as discussed above I suggest we stick with a symmetric notion of coupling to keep the world of SOA a simpler place.

Does normalization apply to Services?

I was listening to a podcast by Thomas Erl this week where he was talking about a pattern for normalization of services. This was interesting because there is not much in SOA with the same theoretical rigour as database normalization inspired by the work of Edgar Codd in the 1970s and the relational algebra and the Structured Query Language (SQL) that followed it.

The normalization that Erl talks about is fairly limited. On his website he summarizes normalization as

Problem
When delivering services as part of a service inventory, there is a constant risk that services will be created with overlapping functional boundaries, making it difficult to enable wide-spread reuse.
Solution
The service inventory needs to be designed with an emphasis on service boundary alignment.

There does not seem to be a lot of science involved in decomposing services to their basic elements so they can have maximum reuse and be easily composed into higher level services. Perhaps more precisely it is a 'soft' science that is all about governance, process and aligning with user requirements.

Normalization of services is not something that has been picked up strongly by other authors however David Chappell has his description of the normalization project which is consistent with Thomas Erl:

An idealized SOA environment provides a cleanly delineated set of services,
with one way to accomplish each function. This kind of normalization makes reuse
self-enforcing, since each service provides access to a specific group of
information in a specific way.

In his podcast Erl gives an example. If we have a service to get an invoice, then it is redundant to have a service or a method to get an invoice header. To have both is not normalized however there are reasons you may require both and therefore this would be 'denormalized'.

I am a little critical of the use of the term 'normalisation' to describe removing service redundancies. The processs of normalization in data modeling (and mathematics such as linear algebra) is a very precise and well developed technique where the concept used by Erl and Chappell while useful, is fairly loose.

If we wanted to do the equivalent of first normal form in service interfaces we might consider getting rid of the service equivalent of the "repeating groups" that destroys the relational model. If there are many 'invoice items' for an invoice, this means you could not accommodate 'invoice item' in your 'invoice' message. The strength of SOA relies on being able to exchange documents that mean something to the user. This is where attempts to get scientific about normalization fail the cause of SOA.

I will take this normalization discussion to its absurd ultimate. When I did my final year of my honours degree I simulated hardware that consisted entire of only two elements. They were the 'Nand gate' (logical not(and (a,b)) ) and a single binary digit of memory (Flip flop). The computer program that ran on this simulated hardware was simply a wiring diagram. Any computer program that can be represented in normal programming languages can be represented as a combination of these two elements. The sort of programs I ran on this simulation were the full adder, 4 bit timer and 3 bit shift register. Programming at this level is more a level in elementary digital electronics than it is in software development.

This was a distributed network of "services" at the very elementary level, but it is not very useful as a software development approach. This is a completely elementary normalized distributed application that can not be reduced further. This could be described as the ultimate implementation of reuse. You only have two 'services' (Nand and memory) both of which are reused constantly. The challenge is to link them together in meaningful ways. These services however are tightly coupled because of their synchronous nature and therefore this is not an SOA.

As a programming language this is worse than assembler language. Trying to wire gates together is even more detailed than assembler coding. Even if we could get SOA performing brilliantly at lower levels of granularity we do not want to go to such a low level of granularity that business entities are unrecognizable.

Beware dogmatic approaches to normalizing or simplifying a service inventory. I suspect the MEST architectural style with its focus on removing verbs or methods is heading down this type of reductionist thinking.

An overriding concern should be to meet business requirements and this might mean implementing redundant services and poorly structured business documents in services. Designing with a focus to eliminate redundancy is a good idea. It is nevertheless useful to seek out redundancies and investigate the option of reducing these if it does not compromise the ability of the services to represent business processes. Too much science in this process can reduce your services to absurd solutions.

Reference:
Kimber, Antony, The General Logic Array: An Abstract Architecture, Honours Report, The University of Adelaide, Department of Computer Science, November 1985.

This posting may have seemed a flimsy excuse to expose this work to the public domain but I hope it has described a useful anti-pattern for SOA.

The above project was done around the time Sun were having success introducing their Reduced Instruction Set Computer (RISC) chips which included simpler instruction sets that performed more quickly. Reducing down to a couple of elementary components is the extreme end of this strategy. There was also early interest in molecular electronics with the hope that if we could get molecules to behave as electronic components we could produce very small electrical circuits. One of the challenges of the project was working out lattice arrangements that would allow the appropriate connections to the two or more types of elementary components to be made.

Seven Levels of Coupling: Seven Deadly Sins or Seventh Heaven?

In this posting I discuss the Zapthink article by Ronald Schmelzer, "The Seven Levels of Loose Coupling".

This week I have found two commentators redefining loose coupling. The first is described in the discussion about virtual endpoints that was the subject of my previous posting and the second was this Zapthink article. I think it is important we clarify our definitions around SOA to avoid the concept being hijacked by commercial interests. So in this posting and the last I have attempted to sort out whether the discussion is adding to our understanding or muddying the waters.

In a previous posting I have identified 18 different types of coupling and I am still finding more. The first thing to realise about the seven levels of coupling is they are looking at a different attribute of coupling than what I refer to as 'types of coupling'. The levels refer to the entities that are being coupled. My discussion on coupling types looks at the coupling of two modules of program code. This code could be in calling programs, objects, subroutines or services.

The following table lists the seven levels. This table makes the distinction between 'Levels of Coupling' and 'Types of Coupling' clear. The core tenet that Zapthink is discussing is the "loose coupling of Service consumers and providers". This has a different meaning than loose coupling between program code in different services. The main difference is that aspects of the service provider, with the exception of the implementation are not program code at all. Another difference is that a service consumer may not be a service either and it also might not contain program code.

	Aspects of the Service Provider	Possible Types of Coupling
Loose Coupling the Implementation	The service implementation	18 types and still counting
Loose Coupling the Service Contract	Service contract	Interface Coupling, Bind-Time Coupling, Service Intermediary Coupling, Infrastructure configuration Coupling
Loose Coupling the Service Policy	Service policy	Bind-Time Coupling, Service Intermediary Coupling, Infrastructure configuration Coupling
Loose Coupling the Process	Service-oriented process (Trivial if this is a service)	18 types and still counting
Loose Coupling the Data Schema	Data schema of provider dataset and data schema of interface.	Data Services Layer Coupling, Data Transformation Service Coupling
Loose Coupling the Infrastructure	Infrastructure of provider	Single-vendor SOA platform Coupling
Loose Coupling at the Semantic Layer	Meaning of message attributes	Dynamic service definition Coupling

The seven layers are not as clearly layered as the phrase 'Levels' might suggest. Certainly policies could be considered a higher level than contracts and processes considered at higher level to implementation. The other levels could be considered to sit atop the infrastructure level and under the semantic level but normally data and process would sit beside each other with neither the master. I also see contracts sitting beside data and process as equal partners.

Coupling takes account of the dependency between two entities and the clarity of this dependency. Thus two services exchanging a message are loosely coupled if every dependency is clear from the format of the message. Services have different reasons for being dependent. These reasons are what we mean by 'Types of coupling'.

The types of coupling for the seven levels are described in the third column of table above. The first level describes service to service coupling which I have described earlier. The fourth level is interesting because the solution is almost trivial. The business process should look like a service and therefore all the coupling types for coupling between services also apply for coupling between a consumer service and provider service that is a composition of services used to implement a process. The sort of things that might cause dependencies with contacts, polices, infrastructure, data schemas and semantics are different to the program code dependencies.

Loose coupling of data schema is partly addressed by another Zapthink article which advocates the building of data transformation services rather than incorporating transformation logic inside services used to contain business logic.

Tight coupling between the consumer service and the infrastructure of the provider could be based around a number of proprietary technologies. The ones highlighted by Zapthink are the non-standard ESB and the single vendor SOA platform.

The types listed in the third column of the above table are those implied in the Zapthink article. I expect that a thorough investigation would uncover other types of coupling as well. The point of listing them in the table is to suggest that each coupling level will have its own types and each level (with the possible exception of the process level) needs to be investigated separately.

We can find coupling examples everywhere. For example Carlos Perez and David Walend discuss whether a DBMS and its application are loosely coupled if the application is accessing the DBMS using SQL. This is interesting but not relevant to SOA. For the point of view of defining SOA, I do not believe the seven levels of loose coupling matter. You can have an SOA as long as the services are loosely coupled. However as a model to assist good practice, I think the seven levels will provide value even if they do overload the 'loose coupling' term.

The concepts discussed in relation to levels of coupling are certainly valid. We do not want service consumers dying or behaving incorrectly due to changes with service provider's infrastructure, policies or any of the other aspects described in the seven levels. Zapthink continue to guide us on the path to software heaven, however in the process of introducing 'levels of coupling' Zapthink commit the sin of causing confusion around the important 'loose coupling' term. I recommend the "Seven Levels" article to SOA developers and I suggest developers need to be clear on what is being coupled whenever they use or encounter the term 'loose coupling'.

End Note:
One way to distinguish other levels of coupling from service to service implementation coupling would be to call the different levels something different other than 'loose coupling'. The names that occurred to me were 'loose buckling' or even 'loose zipping' which would be reminiscent of the zapthink integration zipper. Unfortunately both these terms have implications of embarrassing failure if the buckling or zipping is too loose. We do not want our metaphorical trousers around our ankles.

Code Abstractions and Babushka Dolls

There are various ways of encapsulating program code. We encapsulate code so that it can be invoked from some other program code. The encapsulated code can be used in its abbreviated form. I will also refer to it as an abstraction in this posting.

There are some ways of encapsulating code that encapsulate a small amount of code and are fine grained. There are some ways of encapsulating a large amount of code or other abstractions of code and are coarse grained. I would like to be able to list all the forms of encapsulation from smallest to largest and arrange them like Russian Babushka dolls with a clear scheme for which goes inside which.

At the finest level of code abstraction I will start with functions, procedures and methods. They are pretty simple. The name 'function' has a mathematical significance the pre-dates computers. Procedures, subroutines and methods are all variations on the theme of a group of instructions.

The next level of abstraction is the 'class' or the instance of that class the 'object'. Object is an unimaginative name. We may as well have used the synonymous term 'thing' but that would have sounded like we just did not know what to call this abstraction of code. "Let's build a thing to access our database" sounds a lot less specific than "Let's build an object to access our database".

Then we have services. These should be reasonably coarse grained. They may contain classes which may contain methods. I skip java beans in this abstraction discussion that would seem to inhabit the same level in the hierarchy as services.

My metaphor of the Babushka dolls is already looking a bit shaky. There are methods that can contain methods, classes that contain classes. Recursion does not work very well with Babushka dolls. Also there are abstractions like code libraries and include files that cut-across the above abstractions, but I will press on anyway.

The level of abstraction I would like to look at is the component as described by the Service Component Architecture (SCA). Component is another synonym for 'thing'. Instead of 'object' and 'component' we may as well have used 'thing 1' and 'thing 2' and apologized to Dr Seuss for using his naming convention. According to the SCA we should then consider encapsulating our components in 'composites' and our 'composites' in 'domains'.

'composites' and 'domains' at least are getting way from being synonyms of 'things'. A composite is just a group of things and a domain is somewhere you find things in, a bit like a container. Of course a 'container' is typically seen as more a runtime environment than a way of collecting together code or abstractions.

'Domain' is a rather overused term in IT at present. Do not try to draw any parallel between an SCA domain, a Domain Name Service (DNS) or Domain Driven Design. There is very little connection.

I would like to bring the concept of Roger Sessions' 'Software Fortress' in at this point. At least Roger has come up with an imaginative name for his abstraction. His explanation of the various roles that software plays in relation to the Fortress is also imaginative as these are illustrated by various goblins, fairies and ogres in his documentation.

Definition: Software Fortress
A software fortress is a conglomerate of software systems consisting of potentially many processes running on potentially many machines spread out over a potentially wide geographic area that:
I. Serve a common purpose
II. Are owned by a cohesive group of individuals
III. Work together in a trust relationship.
IV. Are built on a common technology base.

Roger could improve his definition by indicating whether "or" or "and" separated the different lines of his definition. Definition line IV suggests the Fortress is quite similar to the Domain concept from SCA. If we include the other three definition lines we may find that a Fortress has finer granularity than the domain and may be closer to a composite or component.

The recommendation provided in a couple of presentations I have seen from Glenn Smyth is that web services should be built if and only if they are required on the border of a software fortress. The essence for the SCA is that a service appears on the border of all components. Therefore the finer grained interpretation of 'Software Fortress' that incorporates an 'and' between the last lines of the definition would seem more consistent with this view.

One difference between ways of encapsulating program logic is the stage in the software development life cycle. At runtime then objects, services and containers encapsulate running logic. At development time classes, include files and functions are relevant. At requirements definition time the domains in domain driven design have relevance and the "common purpose" aspect of a fortress helps define the encapsulation. We therefore would more correctly have three sets of Babushka dolls.

SCA is all about runtime and development time. There is nothing that suggests what part of the design a component represents. Domain driven design is all about business requirements. There is nothing in this process that advises how you are to encapsulate your code to represent the domain. SOA attempts to span all three stages of the development life cycle both by suggesting business services and then keeping to this granularity in the developed code and the runtime.

The encapsulation and abstraction of logic is a complicated area. I probably have not made it any simpler with this posting. It is increasingly vital in software. There is not much excitement as there used to be around programming languages but how we bundle together program instructions is still an area of active interest. The point to take away from this is that we have to define our terms clearly and use them in the context of the stage of the software development lifecycle. A consistent approach from design through to runtime would be nice, but at least your software development methodology should identify the mapping between different ways of abstracting business logic thus linking your three or more sets of Babushka dolls.

References

David Chappell, Introducing SCA,

http://www.davidchappell.com/articles/Introducing_SCA.pdf

Eric Evans, Domain Driven Design,(WebCast)
http://www.infoq.com/presentations/model-to-work-evans

Eric Evans, Domain Drive/Model Driven
http://domaindrivendesign.org/discussion/blog/evans_eric_ddd_and_mdd.html

Roger Sessions has book on Software Fortresses which I confess I have not read yet but the following article incorporates and overview of this concept and the newsletter below explains some of its origins.
Roger Sessions, Modeling Software Architectures
http://www.objectwatch.com/PaperAndSpreadsheetVersion1-0.zip

Roger Sessions, The Software Fortress Model; A Next Generation Model for Describing Enterprise Software Architectures, ObjectWatch Newsletter # 36, November, 2001

http://www.objectwatch.com/newsletters/issue_36.htm

Glen Smyth's presentation to the ACS is now available at
http://sa.acs.org.au/it_arch/index.php/Presentations_Past%2C_Present_and_Future

Business Requirements: More than a Pretty Face

In the his opinion piece for ComputerWorld, Michael Hugos tries to give us a quick-fix for the complicated world of software development. A couple of good rebuttals of this "CIO at Large" have already appeared on the internet. In particular Bran Selic of the UML task force exposes the lack of Hugos's lack of UML knowledge.

Like Moliere's Monsieur Jourdain, who spoke prose all his life without knowing it, Hugos may be surprised to discover that four of his five diagram types are either valid UML diagrams or can be mapped directly to corresponding UML diagrams, with one important difference: UML diagrams are composed of elements that have a standard semantics associated with them.

"Five Diagrams Beat a Victorian Novel" is a catchy title and we can all relate to a good diagram being easier to absorb than thousands of words. We should encourage meaningful illustratons but there are many parts of our templates that are not amenable to the simple diagram. We need to talk about objectives, risks, non-functional requirements, costs etc. – these are easier to express in words.

Hugos is advocating a warterfall development methodology and not advocating anything new with his 5 diagrams.

The first diagram is a data flow diagram which was developed by Edward Yourdon and then improved by Gane & Sarson in the 1980s. Gane & Sarson added sources and sinks which evolved to be the actors in use-case diagrams and swim-lanes in swim-lane diagrams. The swim-lane diagram is a special case of the UML Activity Diagram. I prefer to see the swim-lane diagrams to the old Yourdon data flow diagram. The old DFDs certainly still have value but I believe the new diagrams are more expressive and better accepted.

The logical data model is good sense and developers have been using these for years. I would hope that any serious business analysis involved a logical data model and that this was based on an enterprise information model.

Hugos writes about a "System Technical Architecture" which is similar to a UML Deployment Diagram. This is not usually the role of the BA and sometimes not even the role of the software developer. The systems architect will often be responsible for what part of the application runs on what hardware and therefore this deployment diagram.

Hugos write about an Object Model which is the equivalent of a UML Class Diagram. It is the role of the programmer or systems analyst to develop these. Again I have no problem with this diagram being included in a solution design.

Storyboarding is a technique that has been around for a while and has been adopted by the XP (extreme programming) agile programming technique. Use cases attempt to portray the same information.

The use of screen mock-ups helps show the user some tangible picture of what the application might look like and is an alternative to prototyping the user interface. There is a good IBM article on this.

There is little question that use cases have a bigger impact on software engineering than storyboards. Use cases are the origin and basis for a well-defined object model that not only manages the functionality and business rules of a system, but also serves more than one HCI [Human Computer Interface]

The Business Analyst(BA) develops requirements for two audiences. It is the BA's call whether to do screen mock-ups or storyboards. If the BA wants to develop these to convey to the user what the solution might look like then this should be commended. However, a BA still needs to develop the use cases for the developers to understand the logic behind the user interface.

I got a difficult email this week. It went something like "I have just found something new. Apparently five diagrams can replace our long business requirement documents. Please investigate and see if we can get rid of RUP". This is what prompted me to prepare this posting. In general I believe diagrams are goodness but we need words to be precise. Hugos would like to regress to a simpler time. UML is a standard and we use well-accepted standards to make our life simpler. We should not be slaves to our methodologies. Develop screen-shots or storyboards or prototypes if they will help the users understand their requirements. We need to remember that the developer will benefit by seeing a use-case in a format that he or she is comfortable with and expresses what happens behind the user interface.

References:

Two responses to the Hugos article:

Letter: UML Shifts Specs to User's Point of View
Bob Jones, October 15, 2007 (Computerworld)
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=304960

Opinion: Five Diagrams Redux
Bran Selic, an IBM distinguished engineer at IBM Canada, is co-chairman of the UML 2 task force in the OMG
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9043002&source=rss_topic73

The IBM article on Storyboarding

Form feeds function: The role of storyboards in requirements elicitation
http://www.ibm.com/developerworks/rational/library/dec05/krebs/index.html

The Low-Down on Bottoms-Up

This is a completely trivial rant on the use of the phrase "bottoms-up" when I the correct term should be "bottom up". We all know what we mean when software is developed using a "bottom up" approach or that ideas in an organisation come from the "bottom up". "Bottoms-up" on the other hand is a friendly remark made before drinking an alcoholic beverage. It also can be a call to get out of your seat an do some hard work although the origin of the "Head down tail up" description of hard work is more apt for digging and cleaning than it is for desk work.

I agree with Jonathan Betz when he says

What's bugging me these days is that a lot of people are trying to describe Google culture as "bottoms up," when they actually mean "bottom up" Bottoms up means a whole different thing. So, to all you Googlers throwing around the phrase "bottoms up" like it's a good thing: please stop. It hurts my ears when you say it.

I also agree with Jonathan Nightingale

I thought it was a rather humourous typo - obviously someone had confused the drinking cheer with the development methodology on a hungover Monday morning and typed the wrong thing.

I first wrote about this in an earlier posting but I have heard it since on many occasions. I list a number of references below who fall into the "Bottoms Up" abyss. Amongst them Bill Gates is quoted a number of times using this phrase. I like to think that this was probably a journalistic mistake rather than a slip-up by the world's richest computer nerd. Bill Inmon, one of the founders of data warehousing also has a couple of bloggers accusing him of advocating a "Bottoms Up" approach but there seems enough who use the correct term to suggest that Mr Inmon himself did not advocate this phrase.

A "Bottoms Up" approach could be when the software developers go out for a drink and come back in to write some code. This would be a new approach to software quality. The phrase could refer to when everyone is tail up and is very busy but I do not believe this is the intent in most cases. Congratulations though to Chris Keyser who has his own definition for a "Bottoms Up" person suggesting some justification for using this phrase:

But I have definitely been there working with a few bottoms up people. That's the person that when I arrive at some moderately elegant system architecture diagram with some pretty pictures in PowerPoint and Visio catches the few key leaps of faith I made when I am doing my hand waving around my vague idea of implementation, and has saved my rear-end more than once.

Still more worrying is when "Bottoms Up" is used with the phrase "Tops Down". Now that does conjure up some interesting images. The best idea I found on this subject comes from a posting response:

Perhaps they have actually installed a free beer cooler somewhere at IBM, and
they really are encouraging bottoms-up development, and anticipate that enough
of this could lead to tops-down development?

References:

The Bill Gates Quotes
http://www.crn.com/software/183701538
Microsoft wants to make collaboration "simple and bottoms up" but also give users visibility into their important business information, Microsoft Chairman Bill Gates told attendees of the company's second annual Microsoft Office System Developers Conference in Redmond, Wash.
"We are adding new capabilities to make collaboration simple and bottoms-up, but also gives [users] visibility into business information. There are new capabilities in Office aimed at filling the skills gap," Gates said.
http://www.microsoft-watch.com/content/operating_systems/gates_pushes_power_to_the_people_message.html

The productivity benefits accrued by companies that aren't afraid to back "bottoms-up empowerment" was a key theme for Gates during his hour-long address, attended by CEOs from Barnes & Noble, Berkshire-Hathaway, Dell, Delta, Fanny Mae, Hewlett-Packard, Home Depot and other Fortune 1000 firms from around the world.

(and the next one contains "Tops-Down")
http://www.microsoft.com/Presspass/exec/billg/speeches/2007/02-20BillGatesITACtranscript.mspx

The consumer Web brought in, with products like SharePoint, into an internal company environment, so you have bottoms-up creativity but no discontinuity as you're taking that bottoms-up creativity and connecting it with sort of tops-down financial systems because the kind of parts and information out of those systems can easily be imported into any of these engines or portals. We just pick a template and go. We don't have to call IT or think about it, and yet the audit trails and the rights management capabilities built into these things are what's appropriate now for this very digital environment.

The Bill Inmon Attributions
http://www.netcrucible.com/blog/2004/04/09/death-of-the-data-warehouse/

In fact, I remember Bill Inmon (the father of Data Warehousing) fighting against the spread of a particular concept of ?data mart?, since he felt it would give people the impression that data warehousing could be done in a bottoms-up fashion.

http://searchdatamanagement.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid91_gci1173583,00.html
You'll have two basic decisions to make and numerous follow-on decisions:
Enterprise vs. Data Mart-Oriented Architecture
Enterprise vs. Bottoms-up Oriented Methodology

Other Miscellaneous Quotes
http://www.ebizq.net/blogs/firstlook/2007/06/scalable_wikis_made_easy_by_mi.php
"This is a bottoms-up form of communication as opposed to a much more traditional centralized expert driven-down sort of communication," he says.

http://lxer.com/module/newswire/view/84664/index.html
Raising Linux Awareness: A "Bottoms-Up" Approach

http://duckdown.blogspot.com/2007/07/enterprise-architecture-top-down-or.html
(This is a rare blemish for James whose blog on Enterprise Architecture is well worth reading).

One of the more interesting things is that a person debating top-down vs bottoms-up needs to take into consideration is historically which one has had better results. Of course, top-down has the advantage that even mediocre delivery will be declared success while bottoms-up mediocrity will probably result in throwing daggers. If we ignore the whole perception is reality crowd for a moment and truly think about success, then I think our conclusions would be different.

http://www.uxcomm.com/corporate_news_ar_2007-01-09a.asp
In contrast, the "bottoms up" approach is characterized by the use of management software that vendors typically ship with their devices, or which is put in play when an Operating System (OS) is loaded onto a device.

http://www.worcestergroupinc.com/blog/index.php/2006/11/24/bottoms-up-or-top-down-the-eternal-business-intelligence-question/
[ # ] Bottoms Up or Top Down? The Eternal Business Intelligence Question

http://news.zdnet.com/2100-3513_22-6087566.html
Significantly, IBM chose not to define and then build a large-scale, sophisticated knowledge management system. Instead, the company is taking a bottoms-up approach, allowing contributors to have a more active hand in how collaborative work is organized.

http://marxsoftware.blogspot.com/2007/10/highly-useful-jax-ws-introductions.html
I have also followed with interest the debate about web services development with JAX-WS with a bottoms-up approach (annotation Java classes and generating appropriate artifacts) versus with a top-down approach (starting with WSDL).

http://weblog.halmacomber.com/2003_05_04_archive.html
Not only is LISP suitable to this style of development, I'd dare say the language anticipates a bottoms-up collaborative approach. Not all software development is collaborative and bottoms-up, just like not all architecture and engineering is collaborative and bottoms-up. The language helps.

Is Your Enterprise Architecture SOA Compliant?

In August I wrote about the Zachman framework and SOA. This was the basis for a couple more slides at the Ark Group conference on interoperability in which I was a presenter. I have covered material from the presentation in couple for previous postings, "Live up to the Legacy that was Left" and "Why SOA?". I will focus on the main theme of my presentation in this posting which was "Ensuring your Enterprise Architecture is SOA Compliant"

Firstly I would like to point out some work that is being done by the SOA Consortium along these lines. They have announced the Enterprise Architecture 2010 Working Group which is to define the high-level roles and deliverables of the Enterprise Architecture team. There is not much tangible yet from this group as it has just been set up but these people also brought us the SOA practitioners' guides so this could prove a valuable initiative.

At the conference mention in the first paragraph I went over the Zachman Framework as I have done in my earlier posting. One thing I pointed out is that different software development approaches can be described in terms of the framework. I then went on a described my organisations use of the Zachman framework using he colour codes I added to the Zachman grid.

When we talk about top-down approaches conveniently you work from the top row (Scope Row) down to the bottom row (Detailed Representation). Similarly when you work bottom-up you work from the bottom of the framework representation.

Assuming that most good software development is done top-down (and I am subscriber to this approach) you can choose different methodologies depending on the column you choose.

If you drive down the 'Data' column you have a Data Driven Architecture. The Enterprise Information Model in this approach is all important. The other cells of the framework will either support the data column or be influenced by it.

The SOA approach is usually focused on business process and this means it is being driven by the 'Function' column. Software development can be driven from Business Process Modelling and 'Function' column without being SOA.

If we are two focused driving our software development from the 'Network' column we may end up with a siloed application based on organisational locations. This generally software development anti-pattern as modern enterprise software should be independent of location.

Similarly, too much focus on the 'People' column can lead to application focussing around individual or organisational units. This is a very common siloed approach and anti-pattern. More positively a people driven architecture will look at the user experience. This is a common approach for smaller applications to look at screen designs, report layouts and user interaction scenarios that will meet user requirements. For enterprise application though it is important to look at the bigger picture, to question what the organisation is trying to achieve and to model the process from start to finish otherwise the people siloed anti-pattern may result.

If we drive down the 'Time' column we will end up with an Event Driven Architecture (EDA). This is an alternate approach to SOA. We look at events and how they initiate and affect business processes. The 'Function' columns and 'Time' columns can be worked with in tandem to produce and event driven SOA application.

If we work down the 'Motivation' column of the Zachman frame work then we are using a Business Rules Approach. My organisation has not made great use of this framework column. For more information about the Business Rules Approach see http://www.businessrulesgroup.org/ . Also this column is valuable for expert systems and decision management.

With the Zachman framework you do not have to fill every cell. You certainly do not only have one enterprise model in each cell. How you uses the Zachman framework is important. So I will describe how our organisation uses it in terms of the diagram above.

The yellow cells in the above diagram are areas where we have not got any significant documentation.

The green cells contain documentation the business has prepared with out prodding from the IT department. The brown cells are documented by IT but with the interaction of the business.

The top two layers do not change much. These are business level documents. Producing good business process models is more important with SOA. The business process is the key in the SOA process not the application.

The blue is driven by SOA. It starts with your Application Architecture. Distributed Architecture works differently with SOA. You might consider and ESB for instance which will give a different architecture than if you use point-to-point communication. These upper two blue cells drive changes at the more detail level below them.

The pink areas are affected by SOA. Data modelling and Human interface architecture are not radically different but they have to take into account the new architecture.

I will now look at the link between EA and SOA in a different way.

Zapthink has a good article on Zachman and SOA where they use arrows emanating from the Application Architecture cell of the framework. I prefer to represent the influence direction as predominately down representing a top-down approach.

We will start in the 'Data' column again. I do not think the Information Model should be different as a result of SOA. However the Logical Data Model probably should. It should document the interfaces and be able to transform the representation in XML. It should also document the persistent data stores. You now have at least two levels of logical data modelling.

The physical data model and data definition can then be XML, XSD and WSDL. At the same time you have your modelling of persistent data which would be your normal Class Model or ER diagram. This data model would eventually be represented in data dictionaries and database schemas.

We will move across to the 'Function' column now. The Business Process Model should influence your application architecture. Application Architecture should advise you of what services you need and what services to reuse. A service becomes part of the enterprise architecture because it is built for reuse. A new service is also part of a solution architecture. The service has to meet both solution and enterprise needs. Also in this cell is your application stack which supports the service model. The service model then drives the more detailed System Design and the programming.

SOA is a distributed architecture so the lower rows of the 'Network' column takes on increased significance compared with traditional software development approaches. There is more flexibility about where you do your processing. This needs to be built into your Distributed System Architecture.

Business logistics (and particularly whether you are dealing with agencies external to the organisation) will impact your applications architecture. There is a stronger linkage from Application Architecture to Distributed System Architecture because SOA is an enabling technology. It enables you to distribute processing, to look at load balancing and to look at interfaces with other agencies. There is therefore a strong influence on distributed architecture increasing the importance of good architecture in this space.

As with the other columns the influence downward is strong. Your Technology Architecture and Network Architecture needs to be built to cope with the new Distributed Systems Architecture.

Human interface architecture in the 'People' column may change to suit SOA but it does not have to. Some interface technologies work better with SOA. Some ways of interacting with an application work better with SOA but if you want to drive it the other way by specifying your Human Interface Architecture first then this can also work.
Security Architecture gets its own arrow. It is not a logical progression from Presentation Architecture. You may have to do security differently under SOA. You may have to make better use of a standard infrastructure. You may have to use SOA security standards.

SOA for my organisation is business process driven although there is still some bottom up development resulting from services based on mainframe data entities. The focus on reuse and the question for what we already have available to reuse demands some bottom-up consideration.

To be SOA compliant your architecture needs to include:

• Documented business processes,
• Enough bottom-up analysis to reuse of existing services, and
• Data modeling of web service interfaces.

This is in addition to SOA infrastructure and distributed models required the lower levels of your architecture. SOA is an Enterprise Architecture approach. You cannot have an effective SOA unless you have some level of Enterprise Architecture. If you focus on a single solution or application and do not look at the wider needs of the enterprise then you are going to miss the major benefits of SOA.

The business process focus and implications around reuse mean that the business needs to be aware of the SOA approach. It is a change to business practice that the SOA consortium is looking at in its Enterprise Architecture 2010 Working Group.

AOP: The worst form of Coupling

I have just had the need to look at Aspect-Oriented Programming. The Wikipedia entry on this is very good and covers the benefits and risk well. The following provides a good summary of what AOP is all about:

…an aspect can alter the behavior of the base code (the non-aspect part of a program) by applying advice (additional behavior) at various join points (points in a program) specified in a quantification or query called a pointcut (that detects whether a given join point matches).

This offends my sense of what is good modularized code. AOP is done under the banner of "Separation of Concerns" but instead of separating parts of programming code in simple and visible ways, AOP is coupling it tightly. AOP is coupling code modules in ways a programmer would not expect from viewing the affected code. This is tight coupling of the worst kind. In my earlier posting on coupling I rated types of coupling from 0 to 14 from loosest to tightest. I will call this type coupling "code coupling" and give it 15, the tightest coupling index of all. The coupling is through the code rather than through data passed to the module. Code coupling is not new but I did not expect it to come up as modern programming paradigm.

Another form of code coupling is "inclusion coupling" which is explained well by Milind Shingade

Inclusion coupling is a form of coupling in which one component includes source code of another component. If the included component changes something on which includer depends or adds something that raises the conflict with something in the includer then the includer must change.

I see inclusion coupling as more benign than using AOP but I will still roll it into my "code coupling" category.

Control Coupling (coupling index 12) is pretty tight. This is where you pass a Boolean or control parameter to a procedure and have code in that procedure which checks the value and executes different logic based on that control parameter. This is one way of affecting the executed logic in the procedure but at least you can read the code and have some expectation of what the behaviour is going to be. With AOP an unsuspecting maintenance programmer may not have any idea that the program code is being affected by an "aspect".

Unfortunately I have to grant there may be a place for AOP. Code for auditing, logging, exception handling, security and range of routine work that needs to happen along with the business logic can make it difficult to discern the main thrust of what program code is trying to achieve. There is a big need for control and proper governance of AOP. If it has to be used then only senior developers should implement the Aspects. It only should be used in very limited circumstances. Ideally there should be clear documentation in program modules affected by an Aspect.

This should not be a pervasive programming paradigm. The programming should not be Aspect Oriented at all. Code should normal, good Object Oriented code but if AOP is supported a developer may introduce one or two Aspects to influence this code.

In summary I think AOP may be a horrible necessity in some programs and should be used with extreme caution. AOP has given me another type of coupling to add to my list. I had expected the next entry to my coupling list to be in the looser coupling range. I recently listened to a Podcast from Jim Webber of ThoughtWorks. Jim claims that MEST uses looser coupling than WSDL. I would like to see that. I am hoping to blog on this soon.

Lego and SOA

Lego has been used many times as a metaphor for SOA. I first came across it during a vendor presentation and was not particularly impressed by it as I immediately saw some shortcomings in the metaphor.

After thinking about how I would explain some of the benefits of SOA like agility and loose coupling I decided Lego might be fairly descriptive after all. I ended up doing my own presentation on SOA at a planning session complete with blocks from my kids' Lego set. I felt this was one of my more entertaining presentations and it helped get across some good messages. I plan to post a transcription of the presentation on this blog. As it was a "show and tell" presentation I need to get my digital camera fixed first.

When I did this talk I did not realise the Lego metaphor for SOA was so common. Google brings up many hits that relate the two. It is sometimes handy in IT to show something tangible to explain our concepts. What we build is never clearly visible and how we build it is often very abstract.

Gian Trotta does a good job of looking at this metaphor.

These components are loosely coupled code components that represent well-defined and self-contained business services. “This is important. Loose coupling is a design criterion that enables agility. It can't depend on the context or state of other services,” she noted.

The business services become part the overall service oriented architecture (SOA). “Service-oriented architectures require well-defined interfaces and this is the evolving role of Web services. Finally, we have a standard interface that everyone's agreed on, enabling service-oriented architectures to take off,” Gold-Bernstein added.

I do not think much of the Swiss army knife metaphor though. For me SOA is not a grab bag of utilities that can be applied to any problem.

Jason Bloomberg was the source of the above illustration (which is not actually Lego, see footnote) but covers the whole Lego issue pretty well.

As with any analogy, the parallels only go so far, but with Lego blocks, we've found four positive salient characteristics -- and four negative ones

Some of Jason's points identify problems with Lego rather than problems with Lego as a metaphor for SOA. He raises the issue that Lego blocks are so unbreakable that they get passed down the generations so new generations do not buy enough new blocks. This is why the new Lego has so many different parts and looks more realistic. The Lego company is trying to get kids to build it and put it on display rather than reuse it. I see this as a strength of the Lego metaphor (or 'analogy' as Jason refers to it). This is excellent reuse but it also shows the importance of overall architecture. The architecture has adapted to meet the purposes of the business.

The physical nature of Lego means that structures are easier to break as they get bigger, composed of more building blocks. It is interesting to speculate on whether this is true for SOA. Will a big SOA implementation become more fragile? My feeling is that with good design and testing it could be fairly robust but in practice any system with more components and connections will have more points of failure.

Jason states "Lego blocks are interoperable with each other, but not with other kinds of toys." He has a very good point here. Adapters can be built for Lego. I used some sticky tape and a bit of wire attached to Lego blocks when I was demonstrating legacy adapters for an SOA. It is not common practice though to interface Lego with other toys.

Jason's last negative characteristic is "Lego blocks are for children, but children couldn't build Legoland". All metaphors have to break down at some time. SOA is not for children. Services are not made of plastic or cannot fit in an ice cream container either. The metaphorical point is that services can be composed together in an ad hoc fashion without much forethought, but to build large systems with many services that meet the needs of the business requires good SOA and discipline.

One failure of the metaphor I do think has merit is from Jack van Hoof who observes

First start building a nice fortress with LEGO blocks. Then after you finished it, just change one little LEGO block somewhere in the middle of the wall. You won't be very happy, I suppose. Not to mention creating a new window or a door. ... LEGO constructs are easy to be build, but a nightmare to be changed ...

The concern I originally had with the Lego metaphor is aptly covered by a quote from an article by Joe McKendrick who in turn is quoting a vendor white paper

But unlike a single LEGO block, which can only be used in one design at a time, a 'service' can be used by several applications at once (much like a letter in a crossword puzzle).

It is an advantage of software that it can used to process many business processes and it does not get consumed in the process. I sometimes like to consider the service is the block design rather than the actual block but I would not want to try to explain this to an audience as the metaphor starts to lose its simplicity.

The Lego metaphor has been used before in IT and will probably be used for other engineering disciplines. It is a popular model for describing why one might want to build things in parts, why we use standards and how we make big, complex systems out of simple building blocks. For me, Lego continues to be descriptive for SOA. Any discussion about its strengths or weaknesses as a metaphor just seem to help explain more concepts about SOA.


Footnote:
The image above has been presented in a number of articles by Jason Bloomberg of Zapthink. Please note that the blocks illustrated here are not standard Lego blocks. This is a photo of some other interconnecting toy. Standard Lego blocks have the Lego trademark on each of the bumps. Standard Lego blocks also have a gap between the bumps which is larger than the radius of the bumps.

References

• Totta, Gian, The Lego/Swiss Army Knife Approach to SOAs http://www.ebizq.net/topics/soa/features/4040.html
• Bloomberg, Jason, The Lego® Model of SOA, http://www.zapthink.com/report.html?id=zapflash-20061212
• Van Hoof, Jack, SOA and LEGO, do you dare to compare?, http://soa-eda.blogspot.com/2006/12/soa-and-lego-do-you-dare-to-compare.html
• Taylor, James, If SOA is Lego then decision services are Mindstorms http://www.ebizq.net/blogs/decision_management/2006/12/if_soa_is_lego_then_decision_s.php
• McKendrick, Joe, Why are we building ‘enterprise’ SOAs with ‘LEGO blocks’?http://blogs.zdnet.com/service-oriented/?p=777

LEGO® is a trademark of the LEGO Group of companies which does not sponsor, authorize or endorse this content.

The Other Michael Jackson

Once upon a time (1975 to be precise) there was a book written by M.A.Jackson called "Principles of Program Design". This was very influential and I was certainly impressed. It provide elegant design patterns that allowed program code to be developed from problems relating to data structures. It was a small handbook of how to navigate through various data structures, how to merge data structures and how to structure programs so they were easily understandable based on these data structures. It was probably the first book to publish patterns of program code. It was language agnostic so it did not matter whether you programmed in COBOL, Algol, Pascal or Fortran there were examples in your favourite language. I bought this book in its hardback edition and it still has a place on bookshelf.

I was not alone in being an admirer of this book. Jackson founded an whole methodology around his ideas know as Jackson Structured Programming. Yes, he was the original owner of the JSP acronym.

The design patterns are still relevant now, although the concept of multiple co-operating process probably makes the work on merging incompatible data structures less relevant. Today's popular design patterns have a different emphasis but it is important we do not lose the basic techniques that Jackson espoused. The important lesson from this work is the concept of design patterns. There is a pattern which is efficient and intelligible. If everyone sticks to the pattern it is much easier to follow what the program is doing.

Some of the Java EE patterns can be found at
http://java.sun.com/blueprints/patterns/

Model View Controller

I was about to write this post about the various frameworks in the Model View Controller (MVC) architectural pattern. I checked Wiki to see what it had to say on MVC only to find that the two frameworks that I wanted to discuss were not listed even though I thought they conformed to this paradigm.

The two frameworks in question are Apache Cocoon and BEA Page Flows. My organisation started down the Cocoon path fairly early in its J2EE (now Java EE) history. This seemed a good open-source product for separating the user interface, data and business logic. We seemed to be a lone voice with other government departments going for Struts, which is listed in Wiki as a MVC framework.

We moved away from Cocoon when we purchased the BEA Weblogic platform and now have adopted Page Flows as our framework of choice. Page flows is not billed as a MVC framework but provides a sufficient level of detail to easily implement MVC. That means we have another choice to make in the pursuit of our set of SOA standards and technologies. If we have settled on the Page Flow framework now we may also have to specify we implement it in an MVC fashion.

Interestingly Cocoon has separation of concerns as a clear focus although it does not claim to be MVC but does provide similar separation.

It would seem that both Cocoon and Page Flows do not dictate an MVC solution however the MVC pattern can be implemented with both of them.

MVC: http://java.sun.com/blueprints/patterns/MVC.html

Cocoon: http://www.java-source.net/open-source/web-frameworks/cocoon

Page Flows: http://dev2dev.bea.com/pub/a/2004/01/Dew.html

Cocoon and MVC: http://webservices.xml.com/pub/a/ws/2003/01/29/cocoon-xforms.html