Coupling with Contracts

I listened to a podcast by Thomas Erl yesterday which gave yet another slant to the subject of coupling in SOA. My previous posting about coupling was inspired by a Zapthink arcticle on "The Seven Levels of Loose Coupling" seven levels of coupling. The Thomas Erl podcast provides more information on at least three of the seven levels described by Ronald Schmelzer in the Zapthink article. These are

• the coupling between the service consumer and the services implementation,
• the coupling between the service consumer and the service contract, and
• the coupling between the service consumer and the business process

In his podcast Erl asks the listener to refer to a diagram on his "SOA Principles" web-site. The types of coupling he focuses on in the podcast and the diagram are:

Service consumers are coupled to the service contract

• The service contract can be coupled to a parent business process
• The service contract can be coupled to the service logic
• The service logic can be coupled to the service contract
• The service logic may be coupled to proprietary vendor technology
• The service logic may be coupled to multiple services it may need to compose
• The service logic may be coupled to various resources that are part of the overall implementation environment

These coupling relationships broaden the possibilities explained in the Zapthink article. Firstly all the coupling relations are not based around the service consumer and secondly the coupling is not a symmetric relationship. Erl explains that it is good to couple the logic to the contract as this is contract-driven development but poor practice to have the contract tied to the logic of the service.

The multitude of coupling relationships was referred to in my earlier posting. When discussing coupling, it is important to be clear on what is being coupled when explaining service coupling. Zapthink discusses implementation, contract, policy, business process, data schema, infrastructure and semantic layer. What Erl describes is the service logic is part of what Zapthink describes as the implementation. Erl does not describe policy coupling. The data schema is part of the implementation environment and the infrastructure could be vendor technology.

I would prefer to keep coupling as a symmetric relationship. In other words if logic is coupled to contract then contract is coupled to logic. Erl is trying to cover methodological issues when he talks about contract-driven development. This is an important 'pattern' but it is not about coupling. It is about what a developer does first. The fact is that the contract and the logic are coupled and if this coupling can be made loose then that will provided more flexibility.

The focus of the Zapthink article is on the coupling with the service consumer. Erl has a twin focus on the contract and the service logic. My previous discussions on coupling tended to focus on coupling between services (or program modules). I tend to think that most important coupling concern in SOA is between the consumer and the service implementation in SOA. The consumer may be a user interface or another service. Contract coupling is important but it is more of a vehicle for loosening the coupling between consumer and provider.

One strength of the Zapthink article is that it has enabled me to analyse the Erl model. As is usual for Thomas Erl, his illustrations provide some clarity to a complex interaction of concepts. In this case his diagram should be taken as a sample of the coupling issues rather than a representation of all of the coupling issues relevant to SOA. Also as discussed above I suggest we stick with a symmetric notion of coupling to keep the world of SOA a simpler place.

Does normalization apply to Services?

I was listening to a podcast by Thomas Erl this week where he was talking about a pattern for normalization of services. This was interesting because there is not much in SOA with the same theoretical rigour as database normalization inspired by the work of Edgar Codd in the 1970s and the relational algebra and the Structured Query Language (SQL) that followed it.

The normalization that Erl talks about is fairly limited. On his website he summarizes normalization as

Problem
When delivering services as part of a service inventory, there is a constant risk that services will be created with overlapping functional boundaries, making it difficult to enable wide-spread reuse.
Solution
The service inventory needs to be designed with an emphasis on service boundary alignment.

There does not seem to be a lot of science involved in decomposing services to their basic elements so they can have maximum reuse and be easily composed into higher level services. Perhaps more precisely it is a 'soft' science that is all about governance, process and aligning with user requirements.

Normalization of services is not something that has been picked up strongly by other authors however David Chappell has his description of the normalization project which is consistent with Thomas Erl:

An idealized SOA environment provides a cleanly delineated set of services,
with one way to accomplish each function. This kind of normalization makes reuse
self-enforcing, since each service provides access to a specific group of
information in a specific way.

In his podcast Erl gives an example. If we have a service to get an invoice, then it is redundant to have a service or a method to get an invoice header. To have both is not normalized however there are reasons you may require both and therefore this would be 'denormalized'.

I am a little critical of the use of the term 'normalisation' to describe removing service redundancies. The processs of normalization in data modeling (and mathematics such as linear algebra) is a very precise and well developed technique where the concept used by Erl and Chappell while useful, is fairly loose.

If we wanted to do the equivalent of first normal form in service interfaces we might consider getting rid of the service equivalent of the "repeating groups" that destroys the relational model. If there are many 'invoice items' for an invoice, this means you could not accommodate 'invoice item' in your 'invoice' message. The strength of SOA relies on being able to exchange documents that mean something to the user. This is where attempts to get scientific about normalization fail the cause of SOA.

I will take this normalization discussion to its absurd ultimate. When I did my final year of my honours degree I simulated hardware that consisted entire of only two elements. They were the 'Nand gate' (logical not(and (a,b)) ) and a single binary digit of memory (Flip flop). The computer program that ran on this simulated hardware was simply a wiring diagram. Any computer program that can be represented in normal programming languages can be represented as a combination of these two elements. The sort of programs I ran on this simulation were the full adder, 4 bit timer and 3 bit shift register. Programming at this level is more a level in elementary digital electronics than it is in software development.

This was a distributed network of "services" at the very elementary level, but it is not very useful as a software development approach. This is a completely elementary normalized distributed application that can not be reduced further. This could be described as the ultimate implementation of reuse. You only have two 'services' (Nand and memory) both of which are reused constantly. The challenge is to link them together in meaningful ways. These services however are tightly coupled because of their synchronous nature and therefore this is not an SOA.

As a programming language this is worse than assembler language. Trying to wire gates together is even more detailed than assembler coding. Even if we could get SOA performing brilliantly at lower levels of granularity we do not want to go to such a low level of granularity that business entities are unrecognizable.

Beware dogmatic approaches to normalizing or simplifying a service inventory. I suspect the MEST architectural style with its focus on removing verbs or methods is heading down this type of reductionist thinking.

An overriding concern should be to meet business requirements and this might mean implementing redundant services and poorly structured business documents in services. Designing with a focus to eliminate redundancy is a good idea. It is nevertheless useful to seek out redundancies and investigate the option of reducing these if it does not compromise the ability of the services to represent business processes. Too much science in this process can reduce your services to absurd solutions.

Reference:
Kimber, Antony, The General Logic Array: An Abstract Architecture, Honours Report, The University of Adelaide, Department of Computer Science, November 1985.

This posting may have seemed a flimsy excuse to expose this work to the public domain but I hope it has described a useful anti-pattern for SOA.

The above project was done around the time Sun were having success introducing their Reduced Instruction Set Computer (RISC) chips which included simpler instruction sets that performed more quickly. Reducing down to a couple of elementary components is the extreme end of this strategy. There was also early interest in molecular electronics with the hope that if we could get molecules to behave as electronic components we could produce very small electrical circuits. One of the challenges of the project was working out lattice arrangements that would allow the appropriate connections to the two or more types of elementary components to be made.

Book Review: Loosely Coupled - The Missing Pieces…

This is a review of the book "Loosely Coupled: The Missing Piece of Web Services", by Doug Kaye. In a previous posting I expressed delight that an author had dedicated a book to this important topic. This perhaps is not the definitive book on loose coupling but there is plenty of value for the reader.

Doug has an easily readable style and sticks to a high level but there is enough technical substance to keep and IT architect or IT manager interested. You will find no code snippets and no programming standard specifications described in this book but there are some good diagrams and deep insights into SOA and how an organization should deliver SOA.

This book was published in 2003 and I kept expecting the information to be dated. There was no mention of an Enterprise Service Bus (ESB) but Doug Kaye describes a number network models, mediation tools and deployment options which have been incorporated into what vendors now label as an ESB. More importantly some of the missing pieces he describes in 2003 are still missing at the end of 2007. Security, transaction integrity and business semantics still do not have accepted solutions within SOA and these missing pieces are described in this book. The WS-* standards have progressed somewhat since the writing of "Loosely Coupled" but basics of SOAP, WSDL and UDDI were in place and many for the WS standards are foretold.

Doug has good perspective on what is required in the corporate world. He advises everyone to have your "elevator pitch" ready for that occasion when the CEO gets into the elevator with you and asks something like "So what's this I keep hearing about web services? What are they and why are they so important" (p27). Doug offers a response that suggests he is either a fast talker or works in a building that is much taller than mine, but the concept of having something ready like this is important if you are going evangelize about web services.

There is only one chapter dedicated to Loose Coupling and Doug has not tried to say everything there is to say on the subject. This justified in his statement "…loose coupling is a methodology or style, rather than a set of established rules and specifications" (p. 131). Figure 10-1 provides a table describing 10 types of coupling and descriptions of tightly and loosely coupled approaches to each. This is a good approach as it is not easy to describe coupling succinctly. I found this approach in another posting although interestingly the contents are different. I have found that in my previous posting that it is difficult to label types and coupling and the three column approach will enable me to do this more effectively. The book covers late binding, heterogeneity, routing and transformations well.

It is true to say that this book is more to do about its subtitle "The Missing Pieces of Web Services" than Loose Coupling. Doug makes the bold prediction that when we have solved all the pieces of the puzzle in the future that it will be so commonplace that "Web services won't be mentioned by the IT trade press, and the topic won't be tracked by analysts". The book searches through these missing pieces in most of the chapters. The fact that Web Services is still a strong topic for analysts and vendors is evidence suggesting that we have not yet found all those missing pieces.

Doug Kaye provides a good description of transaction control although I found the book I read earlier by Krafzig et al to have more detail in this area. Doug has two strong chapters on the problems of web services security and covers this topic in a thorough and systematic way. A couple of reviews have suggested that the book by Mark O'Neil also provides good coverage of Web Service security.

The chapter on deployment option talks about commercial opportunities for third parties in the delivery of services and services infrastructure. This discussed Web Service Networks, Distributed Web Service Networks, Web-Service Delivery Networks and Web Service Providers. Doug takes the service vendor view in his final chapter on providing external services. Both these chapters had little relevance to me. Four years after this book was written I have not been flooded with vendors wanting to sell me services or wanting to host infrastructure for me to operate my web service applications. Perhaps the money is not there, or perhaps it is just easier to use open source products or set up our own infrastructure. I still think Web Services provides some opportunities for commercial intermediaries, but until the vendors are out there and dealing in the government sector it will be difficult for me to get excited about this topic.

Doug writes a couple of good chapters on the management of simple and complex web services projects. In particular he deals with timing the project around web services approaches that have not yet been delivered. The chapter on Service Level Agreements (SLAs) was an important piece of the completion of the web services picture.

The book concludes with a number of checklists that anyone embarking on an SOA project should refer to. This should help the SOA project manager get their ducks in line to navigate the difficult road ahead.

In summary, this was a well written book about the evolving phenomenon of Web Services. There are some valuable insights into Loose Coupling but an exhaustive technical treatment of coupling and the programming patterns that can reduce coupling is out of the scope of this book. There are some important insights to be found, especially on the subjects of security, evangelizing and commercial arrangements. I expect I will refer to this book many times in the future.

Loosely Coupled: The Missing Pieces of Web Services, Doug Kaye ; RDS Press 2003, ISBN 1881378241

Seven Levels of Coupling: Seven Deadly Sins or Seventh Heaven?

In this posting I discuss the Zapthink article by Ronald Schmelzer, "The Seven Levels of Loose Coupling".

This week I have found two commentators redefining loose coupling. The first is described in the discussion about virtual endpoints that was the subject of my previous posting and the second was this Zapthink article. I think it is important we clarify our definitions around SOA to avoid the concept being hijacked by commercial interests. So in this posting and the last I have attempted to sort out whether the discussion is adding to our understanding or muddying the waters.

In a previous posting I have identified 18 different types of coupling and I am still finding more. The first thing to realise about the seven levels of coupling is they are looking at a different attribute of coupling than what I refer to as 'types of coupling'. The levels refer to the entities that are being coupled. My discussion on coupling types looks at the coupling of two modules of program code. This code could be in calling programs, objects, subroutines or services.

The following table lists the seven levels. This table makes the distinction between 'Levels of Coupling' and 'Types of Coupling' clear. The core tenet that Zapthink is discussing is the "loose coupling of Service consumers and providers". This has a different meaning than loose coupling between program code in different services. The main difference is that aspects of the service provider, with the exception of the implementation are not program code at all. Another difference is that a service consumer may not be a service either and it also might not contain program code.

	Aspects of the Service Provider	Possible Types of Coupling
Loose Coupling the Implementation	The service implementation	18 types and still counting
Loose Coupling the Service Contract	Service contract	Interface Coupling, Bind-Time Coupling, Service Intermediary Coupling, Infrastructure configuration Coupling
Loose Coupling the Service Policy	Service policy	Bind-Time Coupling, Service Intermediary Coupling, Infrastructure configuration Coupling
Loose Coupling the Process	Service-oriented process (Trivial if this is a service)	18 types and still counting
Loose Coupling the Data Schema	Data schema of provider dataset and data schema of interface.	Data Services Layer Coupling, Data Transformation Service Coupling
Loose Coupling the Infrastructure	Infrastructure of provider	Single-vendor SOA platform Coupling
Loose Coupling at the Semantic Layer	Meaning of message attributes	Dynamic service definition Coupling

The seven layers are not as clearly layered as the phrase 'Levels' might suggest. Certainly policies could be considered a higher level than contracts and processes considered at higher level to implementation. The other levels could be considered to sit atop the infrastructure level and under the semantic level but normally data and process would sit beside each other with neither the master. I also see contracts sitting beside data and process as equal partners.

Coupling takes account of the dependency between two entities and the clarity of this dependency. Thus two services exchanging a message are loosely coupled if every dependency is clear from the format of the message. Services have different reasons for being dependent. These reasons are what we mean by 'Types of coupling'.

The types of coupling for the seven levels are described in the third column of table above. The first level describes service to service coupling which I have described earlier. The fourth level is interesting because the solution is almost trivial. The business process should look like a service and therefore all the coupling types for coupling between services also apply for coupling between a consumer service and provider service that is a composition of services used to implement a process. The sort of things that might cause dependencies with contacts, polices, infrastructure, data schemas and semantics are different to the program code dependencies.

Loose coupling of data schema is partly addressed by another Zapthink article which advocates the building of data transformation services rather than incorporating transformation logic inside services used to contain business logic.

Tight coupling between the consumer service and the infrastructure of the provider could be based around a number of proprietary technologies. The ones highlighted by Zapthink are the non-standard ESB and the single vendor SOA platform.

The types listed in the third column of the above table are those implied in the Zapthink article. I expect that a thorough investigation would uncover other types of coupling as well. The point of listing them in the table is to suggest that each coupling level will have its own types and each level (with the possible exception of the process level) needs to be investigated separately.

We can find coupling examples everywhere. For example Carlos Perez and David Walend discuss whether a DBMS and its application are loosely coupled if the application is accessing the DBMS using SQL. This is interesting but not relevant to SOA. For the point of view of defining SOA, I do not believe the seven levels of loose coupling matter. You can have an SOA as long as the services are loosely coupled. However as a model to assist good practice, I think the seven levels will provide value even if they do overload the 'loose coupling' term.

The concepts discussed in relation to levels of coupling are certainly valid. We do not want service consumers dying or behaving incorrectly due to changes with service provider's infrastructure, policies or any of the other aspects described in the seven levels. Zapthink continue to guide us on the path to software heaven, however in the process of introducing 'levels of coupling' Zapthink commit the sin of causing confusion around the important 'loose coupling' term. I recommend the "Seven Levels" article to SOA developers and I suggest developers need to be clear on what is being coupled whenever they use or encounter the term 'loose coupling'.

End Note:
One way to distinguish other levels of coupling from service to service implementation coupling would be to call the different levels something different other than 'loose coupling'. The names that occurred to me were 'loose buckling' or even 'loose zipping' which would be reminiscent of the zapthink integration zipper. Unfortunately both these terms have implications of embarrassing failure if the buckling or zipping is too loose. We do not want our metaphorical trousers around our ankles.

Virtual Endpoints: A Loose Definition for Loose Coupling?

I listened to a podcast this week by Miko Matsumura, Deputy CTO of Software AG. His comments on Governance were interesting especially the concept of hot and cold parts of the architecture. Some things like policies should remain "cool" and not change often. Other parts such as how you combine your services into applications should be relatively "warm" and change more often.

The point that jumped out at me was his very concise descript of loose coupling "Loose Coupling has virtualized endpoints". No further definition was given.

This must be a bit of a Software AG thing. John Michelsen, also of Software AG says

…virtual endpoints are a great idea as you think about the scalability and the loose coupling we want in a SOA.

This is better explained in John's posting "An Intro to SOA and Virtualization" but all this virtualisation seems to be is getting rid of hard coded URLs. This does not seem terribly revolutionary and is certainly not synonymous with the concept of loose coupling that is, according to some writers, the whole reason behind the benefits of SOA.

Before I discount this as an overly narrow vendor view I would like to explore the concept a bit more deeply. If I look down my ever-increasing list coupling types I can see a number of properties of service interfaces. I define and endpoint as being a service interface then we can see an endpoint has having the following properties and we can see some corresponding coupling types:

Endpoint Property	Coupling Type (From previous posting)
URL	Registry Coupling
Platform	Platform Coupling
Interface Time	Queue Coupling
Processor Location	Distributed Coupling
Data Field Order & Presence	XML Message Payload Coupling

The only property that may need some explanation is "Interface Time". This is the time that a message leaves from or arrives at a service interface. If no queuing is involved then the only delay between the message leaving from a sending service and arriving at the receiving service is some network latency. If a queue is involved then the delay uncouples the sending time from the receiving time.

The concept of an endpoint does not have much meaning except where messaging is involved. There are some types of coupling in the messaging world that are not covered by the concept of virtualised endpoints. Coupling based on process state (including synchronous coupling) and coupling introduced to manage process control, such as transaction control is not easily expressed in terms of endpoint properties. Logical coupling where service behaviour changes significantly based Boolean or enumerated field values passed in the message also is not covered as an endpoint property.

The "Virtual Endpoint" concept turns out to be a fairly useful. We have some types of coupling that are only dependant on endpoints (or service interfaces) and others that depend more deeply on the implementation of the services or the interpretation of the messages being passed. Virtual endpoints are not the whole story when it comes to loose coupling but are part of it.

Subroutine Call Coupling

It has been more than a month since I did my last posting on coupling. Amazon has delivered my book on loose coupling by Doug Kaye and I am finding it valuable. Once I have finished that I hope to revise my original list of types of coupling.

In this posting I would like to say something about subroutine call semantics versus message passing and the coupling associated with this. I have discussed before that the synchronous approach which is a requirement of the subroutine call semantics is more tightly coupled than an asynchronous approach. There are two other differences between subroutine calling and message passing. These are

• the management of state of the calling module and the called module; and
• the management of the called instance itself.

When you invoke a subroutine it starts executing from its beginning. The invoked routine will usually have no state and complete before the invoking routine resumes again. Another form of routine call is the coroutine where the state in the called module is maintained from one call to another. The issue of state coupling was covered in my earlier posting and labelled as 'Process State Coupling'. This type coupling is intended to cover coroutine behaviour. A coroutine is even more tightly coupled than a normal subroutine call because of the dependency on state.

In a normal subroutine call state is not preserved in the called module from one call to the next however state is preserved in the calling module. It preserves its state by the simple expedient of suspending execution until the called module returns control.

Asynchronous message can still involve process state coupling. More loosely coupled messaging has enough information in the message to process without retaining state. This is, in general, better practice but is not always practical.

There is one other aspect of subroutine call behaviour that is different from message passing behaviour. When a subroutine is invoked it is in effect brought to life at that point. If you have to create an instance of a service in order to use it then this is another form of dependency. A service should be an entity that always "exists" that is ready to serve. In effect messages can wake up receiving services but it should not be the concern of the sending service to create or dispose of an instance of the service. I therefore intend to add 'Creation Coupling' to the coupling list when I next publish one. Creation coupling occurs in Java classes when a new object is created using the 'New' statement. Methods of that object are then called but the creation of new object (and sometimes its destruction) is often the role of the calling object.

It is the system's responsibility to effect and manage the invocation of the service, not that of the calling application. This allows two critical characteristics to be realized: 1) The services are truly independent, and 2) they can be managed. (Anagol-Subbarao)

In the paper "Web Servces Are Not Distributed Objects" by Werner Vogels he draws this distinction clearly. Loosely coupled web services do not have the same behaviour as objects.

An important notion at the core of distributed object technology is the object life cycle:
· Upon request, a factory instantiates the objects.
· The consumer who requested the instantiation performs various operations on the object instance.
· Sometime later, either the consumer releases the instance or the runtime system garbage-collects it.

It is this point that Werner distinguishes between services and objects and is also this point that makes distributed objects more tightly coupled than web services that are coupled using messages.

In this posting we have discussed subroutine semantics and discovered that subroutine a tightly coupled to their calling modules because of:

• Synchronous Coupling
• Process State Couping, and
• Creation Coupling

The use of asynchronous messages is much more loosely coupled as a result.

References:
Anagol-Subbarao, Anjali, J2EE Web Services on BEA WebLogic, Prentice Hall, October, 2004

Vogels, Werner, "Web Services are not Distributed Objects: Common Misconceptions about Service Oriented Architectures" , IEEE Internet Computing, vol. 7, no. 6, pp. 59–66, 2003

This is also available from Werner's blog site at
http://www.allthingsdistributed.com/historical/archives/000343.html

Marlin, Christopher D. Coroutines, Lecture notes in computer science, vol. 95, Springer-Verlag, Berlin, Heidelberg, 1980. ISBN 3 540 10256 6 and ISBN 0 387 10256 6.

Kaye, Doug, Loosely Coupled: The Missing Pieces of Web Services; RDS Press 2003, ISBN 1881378241

Code Abstractions and Babushka Dolls

There are various ways of encapsulating program code. We encapsulate code so that it can be invoked from some other program code. The encapsulated code can be used in its abbreviated form. I will also refer to it as an abstraction in this posting.

There are some ways of encapsulating code that encapsulate a small amount of code and are fine grained. There are some ways of encapsulating a large amount of code or other abstractions of code and are coarse grained. I would like to be able to list all the forms of encapsulation from smallest to largest and arrange them like Russian Babushka dolls with a clear scheme for which goes inside which.

At the finest level of code abstraction I will start with functions, procedures and methods. They are pretty simple. The name 'function' has a mathematical significance the pre-dates computers. Procedures, subroutines and methods are all variations on the theme of a group of instructions.

The next level of abstraction is the 'class' or the instance of that class the 'object'. Object is an unimaginative name. We may as well have used the synonymous term 'thing' but that would have sounded like we just did not know what to call this abstraction of code. "Let's build a thing to access our database" sounds a lot less specific than "Let's build an object to access our database".

Then we have services. These should be reasonably coarse grained. They may contain classes which may contain methods. I skip java beans in this abstraction discussion that would seem to inhabit the same level in the hierarchy as services.

My metaphor of the Babushka dolls is already looking a bit shaky. There are methods that can contain methods, classes that contain classes. Recursion does not work very well with Babushka dolls. Also there are abstractions like code libraries and include files that cut-across the above abstractions, but I will press on anyway.

The level of abstraction I would like to look at is the component as described by the Service Component Architecture (SCA). Component is another synonym for 'thing'. Instead of 'object' and 'component' we may as well have used 'thing 1' and 'thing 2' and apologized to Dr Seuss for using his naming convention. According to the SCA we should then consider encapsulating our components in 'composites' and our 'composites' in 'domains'.

'composites' and 'domains' at least are getting way from being synonyms of 'things'. A composite is just a group of things and a domain is somewhere you find things in, a bit like a container. Of course a 'container' is typically seen as more a runtime environment than a way of collecting together code or abstractions.

'Domain' is a rather overused term in IT at present. Do not try to draw any parallel between an SCA domain, a Domain Name Service (DNS) or Domain Driven Design. There is very little connection.

I would like to bring the concept of Roger Sessions' 'Software Fortress' in at this point. At least Roger has come up with an imaginative name for his abstraction. His explanation of the various roles that software plays in relation to the Fortress is also imaginative as these are illustrated by various goblins, fairies and ogres in his documentation.

Definition: Software Fortress
A software fortress is a conglomerate of software systems consisting of potentially many processes running on potentially many machines spread out over a potentially wide geographic area that:
I. Serve a common purpose
II. Are owned by a cohesive group of individuals
III. Work together in a trust relationship.
IV. Are built on a common technology base.

Roger could improve his definition by indicating whether "or" or "and" separated the different lines of his definition. Definition line IV suggests the Fortress is quite similar to the Domain concept from SCA. If we include the other three definition lines we may find that a Fortress has finer granularity than the domain and may be closer to a composite or component.

The recommendation provided in a couple of presentations I have seen from Glenn Smyth is that web services should be built if and only if they are required on the border of a software fortress. The essence for the SCA is that a service appears on the border of all components. Therefore the finer grained interpretation of 'Software Fortress' that incorporates an 'and' between the last lines of the definition would seem more consistent with this view.

One difference between ways of encapsulating program logic is the stage in the software development life cycle. At runtime then objects, services and containers encapsulate running logic. At development time classes, include files and functions are relevant. At requirements definition time the domains in domain driven design have relevance and the "common purpose" aspect of a fortress helps define the encapsulation. We therefore would more correctly have three sets of Babushka dolls.

SCA is all about runtime and development time. There is nothing that suggests what part of the design a component represents. Domain driven design is all about business requirements. There is nothing in this process that advises how you are to encapsulate your code to represent the domain. SOA attempts to span all three stages of the development life cycle both by suggesting business services and then keeping to this granularity in the developed code and the runtime.

The encapsulation and abstraction of logic is a complicated area. I probably have not made it any simpler with this posting. It is increasingly vital in software. There is not much excitement as there used to be around programming languages but how we bundle together program instructions is still an area of active interest. The point to take away from this is that we have to define our terms clearly and use them in the context of the stage of the software development lifecycle. A consistent approach from design through to runtime would be nice, but at least your software development methodology should identify the mapping between different ways of abstracting business logic thus linking your three or more sets of Babushka dolls.

References

David Chappell, Introducing SCA,

http://www.davidchappell.com/articles/Introducing_SCA.pdf

Eric Evans, Domain Driven Design,(WebCast)
http://www.infoq.com/presentations/model-to-work-evans

Eric Evans, Domain Drive/Model Driven
http://domaindrivendesign.org/discussion/blog/evans_eric_ddd_and_mdd.html

Roger Sessions has book on Software Fortresses which I confess I have not read yet but the following article incorporates and overview of this concept and the newsletter below explains some of its origins.
Roger Sessions, Modeling Software Architectures
http://www.objectwatch.com/PaperAndSpreadsheetVersion1-0.zip

Roger Sessions, The Software Fortress Model; A Next Generation Model for Describing Enterprise Software Architectures, ObjectWatch Newsletter # 36, November, 2001

http://www.objectwatch.com/newsletters/issue_36.htm

Glen Smyth's presentation to the ACS is now available at
http://sa.acs.org.au/it_arch/index.php/Presentations_Past%2C_Present_and_Future

Business Requirements: More than a Pretty Face

In the his opinion piece for ComputerWorld, Michael Hugos tries to give us a quick-fix for the complicated world of software development. A couple of good rebuttals of this "CIO at Large" have already appeared on the internet. In particular Bran Selic of the UML task force exposes the lack of Hugos's lack of UML knowledge.

Like Moliere's Monsieur Jourdain, who spoke prose all his life without knowing it, Hugos may be surprised to discover that four of his five diagram types are either valid UML diagrams or can be mapped directly to corresponding UML diagrams, with one important difference: UML diagrams are composed of elements that have a standard semantics associated with them.

"Five Diagrams Beat a Victorian Novel" is a catchy title and we can all relate to a good diagram being easier to absorb than thousands of words. We should encourage meaningful illustratons but there are many parts of our templates that are not amenable to the simple diagram. We need to talk about objectives, risks, non-functional requirements, costs etc. – these are easier to express in words.

Hugos is advocating a warterfall development methodology and not advocating anything new with his 5 diagrams.

The first diagram is a data flow diagram which was developed by Edward Yourdon and then improved by Gane & Sarson in the 1980s. Gane & Sarson added sources and sinks which evolved to be the actors in use-case diagrams and swim-lanes in swim-lane diagrams. The swim-lane diagram is a special case of the UML Activity Diagram. I prefer to see the swim-lane diagrams to the old Yourdon data flow diagram. The old DFDs certainly still have value but I believe the new diagrams are more expressive and better accepted.

The logical data model is good sense and developers have been using these for years. I would hope that any serious business analysis involved a logical data model and that this was based on an enterprise information model.

Hugos writes about a "System Technical Architecture" which is similar to a UML Deployment Diagram. This is not usually the role of the BA and sometimes not even the role of the software developer. The systems architect will often be responsible for what part of the application runs on what hardware and therefore this deployment diagram.

Hugos write about an Object Model which is the equivalent of a UML Class Diagram. It is the role of the programmer or systems analyst to develop these. Again I have no problem with this diagram being included in a solution design.

Storyboarding is a technique that has been around for a while and has been adopted by the XP (extreme programming) agile programming technique. Use cases attempt to portray the same information.

The use of screen mock-ups helps show the user some tangible picture of what the application might look like and is an alternative to prototyping the user interface. There is a good IBM article on this.

There is little question that use cases have a bigger impact on software engineering than storyboards. Use cases are the origin and basis for a well-defined object model that not only manages the functionality and business rules of a system, but also serves more than one HCI [Human Computer Interface]

The Business Analyst(BA) develops requirements for two audiences. It is the BA's call whether to do screen mock-ups or storyboards. If the BA wants to develop these to convey to the user what the solution might look like then this should be commended. However, a BA still needs to develop the use cases for the developers to understand the logic behind the user interface.

I got a difficult email this week. It went something like "I have just found something new. Apparently five diagrams can replace our long business requirement documents. Please investigate and see if we can get rid of RUP". This is what prompted me to prepare this posting. In general I believe diagrams are goodness but we need words to be precise. Hugos would like to regress to a simpler time. UML is a standard and we use well-accepted standards to make our life simpler. We should not be slaves to our methodologies. Develop screen-shots or storyboards or prototypes if they will help the users understand their requirements. We need to remember that the developer will benefit by seeing a use-case in a format that he or she is comfortable with and expresses what happens behind the user interface.

References:

Two responses to the Hugos article:

Letter: UML Shifts Specs to User's Point of View
Bob Jones, October 15, 2007 (Computerworld)
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=304960

Opinion: Five Diagrams Redux
Bran Selic, an IBM distinguished engineer at IBM Canada, is co-chairman of the UML 2 task force in the OMG
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9043002&source=rss_topic73

The IBM article on Storyboarding

Form feeds function: The role of storyboards in requirements elicitation
http://www.ibm.com/developerworks/rational/library/dec05/krebs/index.html

SOA Governance

I listened to a good Podcast on SOA governance today. It was Phil Windley and Scott Lemon interviewing Tod Biske and Ed Vasquez of Momentum SI. It is clear to me that SOA governance is important in managing SOA and SOA metadata. This podcast provides some ideas on what shape this governance can take and how products may or may not assist with this.

SOA has often been compared with Object Oriented development or even structured programming. The question was asked in the Podcast about why Governance was such an issue for SOA in comparison to these earlier approches. The answer is that the implication of an SOA decision goes beyond the IT group and single applications. The users of the service are significant stakeholders in an SOA and are directly affected by the policies. Organisation partners or organisation divisions sharing services need to agree on a service approach and comply with service interfaces in order to provide effective reuse.

Todd Biske emphasised three key aspects of governance in relation to policies that need to in place. These three aspects can be assisted by automation. They were

• Enforcement of policies
• Creation of policies
• Storage of policies

Later in the discussion two more aspects of governance were provided:

• Communication of polices
• Feedback on policies (policies evolve)

A strong theme of the podcast was that you cannot buy governance. It has to be appropriate for the organisation and a product is not yet out there that provides integrated assistance over all the aspects of governance. The process of building governance occurs by assessing the current situation, working out what processes are involved and then providing the appropriate level of manual operations and automation to support these processes.

Storage of policies is provided by registry and repository products. Registry products are moving toward providing more repository functions. The term "Regository" was coined in this podcast. A parallel can be drawn with LDAP. The registry is becoming much more of a managed data set which is configurable and provides information to humans not just automated processes.

Feedback on policies and services is very important. Without proper feedback the organisation can become a lame duck. Portfolio management is important for services as well as for applications and projects. The discussion reminded me of getting feedback on this statistics for my blog site (using http://www.mybloglog.com/). This tells me where my links are coming from and also gives me a warm feeling that people actually see this stuff. The same thing is true of getting feedback on a website with a product like Webtrends. Services are no different. Metrics about who is using a service, when they are using it and how often, provide valuable feedback.

This feedback and the process of understanding the customers of your services gets the IT person closer to the business. It was suggested in the podcast that in general SOA involves IT doing more tasks previously understood to be business tasks. If you are not experiencing the pressure to do more work related to the business then you are not really doing SOA.

There can be hundreds of policies associated with SOA. Design time policies will include technology choices, product choices and methodologies. Run-time policies include your service contracts and service level agreements (SLAs). These SLAs may apply to each producer-consumer pair and include security policies, operational policies, compliance policies and integration policies.

SOA policy should not be a police state. The approach to implementing policy should be that "The path of following policy is the path of least resistance". An organisation should not reach a gridlock because there are so many policies that the developer cannot move. Policies are something that involves discussion with service users and should not be confined to the IT group.

Effective policy agreements will save time by avoiding the need to build using more than one technology (eg. SOAP, REST and JMS). The value of an agreement is that it saves the cost of building with multiple standards.

Greater governance enables the services in SOA to be more loosely coupled. Loose coupling is about reducing dependencies between services and making those dependencies explicit. The fact that the service contracts are written outside service code and are clearly defined is a key determinate of loose coupling.

Todd's two most important points were

1. Use a service or application portfolio technique to identify what services are good, bad, expensive or good value.
2. To get started on a project ask
   - What services dose this solution expose or use?
   - What business events does this solution publish or subscribe to?
   - What business processes do this solution support?

There were some good insights in this podcast. Governance in SOA is important. It may stop your SOA becoming JBOWS (Just a Bunch of Old Web Services). It makes sure that your organisation gets the best of reuse and does not get lost in mire of standards, products and methodologies.

Footnote:
JBOWS (or JaBOWS) was a term I heard this week in a talk from Glenn Smyth. There is a good post from Joe McKendrick on this acronym. Another reference can by found in an article by Rich Seeley which in turn refers to another artlicle by Brent Carlson which uses the acronym ABOS (A Bunch of Services). It would have been more accurate to write about ABOS in my posting but JBOWS has a nicer ring to it and seems more likely to catch on.

The Low-Down on Bottoms-Up

This is a completely trivial rant on the use of the phrase "bottoms-up" when I the correct term should be "bottom up". We all know what we mean when software is developed using a "bottom up" approach or that ideas in an organisation come from the "bottom up". "Bottoms-up" on the other hand is a friendly remark made before drinking an alcoholic beverage. It also can be a call to get out of your seat an do some hard work although the origin of the "Head down tail up" description of hard work is more apt for digging and cleaning than it is for desk work.

I agree with Jonathan Betz when he says

What's bugging me these days is that a lot of people are trying to describe Google culture as "bottoms up," when they actually mean "bottom up" Bottoms up means a whole different thing. So, to all you Googlers throwing around the phrase "bottoms up" like it's a good thing: please stop. It hurts my ears when you say it.

I also agree with Jonathan Nightingale

I thought it was a rather humourous typo - obviously someone had confused the drinking cheer with the development methodology on a hungover Monday morning and typed the wrong thing.

I first wrote about this in an earlier posting but I have heard it since on many occasions. I list a number of references below who fall into the "Bottoms Up" abyss. Amongst them Bill Gates is quoted a number of times using this phrase. I like to think that this was probably a journalistic mistake rather than a slip-up by the world's richest computer nerd. Bill Inmon, one of the founders of data warehousing also has a couple of bloggers accusing him of advocating a "Bottoms Up" approach but there seems enough who use the correct term to suggest that Mr Inmon himself did not advocate this phrase.

A "Bottoms Up" approach could be when the software developers go out for a drink and come back in to write some code. This would be a new approach to software quality. The phrase could refer to when everyone is tail up and is very busy but I do not believe this is the intent in most cases. Congratulations though to Chris Keyser who has his own definition for a "Bottoms Up" person suggesting some justification for using this phrase:

But I have definitely been there working with a few bottoms up people. That's the person that when I arrive at some moderately elegant system architecture diagram with some pretty pictures in PowerPoint and Visio catches the few key leaps of faith I made when I am doing my hand waving around my vague idea of implementation, and has saved my rear-end more than once.

Still more worrying is when "Bottoms Up" is used with the phrase "Tops Down". Now that does conjure up some interesting images. The best idea I found on this subject comes from a posting response:

Perhaps they have actually installed a free beer cooler somewhere at IBM, and
they really are encouraging bottoms-up development, and anticipate that enough
of this could lead to tops-down development?

References:

The Bill Gates Quotes
http://www.crn.com/software/183701538
Microsoft wants to make collaboration "simple and bottoms up" but also give users visibility into their important business information, Microsoft Chairman Bill Gates told attendees of the company's second annual Microsoft Office System Developers Conference in Redmond, Wash.
"We are adding new capabilities to make collaboration simple and bottoms-up, but also gives [users] visibility into business information. There are new capabilities in Office aimed at filling the skills gap," Gates said.
http://www.microsoft-watch.com/content/operating_systems/gates_pushes_power_to_the_people_message.html

The productivity benefits accrued by companies that aren't afraid to back "bottoms-up empowerment" was a key theme for Gates during his hour-long address, attended by CEOs from Barnes & Noble, Berkshire-Hathaway, Dell, Delta, Fanny Mae, Hewlett-Packard, Home Depot and other Fortune 1000 firms from around the world.

(and the next one contains "Tops-Down")
http://www.microsoft.com/Presspass/exec/billg/speeches/2007/02-20BillGatesITACtranscript.mspx

The consumer Web brought in, with products like SharePoint, into an internal company environment, so you have bottoms-up creativity but no discontinuity as you're taking that bottoms-up creativity and connecting it with sort of tops-down financial systems because the kind of parts and information out of those systems can easily be imported into any of these engines or portals. We just pick a template and go. We don't have to call IT or think about it, and yet the audit trails and the rights management capabilities built into these things are what's appropriate now for this very digital environment.

The Bill Inmon Attributions
http://www.netcrucible.com/blog/2004/04/09/death-of-the-data-warehouse/

In fact, I remember Bill Inmon (the father of Data Warehousing) fighting against the spread of a particular concept of ?data mart?, since he felt it would give people the impression that data warehousing could be done in a bottoms-up fashion.

http://searchdatamanagement.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid91_gci1173583,00.html
You'll have two basic decisions to make and numerous follow-on decisions:
Enterprise vs. Data Mart-Oriented Architecture
Enterprise vs. Bottoms-up Oriented Methodology

Other Miscellaneous Quotes
http://www.ebizq.net/blogs/firstlook/2007/06/scalable_wikis_made_easy_by_mi.php
"This is a bottoms-up form of communication as opposed to a much more traditional centralized expert driven-down sort of communication," he says.

http://lxer.com/module/newswire/view/84664/index.html
Raising Linux Awareness: A "Bottoms-Up" Approach

http://duckdown.blogspot.com/2007/07/enterprise-architecture-top-down-or.html
(This is a rare blemish for James whose blog on Enterprise Architecture is well worth reading).

One of the more interesting things is that a person debating top-down vs bottoms-up needs to take into consideration is historically which one has had better results. Of course, top-down has the advantage that even mediocre delivery will be declared success while bottoms-up mediocrity will probably result in throwing daggers. If we ignore the whole perception is reality crowd for a moment and truly think about success, then I think our conclusions would be different.

http://www.uxcomm.com/corporate_news_ar_2007-01-09a.asp
In contrast, the "bottoms up" approach is characterized by the use of management software that vendors typically ship with their devices, or which is put in play when an Operating System (OS) is loaded onto a device.

http://www.worcestergroupinc.com/blog/index.php/2006/11/24/bottoms-up-or-top-down-the-eternal-business-intelligence-question/
[ # ] Bottoms Up or Top Down? The Eternal Business Intelligence Question

http://news.zdnet.com/2100-3513_22-6087566.html
Significantly, IBM chose not to define and then build a large-scale, sophisticated knowledge management system. Instead, the company is taking a bottoms-up approach, allowing contributors to have a more active hand in how collaborative work is organized.

http://marxsoftware.blogspot.com/2007/10/highly-useful-jax-ws-introductions.html
I have also followed with interest the debate about web services development with JAX-WS with a bottoms-up approach (annotation Java classes and generating appropriate artifacts) versus with a top-down approach (starting with WSDL).

http://weblog.halmacomber.com/2003_05_04_archive.html
Not only is LISP suitable to this style of development, I'd dare say the language anticipates a bottoms-up collaborative approach. Not all software development is collaborative and bottoms-up, just like not all architecture and engineering is collaborative and bottoms-up. The language helps.

Does SOA Rate a Mention in the Strategic Plan?

I was asked to draft a contribution to the IT Strategic Plan last week. My organization is a big public organization and the IT Strategic Plan is a glossy public document that has to be more of a marketing document than a technical plan. I wanted to put something in about SOA but I could not rely on reaching a technical readership or a readership that new much about the organization's IT at all.

This raised my curiosity. How many organizations actually put something about SOA in their IT strategic plan, let alone a corporate strategic plan?

Many commentators suggest SOA is a strategic imperative. For instance David Linthicum suggests "when thinking about SOA a long term strategic plan is best". Christopher Koch goes further and suggests that IT will assume responsibility for business innovation.

The ongoing shift to service-oriented architecture (SOA) will improve IT’s
ability to innovate because it requires that IT understand how the business does
its work. SOA represents the highest order of business process innovation

Surely SOA is of such central importance that it is going to be central to most IT Strategic Plans.

If I look to Forrester Research for some help with the IT Strategic plan I am advised that

IT's strategic plan is an essential tool to run IT like a business. The strategic plan for today's IT is different from the strategic plans that IT may have developed five years ago. It is purpose-driven and a complement to IT governance structures and processes.

Unfortunately SOA does not rate a menton in this article. Enterprise Architecture and Project Portfolio Management however bother get a Guernsey.

I spent some time with Google and found no strategic plans containing SOA and so I decided to look at some IT Strategic plans. Some of the findings are detailed at the end of this posting.

I found four IT strategic plans that put SOA as fairly a central goal or support for the plan. I found four where there was a mention of SOA but not a significant emphasis on it. After not finding as many as I had hoped I thought that perhaps when people write about SOA for the consumption of the layman they use other terms to make the document more accessible. So I looked for mention of 'agility' or 'reuse' in IT Strategic Plans. This I thought is a way for IT people to speak in code about SOA without mentioning those three letters. I found four of those too, although the connection with SOA was not all that obvious in these documents. I have listed nine IT Strategic plans with no mention or allusion to SOA.

This is not a thorough or unbiased study. The people publishing the plans on the web seem to be government and tertiary education groups. This happens to be my interest. The list is not a random sample and I could have missed some important references within the documents, however I have to conclude that SOA is not getting into as many IT Strategic plans as it should, given its strategic importance.

I am left to hypothesize on the reasons for this:

• SOA is not strategically that important
• SOA is only considered a software development issue and is not considered to relate to other IT issues infrastructure, support services or IT procurement
• SOA is too hard to explain to the non-IT person
• SOA is not as business-centric as some SOA authors would have us believe
• SOA is too new to warrant mention in an IT strategic plan
• SOA is too old and like RDBMS everyone accepts it and will move on without having to bring it up in a strategic plan
• The SOA message has not reached those who write IT strategic plans
• There is little marketing value in SOA

This is a reality check for those espousing the strategic importance of SOA but I would not go as far as those who suggest SOA is only for tactical purposes. Whatever the reasons, I will still push for it to be accepted at the strategic level.

In my contribution to my organization's IT Strategic Plan I started with a business definition such as one of the earlier definitions discuss in my earlier postings on the definition of SOA.

I then went through some for the motivations behind SOA and expanded on some of these in reference to the business of my organization. These motivations were Agility; Reuse; Assist Transition from Legacy; Federation; Extensibility; Manage incremental Change; Better alignment between IT and Business Processes; Intrinsic Interoperability; and Vendor Diversity.

This ended up a fairly long description, so I expect most of what I wrote will end up being cut by the editor of the strategic plan. Whether SOA gets edited out of the IT strategic plan is not important as long as the internal focus is on getting the pieces in place to make SOA happen and we can get some support from the business for this. Perhaps in this process I will encounter some of the reasons that SOA is not so prevelant in strategic plans.


IT Strategic Plans with SOA as central or a core goal

Oxford University, UK:
http://www.ict.ox.ac.uk/strategy/plan/plan.xml.ID=S4

The Global Justice Information Sharing Initiative (Global) Advisory Committee
(GAC) (an advisory body to the Assistant Attorney General, Office of
Justice Programs (OJP), and the U.S. Attorney General)
http://www.it.ojp.gov/documents/200409_GAC_Strategic_Plan.pdf

The Commonwealth of Massachusetts
http://www.mass.gov/Aitd/docs/itdstrategicplan.doc

SURF (A Nederlands higher education foundation) http://www.surffoundation.nl/eng/download/SURF-Strategic-Plan_2007-2010.pdf

IT Strategic Plans where SOA rates a minor mention

Carnegie-Mellon University
http://www.cmu.edu/computing/about/2006StrategicPlan.pdf

State of Texas Department of Information Resources
http://www.dir.state.tx.us/pubs/asp2006/asp2006_abbreviations.htm

The University of Western Australia, Library
http://www.library.uwa.edu.au/__data/assets/pdf_file/11068/Scanning_the_environment_2007_NE.pdf

National Labor Relations Board, USA
http://www.nlrb.gov/nlrb/shared_files/reports/FY2007_StrategicPlan_FINAL.pdf

IT Strategic Plans with Some mention of Agility and Reuse but not SOA

State of Wisconsin:
http://enterprise.state.wi.us/home/StratPlan/enterpriseITplan.pdf

Commonwealth of Virginia
http://www.vita.virginia.gov/uploadedFiles/Library/StakeholderWorkshopResults.pdf

University of Iowa:
http://cio.uiowa.edu/strategicplan/

Department of Family and Community Services, Australia
http://www.facsia.gov.au/internet/facsinternet.nsf/via/ict/$file/FaCS_ICT_Strategic_Plan_2004-07.pdf

IT Strategic Plans where SOA rates no mention
(There actually were many but these are examples)

City of Winston-Salem:
http://www.cityofws.org/Assets/CityOfWS/Documents/forms%20and%20reports/Information%20Systems/IT%20Strategic%20Plan.pdf

State of Maine:
http://www.maine.gov/oit/strategic/SITP.doc

US State Department:
http://www.state.gov/m/irm/rls/c13461.htm

Louisiana State University:
http://www.lsu.edu/departments/its/ocio/FITS/fits.html

State of South Carolina:
http://www.cio.sc.gov/ITPlanning/StateStrategicPlan.pdf

New York State:
http://www.cio.state.ny.us/NYS%20IT%20Strategic%20Plan/NYSITStrategicPlan_20Jun2006.pdf

OHIO State University
http://cio.osu.edu/planit/refresh07.html

Dept of Justice, State of Victoria, Australia
http://www.justice.vic.gov.au/wps/wcm/connect/DOJ+Internet/Home/About+Us/Our+Goals/JUSTICE+-+Department+of+Justice+-+Information+Technology+Strategic+Plan+2002-05

Electoral Commission of Qld, Australia
http://www.ecq.qld.gov.au/data/portal/00000005/content/60401001035252002415.pdf

Concluding Conference Comments

This is the final of a series of postings on an SOA conference presentation I gave for the Ark Group in Sydney. I seemed to have a great deal of material which probably explains why I ran short of time. The topics I covered with the relevant postings linked were:

• The definition of SOA
• The motivation behind SOA
• Legacy Transition
• My organisation's software stack
• The link between SOA and Enterprise Architecture
• The metadata required for SOA
• SOA and Metadata: Where you can Put It

Of course, there were other speakers as well, and they were all highly competent and informative. I summarized these in an earlier posting.

To wrap this up series up I will go through my conclusions. Firstly, the changes we need to make to support SOA in an organization involve:
• New Components, New Tools, New Skills
• New Application/Service Architecture
• Governance around reuse and business process
• Less emphasis on applications, more governance at the service level
• Focus on interfaces and contracts particularly in the information model
• More Metadata
• Changes to methodology

New Components, New Tools, New Skills
When we talk about changes it is important to understand the baseline from which you are changing. Some organisations will have well developed skills in distributed portal-based applications. My organisation is yet to deploy its portal product and is still developing skills outside of the mainframe environment. We are still building our skills in Unix and Java. We are still putting together our development environment. None of these components, tools or skills are peculiar to SOA. What we have done specifically to implement SOA is to purchase the BEA Weblogic platform and we are looking at ways of changing our development approach. We find we need to quickly ramp up skills in our new environment as well as SOA Development, SOA Testing and management of metadata.

New Application/Service Architecture
There was a critical cell in the Zachman Framework called “Application Architecture”. This could be relabelled “Service Architecture”. It is services that now should be the focus of our attention. This is working at a different granularity.

Governance around reuse and business process
Governance is required around reuse of services and reuse of artefacts. There is more work in the Enterprise Architecture (EA) space than there is for traditional application-centric development. This should be balanced by relatively less work in the solution architecture space. Modelling Business Processes is more important with SOA.

Less emphasis on applications, more governance at the service level
Application Development is more focused on assembly of services and the development of a presentation layer so there is less emphasis on applications themselves. There will need to be governance changes to introduce SOA. There needs to be stronger EA governance. Now you not only have to find an owner and sponsor of an application. You may have to find and owner or sponsor of a service. This may be easier than finding owner for monolithic application in some cases (and in other cases it may not).

Focus on interfaces and contracts particularly in the information model
The modelling of persistent data store won’t change but in addition we need to model the interfaces and contracts between services. This will be required to drive service development.

More Metadata
All this means we have more enterprise metadata. This presents challenges for managing this metadata. This was the subject of an earlier posting.

Changes to methodology
Software development methodology includes more work on interface, contracts and SLAs. The development of testing services presents a challenge. This is because you may have to test without having and application to test it with. You have to test on behalf of applications that may use it but do not exist yet.


I make the following architecture recommendations to facilitate SOA:

• Have an Enterprise Architecture
• Manage the application portfolio
• Drive the enterprise from business processes or from events
• Apply good governance to services and metadata
• Data modeling should include interface/contract design

SOA must be driven from and EA. It is not something that can be put in place for a single application. The benefits of SOA will not accumulate until you have a number of applications completed.

When you build a service you build it for all the applications that potentially use it. Therefore you have to know about these applications. My organisation has a centralised IT department. This helps enable both EA and SOA. A federated model can work but someone still needs a clear picture of the application portfolio and needs to define interfaces and contracts of your services.

Business Processes will drive SOA. This is to ensure the services have meaning to the business and that they can be combined into processes in the same way that business combines its tasks into processes.

The Governance changes in SOA. I’ve talked about the governance of services and the governance of metadata. Both are important for successful SOA.

Once the business process is sorted out the interface and contract between the services must be modelled. This will then drive the development of the services. This information needs to be kept up-to-date and is the key metadata required for reusing the service.


Some of the points brought up at the conference interactive session were true of most software development:

• You need good business sponsorship
• Choose your first SOA projects carefully
• If the business does not see the need then it's not going to work
• Talk to the business in language it understands
• Get executive support of your SOA program

So my concluding recommendation is not to lose sight of what has worked in the past. A lot of what makes conventional development work, will also help make SOA work.

Is Your Enterprise Architecture SOA Compliant?

In August I wrote about the Zachman framework and SOA. This was the basis for a couple more slides at the Ark Group conference on interoperability in which I was a presenter. I have covered material from the presentation in couple for previous postings, "Live up to the Legacy that was Left" and "Why SOA?". I will focus on the main theme of my presentation in this posting which was "Ensuring your Enterprise Architecture is SOA Compliant"

Firstly I would like to point out some work that is being done by the SOA Consortium along these lines. They have announced the Enterprise Architecture 2010 Working Group which is to define the high-level roles and deliverables of the Enterprise Architecture team. There is not much tangible yet from this group as it has just been set up but these people also brought us the SOA practitioners' guides so this could prove a valuable initiative.

At the conference mention in the first paragraph I went over the Zachman Framework as I have done in my earlier posting. One thing I pointed out is that different software development approaches can be described in terms of the framework. I then went on a described my organisations use of the Zachman framework using he colour codes I added to the Zachman grid.

When we talk about top-down approaches conveniently you work from the top row (Scope Row) down to the bottom row (Detailed Representation). Similarly when you work bottom-up you work from the bottom of the framework representation.

Assuming that most good software development is done top-down (and I am subscriber to this approach) you can choose different methodologies depending on the column you choose.

If you drive down the 'Data' column you have a Data Driven Architecture. The Enterprise Information Model in this approach is all important. The other cells of the framework will either support the data column or be influenced by it.

The SOA approach is usually focused on business process and this means it is being driven by the 'Function' column. Software development can be driven from Business Process Modelling and 'Function' column without being SOA.

If we are two focused driving our software development from the 'Network' column we may end up with a siloed application based on organisational locations. This generally software development anti-pattern as modern enterprise software should be independent of location.

Similarly, too much focus on the 'People' column can lead to application focussing around individual or organisational units. This is a very common siloed approach and anti-pattern. More positively a people driven architecture will look at the user experience. This is a common approach for smaller applications to look at screen designs, report layouts and user interaction scenarios that will meet user requirements. For enterprise application though it is important to look at the bigger picture, to question what the organisation is trying to achieve and to model the process from start to finish otherwise the people siloed anti-pattern may result.

If we drive down the 'Time' column we will end up with an Event Driven Architecture (EDA). This is an alternate approach to SOA. We look at events and how they initiate and affect business processes. The 'Function' columns and 'Time' columns can be worked with in tandem to produce and event driven SOA application.

If we work down the 'Motivation' column of the Zachman frame work then we are using a Business Rules Approach. My organisation has not made great use of this framework column. For more information about the Business Rules Approach see http://www.businessrulesgroup.org/ . Also this column is valuable for expert systems and decision management.

With the Zachman framework you do not have to fill every cell. You certainly do not only have one enterprise model in each cell. How you uses the Zachman framework is important. So I will describe how our organisation uses it in terms of the diagram above.

The yellow cells in the above diagram are areas where we have not got any significant documentation.

The green cells contain documentation the business has prepared with out prodding from the IT department. The brown cells are documented by IT but with the interaction of the business.

The top two layers do not change much. These are business level documents. Producing good business process models is more important with SOA. The business process is the key in the SOA process not the application.

The blue is driven by SOA. It starts with your Application Architecture. Distributed Architecture works differently with SOA. You might consider and ESB for instance which will give a different architecture than if you use point-to-point communication. These upper two blue cells drive changes at the more detail level below them.

The pink areas are affected by SOA. Data modelling and Human interface architecture are not radically different but they have to take into account the new architecture.

I will now look at the link between EA and SOA in a different way.

Zapthink has a good article on Zachman and SOA where they use arrows emanating from the Application Architecture cell of the framework. I prefer to represent the influence direction as predominately down representing a top-down approach.

We will start in the 'Data' column again. I do not think the Information Model should be different as a result of SOA. However the Logical Data Model probably should. It should document the interfaces and be able to transform the representation in XML. It should also document the persistent data stores. You now have at least two levels of logical data modelling.

The physical data model and data definition can then be XML, XSD and WSDL. At the same time you have your modelling of persistent data which would be your normal Class Model or ER diagram. This data model would eventually be represented in data dictionaries and database schemas.

We will move across to the 'Function' column now. The Business Process Model should influence your application architecture. Application Architecture should advise you of what services you need and what services to reuse. A service becomes part of the enterprise architecture because it is built for reuse. A new service is also part of a solution architecture. The service has to meet both solution and enterprise needs. Also in this cell is your application stack which supports the service model. The service model then drives the more detailed System Design and the programming.

SOA is a distributed architecture so the lower rows of the 'Network' column takes on increased significance compared with traditional software development approaches. There is more flexibility about where you do your processing. This needs to be built into your Distributed System Architecture.

Business logistics (and particularly whether you are dealing with agencies external to the organisation) will impact your applications architecture. There is a stronger linkage from Application Architecture to Distributed System Architecture because SOA is an enabling technology. It enables you to distribute processing, to look at load balancing and to look at interfaces with other agencies. There is therefore a strong influence on distributed architecture increasing the importance of good architecture in this space.

As with the other columns the influence downward is strong. Your Technology Architecture and Network Architecture needs to be built to cope with the new Distributed Systems Architecture.

Human interface architecture in the 'People' column may change to suit SOA but it does not have to. Some interface technologies work better with SOA. Some ways of interacting with an application work better with SOA but if you want to drive it the other way by specifying your Human Interface Architecture first then this can also work.
Security Architecture gets its own arrow. It is not a logical progression from Presentation Architecture. You may have to do security differently under SOA. You may have to make better use of a standard infrastructure. You may have to use SOA security standards.

SOA for my organisation is business process driven although there is still some bottom up development resulting from services based on mainframe data entities. The focus on reuse and the question for what we already have available to reuse demands some bottom-up consideration.

To be SOA compliant your architecture needs to include:

• Documented business processes,
• Enough bottom-up analysis to reuse of existing services, and
• Data modeling of web service interfaces.

This is in addition to SOA infrastructure and distributed models required the lower levels of your architecture. SOA is an Enterprise Architecture approach. You cannot have an effective SOA unless you have some level of Enterprise Architecture. If you focus on a single solution or application and do not look at the wider needs of the enterprise then you are going to miss the major benefits of SOA.

The business process focus and implications around reuse mean that the business needs to be aware of the SOA approach. It is a change to business practice that the SOA consortium is looking at in its Enterprise Architecture 2010 Working Group.

Live up to the Legacy that was Left …

I presented at a conference last week. After discussing my organisation I provided my definition for SOA and a bit about why my organisation was interested in SOA. I then got onto the topic of this posting which is our legacy transition and in particular how we were going to use SOA to move away from mainframe technologies.

First I presented an age profile of our applications which showed many of our more strategic systems were over 20 years old. Our more integrated systems are written using the IDMS database and ADSO/COBOL. Our more siloed applications are written using the ADABAS database and the NATURAL language.

The mainframe is costly and inflexible environment. It is hard to find developers who know IDMS and the software vendors don’t provide much support for this database technology any more. It is not as agile as a relational database management system. The same is true for our ADABAS applications. It is difficult to find skills. There is at least more of an upgrade path provided by Software AG to move onto newer technologies.

I presented a slide of a plan that was developed more than 4 year ago. It is illustrated here.

The above slide depicts moving from a mainframe environment to a transition stage consisting of an environment where we have some mainframe applications and some J2EE applications accessing mainframe data. The business logic layer is going to be moved off the mainframe leaving applications accessing mainframe data using SQL adapters. The next step is that we move completely off our legacy databases onto relational databases.

This is still the plan. We have made some progress, but progress is slow.

On the right hand side we had n-tier J2EE applications. Now we hope to target a Services Oriented Architecture. This will give us good integration to other agencies and good integration to national bodies. It will assist us to break up the legacy applications and migrate bits of them to newer technologies. It will assist SAPOL to be more agile in its development of software changes.

There is a flaw in this diagram though. This is meant to be a gradual process. We are not going to rewrite a whole suite of applications quickly. We do not have a big budget to break the back of this work. We do not want to risk a huge project that takes years to implement.

Where is the business logic in the old mainframe environment? It is on the mainframe.

Where is the business logic in the co-existence? It is all in the newer technology (blue) areas. What we are finding is that there is so much we have to move to the newer area before we have our first significant application. The step to the transition stage is too big and too risky.

Version 2 of our transition to legacy is illustrated here. Here we have repeated the transition stage and the new environment stage. There is not much difference in the New Environment stage. Except we have encapsulated our business logic and data access in services.

The main difference in the transition stage s that Business Services that are depicted as sitting on the mainframe.

The hope is that we can feasibly wrap up all the mainframe logic in services and reorganise it in custom written business services and combine it with services already written in the new environment.

We would have to make some compromises about what these services might look like, and this would drive a bottom up process in the lower levels of the architecture. But this means that we gradually move our business services over from mainframe to the new environment without having to write huge amounts of business logic at once.

This is done at present using some custom built techniques that duplicates some of the business logic in the mainframe. This is probably not sustainable for rewriting all of the mainframe business logic. Another solution is to encapsulate with more automation. This can be done according to vendors (Neon and iWay). We can encapsulate our existing green screen forms into services without much fuss according the vendors. In actual fact these service adapters cost an arm and a leg and installation and configuration is agony, but that is a familiar story.

The full quote in the title of this posting is

It is up to us to live up to the legacy that was left for us, and to leave a legacy that is worthy of our children and of future generations.

We look forward to making the most of a legacy environment and providing something better for those that follow us.

Reference:
The above quote is from Christine Gregoire, Governor of Washington State, USA in her Inaugural Address, January 12, 2005