SOA Governance

I listened to a good Podcast on SOA governance today. It was Phil Windley and Scott Lemon interviewing Tod Biske and Ed Vasquez of Momentum SI. It is clear to me that SOA governance is important in managing SOA and SOA metadata. This podcast provides some ideas on what shape this governance can take and how products may or may not assist with this.

SOA has often been compared with Object Oriented development or even structured programming. The question was asked in the Podcast about why Governance was such an issue for SOA in comparison to these earlier approches. The answer is that the implication of an SOA decision goes beyond the IT group and single applications. The users of the service are significant stakeholders in an SOA and are directly affected by the policies. Organisation partners or organisation divisions sharing services need to agree on a service approach and comply with service interfaces in order to provide effective reuse.

Todd Biske emphasised three key aspects of governance in relation to policies that need to in place. These three aspects can be assisted by automation. They were

• Enforcement of policies
• Creation of policies
• Storage of policies

Later in the discussion two more aspects of governance were provided:

• Communication of polices
• Feedback on policies (policies evolve)

A strong theme of the podcast was that you cannot buy governance. It has to be appropriate for the organisation and a product is not yet out there that provides integrated assistance over all the aspects of governance. The process of building governance occurs by assessing the current situation, working out what processes are involved and then providing the appropriate level of manual operations and automation to support these processes.

Storage of policies is provided by registry and repository products. Registry products are moving toward providing more repository functions. The term "Regository" was coined in this podcast. A parallel can be drawn with LDAP. The registry is becoming much more of a managed data set which is configurable and provides information to humans not just automated processes.

Feedback on policies and services is very important. Without proper feedback the organisation can become a lame duck. Portfolio management is important for services as well as for applications and projects. The discussion reminded me of getting feedback on this statistics for my blog site (using http://www.mybloglog.com/). This tells me where my links are coming from and also gives me a warm feeling that people actually see this stuff. The same thing is true of getting feedback on a website with a product like Webtrends. Services are no different. Metrics about who is using a service, when they are using it and how often, provide valuable feedback.

This feedback and the process of understanding the customers of your services gets the IT person closer to the business. It was suggested in the podcast that in general SOA involves IT doing more tasks previously understood to be business tasks. If you are not experiencing the pressure to do more work related to the business then you are not really doing SOA.

There can be hundreds of policies associated with SOA. Design time policies will include technology choices, product choices and methodologies. Run-time policies include your service contracts and service level agreements (SLAs). These SLAs may apply to each producer-consumer pair and include security policies, operational policies, compliance policies and integration policies.

SOA policy should not be a police state. The approach to implementing policy should be that "The path of following policy is the path of least resistance". An organisation should not reach a gridlock because there are so many policies that the developer cannot move. Policies are something that involves discussion with service users and should not be confined to the IT group.

Effective policy agreements will save time by avoiding the need to build using more than one technology (eg. SOAP, REST and JMS). The value of an agreement is that it saves the cost of building with multiple standards.

Greater governance enables the services in SOA to be more loosely coupled. Loose coupling is about reducing dependencies between services and making those dependencies explicit. The fact that the service contracts are written outside service code and are clearly defined is a key determinate of loose coupling.

Todd's two most important points were

1. Use a service or application portfolio technique to identify what services are good, bad, expensive or good value.
2. To get started on a project ask
   - What services dose this solution expose or use?
   - What business events does this solution publish or subscribe to?
   - What business processes do this solution support?

There were some good insights in this podcast. Governance in SOA is important. It may stop your SOA becoming JBOWS (Just a Bunch of Old Web Services). It makes sure that your organisation gets the best of reuse and does not get lost in mire of standards, products and methodologies.

Footnote:
JBOWS (or JaBOWS) was a term I heard this week in a talk from Glenn Smyth. There is a good post from Joe McKendrick on this acronym. Another reference can by found in an article by Rich Seeley which in turn refers to another artlicle by Brent Carlson which uses the acronym ABOS (A Bunch of Services). It would have been more accurate to write about ABOS in my posting but JBOWS has a nicer ring to it and seems more likely to catch on.

Concluding Conference Comments

This is the final of a series of postings on an SOA conference presentation I gave for the Ark Group in Sydney. I seemed to have a great deal of material which probably explains why I ran short of time. The topics I covered with the relevant postings linked were:

• The definition of SOA
• The motivation behind SOA
• Legacy Transition
• My organisation's software stack
• The link between SOA and Enterprise Architecture
• The metadata required for SOA
• SOA and Metadata: Where you can Put It

Of course, there were other speakers as well, and they were all highly competent and informative. I summarized these in an earlier posting.

To wrap this up series up I will go through my conclusions. Firstly, the changes we need to make to support SOA in an organization involve:
• New Components, New Tools, New Skills
• New Application/Service Architecture
• Governance around reuse and business process
• Less emphasis on applications, more governance at the service level
• Focus on interfaces and contracts particularly in the information model
• More Metadata
• Changes to methodology

New Components, New Tools, New Skills
When we talk about changes it is important to understand the baseline from which you are changing. Some organisations will have well developed skills in distributed portal-based applications. My organisation is yet to deploy its portal product and is still developing skills outside of the mainframe environment. We are still building our skills in Unix and Java. We are still putting together our development environment. None of these components, tools or skills are peculiar to SOA. What we have done specifically to implement SOA is to purchase the BEA Weblogic platform and we are looking at ways of changing our development approach. We find we need to quickly ramp up skills in our new environment as well as SOA Development, SOA Testing and management of metadata.

New Application/Service Architecture
There was a critical cell in the Zachman Framework called “Application Architecture”. This could be relabelled “Service Architecture”. It is services that now should be the focus of our attention. This is working at a different granularity.

Governance around reuse and business process
Governance is required around reuse of services and reuse of artefacts. There is more work in the Enterprise Architecture (EA) space than there is for traditional application-centric development. This should be balanced by relatively less work in the solution architecture space. Modelling Business Processes is more important with SOA.

Less emphasis on applications, more governance at the service level
Application Development is more focused on assembly of services and the development of a presentation layer so there is less emphasis on applications themselves. There will need to be governance changes to introduce SOA. There needs to be stronger EA governance. Now you not only have to find an owner and sponsor of an application. You may have to find and owner or sponsor of a service. This may be easier than finding owner for monolithic application in some cases (and in other cases it may not).

Focus on interfaces and contracts particularly in the information model
The modelling of persistent data store won’t change but in addition we need to model the interfaces and contracts between services. This will be required to drive service development.

More Metadata
All this means we have more enterprise metadata. This presents challenges for managing this metadata. This was the subject of an earlier posting.

Changes to methodology
Software development methodology includes more work on interface, contracts and SLAs. The development of testing services presents a challenge. This is because you may have to test without having and application to test it with. You have to test on behalf of applications that may use it but do not exist yet.


I make the following architecture recommendations to facilitate SOA:

• Have an Enterprise Architecture
• Manage the application portfolio
• Drive the enterprise from business processes or from events
• Apply good governance to services and metadata
• Data modeling should include interface/contract design

SOA must be driven from and EA. It is not something that can be put in place for a single application. The benefits of SOA will not accumulate until you have a number of applications completed.

When you build a service you build it for all the applications that potentially use it. Therefore you have to know about these applications. My organisation has a centralised IT department. This helps enable both EA and SOA. A federated model can work but someone still needs a clear picture of the application portfolio and needs to define interfaces and contracts of your services.

Business Processes will drive SOA. This is to ensure the services have meaning to the business and that they can be combined into processes in the same way that business combines its tasks into processes.

The Governance changes in SOA. I’ve talked about the governance of services and the governance of metadata. Both are important for successful SOA.

Once the business process is sorted out the interface and contract between the services must be modelled. This will then drive the development of the services. This information needs to be kept up-to-date and is the key metadata required for reusing the service.


Some of the points brought up at the conference interactive session were true of most software development:

• You need good business sponsorship
• Choose your first SOA projects carefully
• If the business does not see the need then it's not going to work
• Talk to the business in language it understands
• Get executive support of your SOA program

So my concluding recommendation is not to lose sight of what has worked in the past. A lot of what makes conventional development work, will also help make SOA work.

Conference Summary Served with Spaghetti

I presented at a conference “Achieving Interoperability in Systems Architecture” on Wednesday. The occasional get together of architects, managers and developers is valuable. You can grab the odd new insight and find new approaches but it is also good to see others attacking problems in the same way and validating your own approaches.

After the conference I thought I would finish this summary on the aeroplane home. I have always wondered how to spend the time on long flights profitably and I admired a couple of passengers using laptops on my previous flight. Unfortunately as I try to use the laptop in flight I am in a middle seat with no elbow room on either side, and now I am served dinner in middle of the flight. I am juggling my laptop and the worst Spaghetti Bolognaise I have ever tasted. Still, here is the summary.

Josh Graham of Thoughtworks kept time well and came up with the comment “Why should I expect straight lines on my Business Process Model when business is inherently messy.” He also introduced one of our New Zealand guests as coming from the land of the “Wrong White Crowd” which sounds like a political comment and spoonerism wrapped up into one.

Mark Murphy of Tower Aust Ltd advocated we start small with SOA. He quoted IDC showing growth in SOA from $3.5bn in 2005 to $33bn estimated in 2010. Mark looked at some SOA definitions and aptly described the current situation where people build silo applications and then expect IT to join them together. He showed a couple of models from Sun and Microsoft showing SOA models that illustrated logical SOA.

Stephen Smith of Arcbok presented a simple framework for assessing the readiness of an organization for SOA. The framework consisted of assessing an organization on the dimensions of Leadership & Culture, Business, Technology, Knowledge, Funding and Sustainability. Each dimension has a number of questions to ask about an organization to asses its readiness. The sustainability dimension described whether the organization had the appropriate support, operations and governance in place for sustain the SOA. This also could include appropriate operational monitoring, measuring and standards. A readiness assessment tool is about risk management. Beware vendor maturity assessments that are used as tools to sell more services.

Krist Davood of AGL and Sensis has achieved something that everyone in the room envied. He had success justified SOA as an initiative using fairly hard-nosed accounting principles. He had justified SOA because of its ability to avoid periodic failures in the business because of the current way of doing things. He had been able cost these failures and justify the project to introduce SOA into the organization. SOA as a strategy to eliminate risk had been demonstrated to be effective.

Centrelink is a huge SOA operation and it was good to hear from Rob Doughty who is their Applications Architect. I liked his characterization of users as “Human Task Delegates” in relation to workflow applications. It was interesting for me to hear about the legacy system challenges that Centrelink have, but also how their portal work in benefiting other government departments. The focus of the talk was the deployment of a portal product which they will be basing all their systems on. I liked the comment that “Portals bring people and process together”. Rob talked about the challenge of working with various IT groups to deliver on a particular pattern of use in relation to the portal. He advocated the use of an “Integration Competency Centre”.

I then presented on Enterprise Architecture and its links to metadata and SOA. Some of this material is already on my posting on SOA Definitions, SOA Metadata and the Zachman Framework. I will elaborate in a future posting.

John Fisher from the NZ Ministry of Education showed us some great business modelling techniques that had found favour with his users. The technique involved Business Element Models, Business Component Models, Business Role Responsibilities and Business Service Models. The models were focussed on producing requirements for widely federated (eg. For 2,700 schools) SOA systems. He introduced the analogy of using power adapter so could use a drill brought in the UK to explain SOA to users.

Jan McConchie of SA Government ICT Services provided an overview of the challenges that she has to bring together the South Australian Government Services to present a common face to the South Australian citizen. This was interesting because the SA Government has established and Office of the CIO and this is showing the desire to coordinate and rationalise government IT activities. As one of the agencies involved, my organisation will be affected by standardisation and IT governance changes. Jan drew on research by Gartner and Saugutuck Technology to explain what SOA meant to her and discussed some initiatives in place for the SA Government.

Chris Howard was from the State Revenue Office of Victoria. Chris focused on risk and SOA and quoted Gartner, advising that it would take 3-4 years before investments in SOA payed off. He discussed a long term plan for legacy transition which was underpinned by full and frank statements about costs, benefits and risks with his business. Chris endorsed ITIL as an enabler of SOA.

Josh ran an interactive session where we discussed the issues of SOA and getting the business on board. It was never clear what you will find when you "turn over rocks" in the analysis process. We discussed what services to build first. It was not always best to do the low risk, high value project first. In the end it might just be business or legislative imperatives that help you make the decision. It was difficult to decide when to do those cross-cutting "hygiene" services.

George Cascales from Integral Energy discussed how he was implementing SOA in an out-sourced environment. He had a small group of architects and had succeeded in being able to develop new applications quickly using reused services. His group focused on standards and governance rather than the actual development. A set of tools and standards would be given to contract developers to develop the services and applications.

Rajat Chopra, the international speaker from Bell Canada provided a good description of how Bell embraced SOA. It was important for Bell to provide common services to serve its various customer channels for selling telecommunication facilities. Rajat provided us with a number of analogies for explaining SOA to management including Lego blocks, a car factory and diet. With diets, you can latch onto gimmicks but in the end it is lifestyle change that makes the difference. This is the same with SOA.

Alex Jouravlev representing the ACS presented twice. Alex’s first presentation was titled Prepare IT and Business for SOA. He traces the origins of SOA back to 1988. He made the point that our business cases do not really engage the business and we need to strive for simple messages using language the business understands to achieve a ‘mental ownership’. Alex supported the model driven architecture approaches but not for the reasons that they were developed. Business requirements should be developed with referencing any ‘system’ and should be modelled with Visual UML Models. Alex noted that several modelling practices that were usually acknowledged as important were often missed. These include Conceptual Business Modelling, Conceptual Data Modelling, Business Object Modelling and producing an Enterprise Ontology.

Later in the day Alex was able to present a model based on CMMI maturity model in an easy-to-understand fashion with the purpose of moving the SOA objective from IT to the business as an organisation obtains more SOA maturity. Alex was able to give us a realistic view of the limitations that would be expected at each level of maturity and provide an insight into how the role of key players like the Enterprise Architect and Business Analyst would change at the various levels.

Ram Kumar was from OASIS and provided an insight into the work of this organisation that is emersed in creating SOA standards. He discussed his role in developing the Justice Sector Information Interchange standard. This was of special interest to me because of my involvement in this sector. Ram emphasised the importance of XML governance and advocated this for successful SOA.

Josh summarised the conference and he may discuss this on his own blog site. Many themes resonated with me but one stands out as I wrestle with my in-flight dinner. We had slides from at least three of the speakers that showed insanely complicated maps of their existing systems. It seems we are all lumbered with integration spaghetti and our hope is that SOA will resolve this.

Getting Independent Advice

Last Tuesday we got a couple of consultants from interstate to advise us on how my organization is sailing with its initial steps toward SOA. Their brief was fairly broad. They spent a couple of days meeting with various groups including management, legacy developers, java developers and architects.

The idea was to get some idea of what we are doing wrong, what we doing right and what we should do in the future. This is a good thing to do if you can find respected practitioners who can provide this advice. It is easy to be insular and head off in your own direction without considering other options. This can apply to the organization, to groups of organisations and it can even apply to whole regions. I always object when the Federal Government or the eastern states regard South Australia as a regional outpost, but when it comes to best practice in IT we have to acknowledge that our expertise in major enterprise application development can be thin.

I was involved in a few of the meetings. I expect them to focus on our run-time stack, our development standards and development methodologies. The major focus was on our legacy software. This is a big problem and one which I was happy for them to take head on. We have many large systems written on the mainframe in technologies that are out of date and do not easily lend themselves to being accessed through services. The few tools that are available to assist are prohibitively expensive.

The report from the consultants has not come in yet but I was able to glean areas that the consultants seemed to be targeting. One was how to address issues such as Data Integrity, Process Integrity, processing exceptions and auditing in relation to making providing services using our legacy applications. These are the concepts discussed in chapter 8 of Krafzig et al "Managing Process Integrity". They are common application integration problems which pre-date SOA and they are the sort of problems that do not yield to simply adopting particular middleware or standards.

The other issue that the consultants were targeting was how much we were investing in our SOA. We are trying to fund SOA by paying for infrastructure and the development of skills out of project funding for the development of particular applications and enhancements. This may not be sufficient to get the critical momentum we need to get to a desired maturity with our SOA. It would be a good outcome if our independent advisers can report to our executive that we need more money specifically for SOA. It is a bit hard to do this from the perspective of the IT Department. The executive would suspect we were investing the money for no good business reason, perhaps just for buying new toys for IT. The opinion of the consultants may help to put our case.

Of course the consultants might be hoping we spend more money on them. We have to be on lookout for such bias in consultant findings. In general though it is always valuable to get a second opinion and I eagerly await the report from this exercise.

Reference

Krafzig, Dirk, Banke, Karl and Slama, Dirk, Enterprise SOA: Service-Oriented Architecture Best Practices, Prentice Hall, 2004

ESB or not ESB?

I work for an organisation that is pretty well fixed on achieving a goal of an SOA but we are still wondering how to get there. The worst people to ask are software vendors, because the answer will always be to buy more software. This stuff costs big time though.

For the cost it took to set up our application servers I could have bought two comfortable suburban homes and then we pay the software maintenance which I have to invest in at the cost of having a couple of staff. It is not as if I can do without the staff though. I now have to recruit someone to administer the application server and find developers with the skills in this particular applications server. So when the software vendor tells me that all I need to is the pay the cost of another suburban house and sacrifice funds for another staff member to buy an ESB I need to be pretty clear that this will provide some benefit.

There is some disagreement about what an ESB is or what benefit it brings. Mark Richards looks at some of the definitions of ESB in his presentation for the NFJS conference (video clip at http://www.infoq.com/presentations/Enterprise-Service-Bus). He then defines the ESB in terms of the capabilities it might bring these are listed as:

1. Routing
2. Message Transformation
3. Message Enhancement
4. Protocol Transformation
5. Service Mapping
6. Message Processing
7. Process Choreography
8. Services Orchestration
9. Transaction Management
10. Security

If software contains about five or more of these capabilities then it is an ESB. This would normally include first three capabilities. Together Mark Richards refers to these as "mediation".

This list is handy because it allows you to look at your needs and determine whether you should purchase such a product with these capabilities. You can determine whether these capabilities are available in your application services already, whether you need to acquire a niche product to address a capability or whether you can handle some of this capability in your own code.

This list is also useful to determine what you are not getting in a particular product offering. One consultancy firm demonstrated to me a great way of addressing the routing problem in application code and claimed I did not need an ESB because of this. It was certainly useful but it will not help if I need the other nine capabilities.

There is certainly some difference of opinion on whether you need and ESB. Vendors will say you need to have an ESB to do SOA. Sandy Carter of IBM in her web presentation (http://www.infoq.com/interviews/sandy-carter-soa) says that an ESB is a a "crucial part of an SOA" and a "spinal chord of your business".

On the other hand a presentation by Jim Webber of Thoughtworks is critical of the ESB approach and believes it works against a good SOA (http://jim.webber.name/downloads/presentations/2006-10-EJA-Guerrilla-SOA.ppt). Jim claims that ESB adds a proprietary middleware to you SOA which will lock you an your software in to a particular vendor. He goes onto to say that "spaghetti" in SOA is inevitable and one of the strengths of SOA is that it copes with this. ESB on the other hand can make spaghetti worse.

One of the best quotes I found was in article by Colleen Frye discussing ESBs with Jason Bloomberg and others:

This bandwagon's a bus and ... vendors positioning in the SOA market have jumped on it over the last few years.

I don't believe my organisation is mature enough in its SOA progress to consider and ESB. We have to develop a critical mass of services before we should look at investing in more glue (as Jim Webber describes it) to organise and bring these services together.

References:

Mark Richards http://www.infoq.com/presentations/Enterprise-Service-Bus
I liked marks insight in that SOA "Presents IT Services as Business Services".
Also he mentioned a couple of open source ESBs
Mule: http://mule.codehause.org/
ServiceMix: http://servicemix.org/ (JBI = JSR-208 compatible).

Sandy Carter: http://www.infoq.com/interviews/sandy-carter-soa
Some good parts of this talk excuse her from being a vendor and almost excuse her from using the word "leverage" too much.
"ESB as a pattern of SOA."
A quote from Steve Mills "SOA will be with us for decades"
"I think that BPM and SOA are two sides of the same coin. I don't think you can do BPM without leveraging SOA."

Jim Webber: http://jim.webber.name/downloads/presentations/2006-10-EJA-Guerrilla-SOA.ppt
This is a powerpoint slide presentation. He is also promoting his book that looks interesting.

Colleen Frye: ESB Market Report, http://searchsoa.techtarget.com/originalContent/0,289142,sid26_gci1226497,00.html