Beyond the hype

What’s driving the recent surge in interest in SOA and what benefits can it bring to FSIs. We brought together five experts in the field to find some answers to these questions and to get a real insight to why we need SOA now more than ever.

Recently, it seems that service oriented architecture has received considerable exposure and everyone has been scrambling to get on the bandwagon. But what’s driving this demand for SOA in the financial services industry – what benefits can the new breed of SOA vendors bring to existing infrastructure and what can forward-thinking IT departments expect next in the evolution of SOA? FST brought together six experts in the field to find some answers to these questions and to get a real insight to why we need SOA now more than ever.

With:
John Davies, Technical Strategist EMEA, IONA Technologies
Robert LeBlanc, General Manager, IBM WebSphere Software
Dave Chappell, Vice President and Chief Technology Evangelist, Progress Software Corporation, Sonic Products
Robert Skinner, Intersystems
Andy Gutteridge, Director International Sales, DataDirect Technologies Ltd

FST. What do you put the growing interest in SOA down to?
DC. From my experience with enterprise architects, project leaders and CIOs over the past few years, I’ve witnessed a dramatic change. SOA has gone from being a twinkle in the eye of a handful of people in advanced technology groups to becoming a mainstream concept for IT professionals. This can, in part, be attributed to maturing web services standards, but also to the ongoing evangelism and education efforts of industry analysts and vendors, and the broadening availability of SOA infrastructure offerings.

There are also a number of SOA deployments across a variety of industries segments, including financial services, that IT can highlight as success stories to be repeated and built upon. In financial services, an SOA built on an ESB can deliver a trading platform that allows traders of different asset classes visibility and access to the real-time positions on other desks, as required for cross-asset trading and risk management.

RL. I agree that SOA is becoming a somewhat ubiquitous topic and I know from speaking with customers and prospects that it is top of mind at the executive level. This is likely due to an SOA’s ability to more closely align technology with the needs of the business. However, behind the scenes, it’s important to note that only now has technology evolved to the point where it can fully link the disparate parts of the organisation. This is at the heart of an SOA and helps explain the phenomenal growth of this market segment.

RS. The resurgence in popularity of SOA is based on the fact that the technology is now mostly fit for purpose, and the business needs are better understood. SOA enables businesses to satisfy consumer demand for accessible easy-to-use systems whilst aligning IT with measurable goals and reduced change management overheads. We hear from industry gurus that markets are moving from mass-production with mass-automation to mass-customisation. InterSystems has many customers who are using SOA to facilitate mass-customisation by enabling business analysts and managers to dynamically reconfigure their processes to service their customers more effectively whilst competing in the market more efficiently.

InterSystems is also using SOA to satisfy the users’ need to evolve systems and reiteratively deliver, often in only three months, return-on-investment. SOA makes it practical for us to abandon ‘rip-and-replace’ strategies in favour of a ‘leave-and layer’ approach. I also think that it is a result of businesses recognizing that service delivery, innovation, performance, compliance and the speed of change or time-to-market can all be improved.

JD. SOA is the natural evolution for complex systems and it is only the addition of web services that has created this new breed of SOA vendors. People have been exposing systems as services long before the term SOA came along and today there are many successful SOA deployments already in operation. The Credit Suisse SOA is a well-documented example. However, much of SOA is just EAI re-labelled. EAI is not SOA, it is an architecture; EAI is just the process of integration whether it is service-oriented or not.

AG. The hype is warranted, as there’s real potential for long-term business benefits to be delivered by adopting a Service Oriented Architecture. The ‘Holy Grail’ for any business is ‘agility’; adopting SOA can contribute greatly towards increased flexibility within your IT infrastructure and faster delivery of new IT applications to meet changing business needs. Evidence of this is already emerging in some corporate IT departments, where the concept of SOA, if not the actual implementation, is widely understood and accepted. In short, SOA offers more than just technological hype, it delivers a standardised approach to the development and deployment of enterprise applications creating ‘business agility’.

Today, organisations are already building components and web services on distributed platforms, be they J2EE compliant or .NET centric, however legacy reliance is ever present. Therefore, to keep pace with technological innovation, respond to new business opportunities and at the same time maintain existing operational service levels, enterprise organisations must incorporate legacy applications within the modern SOA. In fact, an SOA only achieves true ROI and universal integration throughout any large enterprise by incorporating legacy applications. Since simply re-coding legacy applications into a new language on a distributed platform is too costly and risky, enterprise developers must leverage a tool to isolate specific business transactions from within multiple legacy systems and expose these transactions as reusable components and web services. Many of the largest companies in the world continue to run their most important applications on IBM mainframes, and consequently, the potential for delivering real business benefits with SOA increases accordingly, if extended to this platform. However, in our experience, the adoption of SOA techniques lags behind the rest of the IT world, despite the availability of robust, feature-rich software solutions – such as DataDirect’s Shadow RTE platform.

FST. So, how can an enterprise utilise SOA solutions to help automate its business processes and increase operational visibility?
DC. The trend is moving toward real-time decision-making rather than using data warehousing and business intelligence tools to analyse yesterday’s data in order to make predictions about the future. As an enterprise moves toward a real-time sharing of information through an SOA, significant information about the business may be fed into business activity monitoring (BAM) dashboards, enabling you to see business exceptions as they occur and make course corrections along the way.

An SOA allows an enterprise to build new applications that automate business functions. It enables an ongoing partnership between IT and the business, by showing a more direct associating between the software projects under way and the business problems they address.

Using visual modelling tools, applications are built as composite assemblies of services that leverage existing application logic and data sources. The key to building flexible business processes lies in loose coupling between services where making a change in one service does not adversely affect the other services that are involved in the business process. The details of the service deployment artefacts and the interactions between the services are configured rather than coded, providing quicker construction of business processes and allowing greater flexibility for future changes.

RL. The very basis of an SOA is to help automate business processes and increase operational visibility. This can be done through a variety of tools, although it’s recommended to tap into business process modelling and monitoring software. The former allows you to map out how your SOA should be structured before time and resources are spent in deployment. Monitoring software also allows you to oversee the SOA’s success and its impact on every function in the company and to make adjustments accordingly without having to rip and replace.

RS. We believe that a successful SOA environment actively embraces all areas of a business – making it ‘fully connected’. Not just the applications, but the people, processes and data. For example, SOA is enabling financial service organisations to pull together their disparate islands of information, to tie in the front-end web or call-centre information to their back-end data repositories. But, by using SOA, the benefits go beyond just the data integration. They are building a business platform where end-to-end customer-facing processes can be established across the whole enterprise. The business is more self-aware with real-time feedback from the processes of operational data, so that managers at all levels are able to make intuitive decisions to cross-sell or offer further services to increase revenue.

We can see that our enterprise customers want to move away from time-consuming and costly change control procedures based on monolithic architectures. SOA enables them to transform IT into small flexible re-usable business services that can adapt and change quickly without upsetting the entire IT operation.

JD. SOA requires applications within the enterprise to be service-enabled and SOA solutions help that process of service enablement. Once enabled, services can be monitored, managed and reused; a manageable service can be exposed to BPM tools. The ability to monitor services allows BAM tools to effectively monitor business activity. Being able to see what is happening in real-time, not only increases operational visibility, but also reduces operational risk.

AG. One of the main drivers for SOA is the ability to reuse existing business logic and data regardless of the platform, operating system or underlying programming language. This allows an organisation to very quickly assemble the appropriate building blocks for a business solution and to then publish quickly for use across the organisation. Indeed this can also be utilised by partners or customers with the sufficient controls in place. The addition of an ESB platform will help ensure that an appropriate level of abstraction is available across the IT infrastructure such that the concept of a standardised ‘service’ can be applied. This, in turn, supports a ‘composite application development’ approach whereby collections of services can be assembled to support a new business process.

However, an SOA is by definition, a ‘request-reply’ model and in and it cannot deliver operational automation of a business process. Organisations who wish to automate certain business processes must consider an Event Driven Architecture (EDA). This application can then drive further applications based upon the content of the XML ‘package’. A simple example is a customer service application for insurance claims somehow the business needs to alert management when a claim of over £10,000 is registered from a customer within a specific location. When the claim is created, this will result in a database update. In an EDA, there would be a software module called an ‘agent’ that recognises the change, captures the data, analyses it (based on the business rules) and creates an XML ‘package’. The XML may be enriched with data from another database (such as policy number, claim history etc.), and then publishes it to a dashboard application.

FST. Is it costly to implement, and how well does this fit with existing IT investments? How can short-term and long-term costs be minimized?
RL. It’s actually the most cost-effective approach because it allows you to make the most of existing legacy and home-grown systems. It does this by eliminating redundant business processes and installing consistent strategies throughout every department in your company. This strategy saves you money because the building of the SOA is incremental and drives reuse throughout the company as different teams develop and share proven best practices.

RS. Implementing SOA can just be about fitting new capabilities with existing IT investments, for example, delivering compliance with application integration, so spreading the SOA adoption costs across the line-of-business projects.

However, to fully embrace SOA, a cross-department governance team is necessary – it has to be supported through all of its working practices and not just run by the IT department. It is essential that this team embodies or has full access to the knowledge of the business and its processes, with the target market, and that it is empowered by senior management. This enables informed decisions that determine which applications should be converted to services and which should be retired or outsourced.

It is therefore no surprise that minimising SOA adoption costs is achieved largely from cultural decisions. But, the choice of technology counts too. Analysts are reporting high failure rates for SOA projects that choose technology that will not scale in capacity or volume or handle the required levels of complexity. For immediate and sustained benefits, the IT department needs to work with the business to manage consumption not just supply. For many companies the old default options of choosing the technology based on using big-name vendors or existing skill-sets is also proving a cul-de-sac. Instead, businesses need to ensure that the technology has been proven by a proof-of-concept exercise so they can be certain that it is fit for the given purpose.

FST. You also have to ask yourself the costs of not adopting SOA. You can stand and watch as your competitors embrace it, but how much it will cost the business if you cannot compete in terms of offering the same level of service or react to business opportunities before or at the same time as them?
DC. Adopting SOA should allow you to leverage investment in existing assets. Short-term costs may be minimised by identifying reasonably scoped projects that can be achievable within a matter of months. This allows you to build repeatable success patterns that can be carried from one project to the next, demonstrating ROI for each successful project. This, however, must be accompanied by a long-term vision and a plan toward where you are trying to get to, which is a broad scale SOA implementation.

Long-term costs may be minimised by standardising on infrastructure that is flexible and capable of adapting to change as business requirements change over time. SOA infrastructure such as an enterprise service bus and an SOA management platform can delegate the busy work of security, transports, process orchestration and policy-based governance into its own configurable layer, rather than coding those things into each service.

JD. The most effective way to expose applications as services is to enable the application at the lowest level possible, i.e. either by changing the code or by providing run-time plug-ins, ‘agents’. Adding a hub that requires further integration adds complexity, requires more hardware, management and therefore cost. Enabling applications at a lower level is more cost-effective, increases efficiency and reduces load. An agent-based approach is also transport- and container-agnostic, giving more options and a longer shelf-life.

AG. One way to succeed with an SOA implementation is to start with small easily achievable deliverables, allowing an SOA project to quickly gain traction and acceptance within the organisation and other departments can see the benefits. It is important, however, whilst looking for quick wins, to keep a firm eye on the underlying technologies that are used to base the SOA and web services implementation on. The use of web services can grow very rapidly, so it is vital that enterprise strength solutions are used to underpin the implementation. Don’t just think about the first project you are building, but consider how this will grow and how well the solutions you choose will scale. Also do these solutions support complimentary technologies to SOA such as EDAs? It is worth remembering that one of the main drivers for an SOA is reusability. Rather than tear down what you have already invested a lot of money in, the aim is to reuse it and open up code to platforms and programmers who otherwise would not know how to invoke that piece of code. This means that your initial IT investments are protected and also their uses are expanded. It also allows you to get rid of waste and prevents the need to maintain two pieces of code which fundamentally do the same thing - they just do it on another platform.

Complexity is the enemy of a successful SOA implementation. However, if you follow the principles of SOA – granular services, easily available, composite application development - then complexity is inevitable. Therefore, an SOA management platform is an essential tool to help keep control and minimise long-term costs.

In the legacy mainframe world, having a single technology platform is critical to the elimination of complexity and the reduction of cost, both in terms of reduction of development time and effort, and on-going management of the SOA environment. Equally important is to have a tool that is non-invasive to the legacy application code base.

FST. What, to date have been the biggest concerns for enterprises in implementing an effective SOA strategy – what has traditionally prevented them from fully realising its benefits?
RS. Enterprises have suffered from a lack of real understanding about the capabilities of a SOA. They have been distracted by the technology, the ‘how’, and not focused enough on the ‘why’ that establishes the business value. Without a clear benefit statement most IT projects will fail. SOA is not about web services or enterprise service buses (ESB) – they are just the means to the end.

Whilst businesses are starting to see the benefits and business advantages that can be gained, there are still concerns over where to start. Systems integrators and consultancies practices that have proven track records and domain skills can help them ask the hard questions. Business need to decide which applications are ‘utilities’ where efficiency and cost management are important, and ‘frontier’ where capability and competitive edge define their value. This can help determine where in the business SOA can be first applied.

Above all, for SOA to be effective companies need strong governance to enforce re-use of processes and components – so reducing complexity rather than increasing it.

DC. The biggest challenge in implementing SOA is not the technology, but making the necessary corporate cultural changes. IT departments are typically organised around silos of application logic and silos of data, with CIOs keeping a tight lid on access to corporate data. The adoption of SOA requires that application functionality and data silos be opened up; it requires cross-departmental communication and formation of new architectural teams that look at the broader picture to answer questions such as what services will be created, how many, and at what level of granularity will service interfaces be exposed.

The adoption of SOA also opens up new issues surrounding ownership of composite business processes and governance of shared services across and enterprise. Who now owns the composite application that leverages business logic across several siloed applications, and who will enforce policy decisions across a set of reusable shared services?

Equally as important is seeking alignment between IT and the business, and working with the business to understand their most important business imperatives and ensure that the SOA projects that the IT side is working on are meeting the needs of the business. Successful adoption requires buy-in at all levels across the organisation, which includes executive sponsorship across disciplines.

RL. I think the biggest challenge for organisations has been knowing where to start. While both business and IT realise the benefits of SOA, they sometimes get stymied into viewing it as a whole, instead of being more than the sum of its parts. The full benefits of an SOA can be realised when both IT and business leaders collaborate on the overall goals and strategy and then take incremental approaches to the creation of the SOA. Recognising the small successes through a step-by-step approach quickly resonates into tangible ROI.

JD. In my experience, the impact of adoption has been the biggest concern. This isn’t helped by the fact that many vendors sell service-oriented ‘applications’ rather than ‘architectures’, i.e. application servers and JMS-based ESBs. The need to add yet another application server, JMS or message bus further complicates the existing enterprise systems.

It is fairly common knowledge that ‘classic’ XML-based WebServices add significant latency and CPU costs, which makes ‘classic’ SOA difficult to sell internally.

AG. Recent major surveys on SOA adoption cite a number of factors that have contributed to a slower adoption of SOA than predicted. These include performance/throughput, complexity of integrating services, and the development effort required for composite applications. Another key factor has been the absence of web services standards that could make a fully robust enterprise class solution more achievable. As an example SSL was used as an encryption method whilst WS-Security was coming to fruition. However, SSL has several limitations – it is expensive and only a point-to-point solution making it a less than ideal. The same is also true for transactional support in web services, which will be addressed with WS-Atomic Transaction. There is also concern, especially on the mainframe front, that breaking down business components that reside on the mainframe and then gluing these back together again in a client-side application will increase network traffic as multiple requests are sent to the mainframe instead of just one.

An equally significant challenge is how to embrace the organisational changes necessary to fully implement an SOA approach. It is typically desirable to create services that can span existing application boundaries. This approach does not fit the existing structure of most IT departments, being typically ‘stove-piped’ within functional departments.

FST. How important is interoperability in an SOA solution and what other characteristics should an enterprise be looking for today in order to maximise cost-effectiveness and achieve better ROI in the long-term?
RL. You can’t underestimate the importance of interoperability in an SOA. When you consider that an SOA is all about making each part of the organisation freely and securely ‘talk’ to one and other as well as customers and partners, you quickly realise that the most successful and effective SOA is one based on open standards that enables full interoperability among applications, services and hardware.

JD. Interoperability is vital, which is one reason why so many vendors have stuck with ‘classic’ XML over HTTP. For many, HTTP is the universal transport and SOAP/XML is the universal payload. While XML will probably remain the principal payload, it is still prohibitive for high performance and low latency. HTTP is great for the web but a terrible transport in most other areas. Many prefer to use MQ Series or provide an equivalent replacement, usually JMS.

In the very near future, we will start to see open, wire-level protocols like AMQ from the banking world eating into the transport layer. Interoperability will then become XML or Java over AMQ. Artix does not predicate a transport or payload and supports all of the above. Several vendors actually embed Artix to provide a bridge into their own internal technologies, Artix is therefore ideal for interoperability in the short, medium and long term.

RS. Without interoperability, SOA is ‘just another IT project’ creating silos of functionality that cannot be easily leveraged in the rest of the business. The principal idea behind implementing a SOA is to package IT resources, such as applications, data and the processes they support, into a unified collection of reusable services. These can then be presented as building blocks of functionality that are integrated into the human workflow. Defining the services to be provided is key to making SOA projects successful. The ability to easily orchestrate services to line up the end-to-end activities, regardless of whether they are back-end or front-end requests, characterises successful a SOA platform.

It used to be said that the ‘network is the system’, now we’ve moved on to say that ‘the processes are the system’. SOA is the ideal platform for their discovery, development and maintenance and for dynamically packaging them as services for the benefit of the entire, agile enterprise. But, I repeat, the technology is not the whole story. The organisation’s culture has to recognise that the technology needs to be governed by the business. Otherwise we fall back to the old habit of letting the technology set our business agendas. A well planned and efficiently implemented SOA will enable organisations to manage change and continuously innovate. With increased and timely awareness of the business processes the organization can manage its costs most effectively and consistently measure ROI.

DC. Until recently, web services standards have not been mature enough to provide capabilities necessary for broad-scale enterprise deployment. The platform vendors and ESB vendors continue to work out the interoperability details as advanced web services specifications and implementations become mature. Even when that happens, no IT department has the luxury of retrofitting all of their applications across their enterprise to conform to any one style of interface technology.

We are addressing the technology barriers by providing an ESB that allows applications and data sources to be brought together into a SOA using the protocol, interface or adapter most appropriate for the individual platform in the most non-intrusive way. It’s advisable to plan to migrate all applications to web services technologies, but that may not be practical to do all at once. Sonic ESB allows full use of advanced web services, and also the not-so-modern standards such as FTP and SMTP. Even if you can only interface to other applications using batch processing you may still get on the bus and start participating in the SOA immediately. This allows you to put together a longer-term plan to simplify diversity, consolidate platforms and standardise on protocols and interfaces at your own pace.

AG. Interoperability is a key deliverable of SOA, which is why the WS standards, and the adoption of them within SOA tools, is very important. For the legacy mainframe environment, ensuring that the SOA solution you adopt is implemented within a single, unified platform will minimise ongoing management costs. The IDE is also important, as development time and effort will contribute to the total cost of the project as well as ongoing maintenance.

It is also important that solutions put in place to address SOA, are robust enough to cater for expected and unexpected growth. It is a false economy to think only of the current project and cobble together a solution that will not meet the whole organisation’s needs. You will end up with reusable business components but find that the technology you are using is NOT reusable.

FST. Other than financial considerations, what would you consider to be the most crucial component of an SOA? For example, how important is it that it enables visibility of information, supports multi-channels, etc?
DC. The most critical aspect of a SOA is that it enables a business to transform itself to be more predictive by gaining real-time insight into the business data as it flows across the enterprise. Through tools and infrastructure that enable business process visibility, an enterprise can make real-time business decisions based on collection of business metrics, the capture and reporting of business exceptions, and immediate course corrections that will keep their customers happy.

An SOA should also allow an organisation to more quickly and readily implement new business imperatives that increase the top-line revenue by attracting and retaining customers. This can be achieved through loosely coupled service level interfaces that leverage existing assets, a flexible configuration and modelling tools, such as provided by an ESB. An ESB may also provide operational continuity and continuous uptime of the SOA infrastructure.

JD. Visibility of information and the ability to adapt to change are probably the biggest winners for SOA; together these reduce risk and increase agility. Remember that SOA is an architecture not an application, it’s how you apply the architecture that dictates the ROI.

RL. When it comes to financial considerations, let’s understand that an SOA is the most cost-effective use of IT and business resources because it delivers considerable pay back – so it’s a cost saver. Visibility of information and supporting multi-channels are at the cornerstone of an SOA strategy. Without a clear view into your business, you risk first mover advantage and lack the ability to quickly respond to competitive threats, which really represent two of the biggest drivers for SOA adoption today.

RS. Well, it depends on what the line-of-business is using SOA to do. Sometimes the focus is on business process management, sometimes data federation. Beyond these use-cases, and others like workflow and business activity management, I think the crucial factor is making it all work together, seamlessly.

I go back to what I said earlier, the vision should be to make the organization ‘fully connected’, in terms of the applications, people, processes and data. In my opinion, the financial services sector is probably the most rigorous, stringent and compliance demanding sector where getting the technology right means success or failure. Therefore the business areas within financial services tend to be very demanding of IT, in areas such as performance, integrity, security, time to market, ROI and scalability.

With that in mind any IT or business area within financial services that is considering an SOA infrastructure has to look at the pedigree of the company and solution they are intending to implement, to ensure that the implementation can meet all the high demands mentioned above, as a minimum.

AG. The most crucial aspect of an SOA is that it delivers a more ‘agile’ IT infrastructure that can react to new business imperatives much faster than is currently possible. The ability to take a more granular approach to application logic – represented as a collection of standardised services – which can be quickly combined to deliver new applications will be a significant competitive edge for any business. The additional benefit is that with a higher instance of code-reuse, and the adoption of appropriate standards, the cost of ownership of IT should decrease in the longer term.

An important aspect on the road to SOA is the ability to extend it to the legacy environment where traditional application boundaries can be broken down and the valuable business logic and data can be exposed as standards-based services, in the same way as any other platform.

At an operational level, SOA can help enable the Real-Time delivery of critical business information. Equally, the ability to adopt an Event Driven Architecture alongside an SOA will be increasingly important in order to realise the true ‘Real-Time Enterprise’. DataDirect’s Shadow RTE platform enables both SOA and EDA for the IBM zOS mainframe environment, in a single unified platform that is market proven within some of the most demanding operational environments in the financial service market.