Financial institutions have made significant investments in messaging and service-oriented architecture (SOA) projects to integrate disparate systems and gain control of complex, heterogeneous IT environments. Unfortunately, many of these projects have not efficiently addressed a fundamental component of integration – normalising the disparate data sources used by system endpoints. Failure to manage the data component of application integration has significantly diminished the success of many integration projects.
Sometimes referred to as the “missing link in application integration”, data services are reusable data integration tasks performing a range of data mediation functions and are loosely coupled from the data sources they serve. Data services can be registered, discovered and reused across local and remote projects. Open, standards-based, model-driven tools are now available to build a “data services layer”, allowing organisations to more efficiently implement and manage heterogeneous data models, while vastly improving the quality of data flowing between applications and partners. This modern approach to data integration can reduce integration effort and time-to-market by 50% by using data about data to drive efficiency and reduce risk.
Messaging and SOA Data Integration Challenges
The demand for information in real time combined with reducing operations costs and shortening project cycles are causing IT to rethink data integration investments. The new challenge is to build a flexible and efficient architecture that addresses evolving data demands:
=> More data to manage. The amount of data that IT teams need to manage and map is doubling every two years. In fact, many large companies have deployed over 15,000+ data sources and 100,000+ business processes.
=> More data consumers. Additional departments want message segments for analysis, e.g. the auditing group needs a copy of transactions for regulatory compliance or the risk management group needs to see real-time settlement exposure.
=> Adopting and managing industry standards. External pressures for partner or regulatory requirements mean supporting additional standards such as SWIFTNet, SWIFTSolutions, FpML, ISO 20022, SEPA, MDDL, ORIGO and FIX. Standards are open to interpretation, so typically each partner and internal user has unique implementations. Finally, standards change frequently requiring companies to support multiple versions for each channel.
From a business perspective, data quality issues can lead to lost customers, lost business, risk and regulatory fines.
Semantic Data Interoperability
When a developer builds a process such as “unified customer view” they will most likely find syntax and semantic mismatches across the connected data sources. Reconciling the syntax differences, for example FirstName = FName is straightforward; however, normalising the semantics can be very complex. Semantic rules are to ensure the business logic usage of the data is correct. This is a key requirement for successful automated processing. Consider the following validation rule for a financial services SWIFT MT103 message:
The above rule is just one of 19 conditions for this message, which is actually one of the simpler SWIFT messages!
Custom coding transformations and validation rules between multiple data models is not an effective use of time, leads to higher development and testing costs, and significantly increases the risk of transaction failure. In fact, a recent analyst report found that over 80% of developers write code to integrate data sources, increasing the cycle times and costs to maintain these applications. Furthermore, each time a new application is integrated into an existing SOA or messaging project, there is also an 80% probability that the resulting schema and semantic change needs to be made to the business process.
Creating a Data Services Layer
While there are many definitions of a data service and its purpose, there is common agreement that a data service has the following properties:
“Messaging” data services are being driven by application and SOA architects looking for a more efficient way to develop, deploy and manage messages as they flow between systems. Their project may have created services that addressed the application interfaces, but did not reconcile the different data formats and semantics. Or, they could have a project that has an industry standard library like FpML that they want to use as a service to validate incoming and outgoing messages between partners.
Example of Messaging Data Services
Best Practices for Implementation Data Services
Each project adds to the complexity of data management – where the number of data mappings and rules that needs to be supported grows exponentially.
In addition to the specialisation features mentioned previously, the tool should have a change impact analysis component that allows users to visually see the differences when updating models, and even multiple derivations of a base reference model. The ability to manage versioning of changes within the data model by integrated version stamping and change control management within the development environment is also important. Interoperation with existing change management and version control systems is obviously desirable.
Benefits of Messaging Data Services
By abstracting metadata from an application into data services, it becomes easier for other projects to correctly reuse and access the data. This type of architecture has the following benefits:
In summary, data services support a more efficient architectural model that enables reduced development and maintenance costs through reuse of data services, better rates of automation, and more flexibility to accommodate change. Furthermore, this technology is already being used in some of the most demanding business domains and has been proven to reduce the risks of transaction failure and vastly improve the overall efficiency of the IT organisation.
 Where n is the number of endpoints, e.g. 10 systems each with unique schemas and validation rules, could mean 10*(10-1)/2 or 45 point to point data mappings