Data Services for a distributed world

Financial institutions have made significant investments in messaging and service-oriented architecture (SOA) projects to integrate disparate systems and gain control of complex, heterogeneous IT environments. Unfortunately, many of these projects have not efficiently addressed a fundamental component of integration – normalising the disparate data sources used by system endpoints. Failure to manage the data component of application integration has significantly diminished the success of many integration projects.

Sometimes referred to as the “missing link in application integration”, data services are reusable data integration tasks performing a range of data mediation functions and are loosely coupled from the data sources they serve. Data services can be registered, discovered and reused across local and remote projects. Open, standards-based, model-driven tools are now available to build a “data services layer”, allowing organisations to more efficiently implement and manage heterogeneous data models, while vastly improving the quality of data flowing between applications and partners. This modern approach to data integration can reduce integration effort and time-to-market by 50% by using data about data to drive efficiency and reduce risk.

Messaging and SOA Data Integration Challenges
The demand for information in real time combined with reducing operations costs and shortening project cycles are causing IT to rethink data integration investments. The new challenge is to build a flexible and efficient architecture that addresses evolving data demands:

=> More data to manage. The amount of data that IT teams need to manage and map is doubling every two years. In fact, many large companies have deployed over 15,000+ data sources and 100,000+ business processes.

=> More data consumers. Additional departments want message segments for analysis, e.g. the auditing group needs a copy of transactions for regulatory compliance or the risk management group needs to see real-time settlement exposure.

=> Adopting and managing industry standards. External pressures for partner or regulatory requirements mean supporting additional standards such as SWIFTNet, SWIFTSolutions, FpML, ISO 20022, SEPA, MDDL, ORIGO and FIX. Standards are open to interpretation, so typically each partner and internal user has unique implementations. Finally, standards change frequently requiring companies to support multiple versions for each channel.

From a business perspective, data quality issues can lead to lost customers, lost business, risk and regulatory fines.

Semantic Data Interoperability
When a developer builds a process such as “unified customer view” they will most likely find syntax and semantic mismatches across the connected data sources. Reconciling the syntax differences, for example FirstName = FName is straightforward; however, normalising the semantics can be very complex. Semantic rules are to ensure the business logic usage of the data is correct. This is a key requirement for successful automated processing. Consider the following validation rule for a financial services SWIFT MT103 message:

• C3: If field 23B contains the code SPRI, field 23E may contain only the codes SDVA, TELB,PHOB, INTC (Error code(s): E01). If field 23B contains one of the codes SSTD or SPAY, field 23E must not be used (Error code(s): E02).

The above rule is just one of 19 conditions for this message, which is actually one of the simpler SWIFT messages!

Custom coding transformations and validation rules between multiple data models is not an effective use of time, leads to higher development and testing costs, and significantly increases the risk of transaction failure. In fact, a recent analyst report found that over 80% of developers write code to integrate data sources, increasing the cycle times and costs to maintain these applications. Furthermore, each time a new application is integrated into an existing SOA or messaging project, there is also an 80% probability that the resulting schema and semantic change needs to be made to the business process.

Creating a Data Services Layer
While there are many definitions of a data service and its purpose, there is common agreement that a data service has the following properties:

• It performs a data integration task, such as data parsing, transformation or validation.
• It is loosely-coupled between consumers and producers of data, so a change can be made to the service or an endpoint without impacting all other connected endpoints.
• It has a well-defined interface. This interface contains the properties of the service independently of the specific implementation and includes the format the data consumes and produces along with any qualities of service.
• It is coarse-grained and modular and can serve static and dynamic applications, such as in-flight messages, bulk data imports and exports.
• It is designed for reuse. It is discoverable and available to be reused within a single project as well as across multiple projects.

“Messaging” data services are being driven by application and SOA architects looking for a more efficient way to develop, deploy and manage messages as they flow between systems. Their project may have created services that addressed the application interfaces, but did not reconcile the different data formats and semantics. Or, they could have a project that has an industry standard library like FpML that they want to use as a service to validate incoming and outgoing messages between partners.

Example of Messaging Data Services

Best Practices for Implementation Data Services

• Open Tooling – An open, intuitive and graphical toolset to import and export data models, define transformations and validation rules, then deploy them as runtime data services improves the ability of a company to develop and manage data services, and results in a significant savings over hand coding. Look for tools that support standards such as Java, XML Schema, Database Definition Language (DDL), interchange with other metadata definitions and the capability to import and map any binary or ascii data.
• Model-based Approach – In a data services context, model-driven data services can model any data source [binary, delimited, fixed format, tag/value, XML] or industry standard such as SWIFT, FpML or SEPA. The model includes mediation events and validation rules, can be specialised, i.e. inherit the base model and restrict or extend it based on the needs of the partner, company or market best practice. Allowing departments to customise the data model to create their own view and version, while retaining the associated relationships of the base model, facilitates the adoption of the model companywide.
• Common Data Model – A common data model is in reference to the model that is used (and agreed upon) by company, department or project-wide–a universal schema and set of semantics across all applications and systems. As new applications are integrated, there is now a lingua franca for data interoperability. If steps can be made to move from tens or hundreds of models to a few abstracted models, a company will certainly become more responsive to change.
• Distributed Architecture – The centralised hub approach is becoming a bottleneck for IT. With a distributed approach to data integration, transformation, normalisation and validation are completed at the endpoint either when the message is received, sent or both. Developing a distributed architecture is an efficient way to achieve greater scalability, and to lower the platform costs because the data normalisation function can be performed on the existing hardware running the application.
• Lifecycle Management – When dealing with diverse and distributed data models that are constantly changing, it is important that tools manage the entire lifecycle (model, develop, test, maintain, retire). The following illustration highlights how complex the problem is in managing data relationships and the need for lifecycle management tools.

Each project adds to the complexity of data management – where the number of data mappings and rules that needs to be supported grows exponentially.

In addition to the specialisation features mentioned previously, the tool should have a change impact analysis component that allows users to visually see the differences when updating models, and even multiple derivations of a base reference model. The ability to manage versioning of changes within the data model by integrated version stamping and change control management within the development environment is also important. Interoperation with existing change management and version control systems is obviously desirable.

Benefits of Messaging Data Services
By abstracting metadata from an application into data services, it becomes easier for other projects to correctly reuse and access the data. This type of architecture has the following benefits:

• Flexible architecture facilitates change: When a data source is updated, instead of making [1] updates as in a tightly-coupled, point-to-point scenario, only one update needs to be made between the changing data source and the service.
• Provides greater reuse, minimising data duplication: Other processes and applications can reuse proven, high quality data services, reducing the time it takes to code to a specific data source.
• Provides a common tool and language across the company and partners: A single development tool and documented data model that can be shared across the enterprise (developers, architects and business analysts) increases collaboration between departments.
• Improves data integrity: Implementing real-time data validation services as messages flow between endpoints improves the quality of information in the data sources and minimises the amount of fixing failed transactions or inconsistencies.
• Improved scalability and availability: Distributed data services significantly improve scalability and availability, and a distributed approach reduces the risk of a centralised bottleneck and single point of failure.
• Lower total cost of ownership and improved organisational capacity:
  Example metrics include:

1. A 20% reduction in the initial design phase by using a graphical IDE designed for managing data models over hand coding transformations and validation rules.
2. A 50% reduction in cost to develop and test updates by using model-driven tools that generate high quality code
3. Increased collaboration between IT and business analysts through documented services.

In summary, data services support a more efficient architectural model that enables reduced development and maintenance costs through reuse of data services, better rates of automation, and more flexibility to accommodate change. Furthermore, this technology is already being used in some of the most demanding business domains and has been proven to reduce the risks of transaction failure and vastly improve the overall efficiency of the IT organisation.

Reference:

[1] Where n is the number of endpoints, e.g. 10 systems each with unique schemas and validation rules, could mean 10*(10-1)/2 or 45 point to point data mappings