Asynchronous Messaging Architecure as an Integration Solution

Fundamental challenges of integration solutions include most of the fallacies of distributed application
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

We can summarize the major challenges of integration solutions into to the following

• Networks are unreliable
• Networks are slow
• Any two applications are different
• Change is inevitable
  

When we are trying to integrate two Apps A and B ,there are 4 main approaches to do this and overcome the above mentioned challenges

1. File Transfer : One application writes a file and another application reads from the file. The applications will need to agree on filename,location and format
2. Shared Database: Multiple applications share the same database schema, the integration happens through the database.
3. Remote Procedure Invocation/ Web services: One application exposes one sort of functionality that can be accessed remotely by other application as remote procedure. This could as well be a web service. Communication occurs in real time and synchronously.
4. Messaging: One application publishes a message to a common message channel. Apps need to agree upon message channel and format. Communication is asynchronous.

The issues with approaches in 1,2,3 include

• Apps are tightly coupled.
• Apps A and B need to scaled together
• If A needs high availability then even B needs to made highly available.
• The model of communication involves the applications to be polling for the availability of the data from the other application.

Messaging is more immediate than file transfer, better encapsulated than a shared database and more reliable than a web service/RPI.Advantages of going with the messaging approach include:

• Apps involved in integration (A,B) are loosely coupled .
• Apps can be scaled independently of each other
• Apps can functionally be available independent of each other. App A can be up and functional even when B is down. App A will queue up the data that is required by B , When B is up it start consuming the events from A.
  
• Apps do not have to be build to support peak load as the asynchronous messaging allows the processing to be spread out over time. Asynchronous dampens the load.
• Throttling Ability.
  
• Reliable Communication - reliable and guaranteed delivery
• Platform/Language agnostic integration

Asynchronous Messaging Architectures have in turn have its own challenges.

1. It is a complex programming model. Asynchronous messaging requires developers to work with an event driven programming model.
2. Message sequence issues - Message channels guarantee message delivery but they do not guarantee when message will be delivered. Might require some logic to re-sequence the events at the consuming end if IN ORDER Delivery is a required SLA.
3. Synchronous Scenarios- There might be scenarios where synchronous is required for better user experience. Intelligent workaround schemes might be required to support it.
4. Performance-- Messaging is best suited to keep the system in sync after the initial data replication. Messaging is not suited from the performance aspect to be used as mechanism to replicate data.
5. Limited platform support. This methodology is not yet mainstream and the support and best practices are in limited availability.
6. Vendor Lock-in: Messaging brokers used might result in a vendor-lockin problem. But as along as we stick to to the standards like JMS and stay away from using the broker specific features most of the vendor lock problems can be mitigated.
   

Why should we avoid distributed transactions ?

A distributed transaction is a transaction that spans multiple resources. A distributed transactions are also at times referred to as XA transactions. XA transaction is X/Open group specification on distributed transactions. XA transaction is required to guarantee the ACID (Atomicity,Consistency,Isolation,Durability) properties of a transaction when the transaction spans multiple resources - be it multiple databases,database and JMS connection, database and file system or any resource. XA transactions require transaction manager to co-ordinate the multiple resource managers involved in the transaction. Transaction Managers co-ordinate the resource managers involved using 2-Phase Commit protocol (2-PC).
Usage of distributed/XA transactions in turn come with its own issues and complexities.Some of the problems/complexities with distributed transactions include

1. 2-PC protocol is very chatty protocol and does a lot of logging to be able to recover from any failure scenario.
2. Too much overhead to 99.9% of the cases to handle less than 0.1% of the failure/exception cases
   
3. Increases the amount of time the locks are held on the databases. This increases the chances for deadlocks on the database. This also lowers the overall performance of the system.
4. Distributed transactions are sort of the bane of scalability. It sort of grinds the entire system to halt by adding overhead to each and every transaction.
   
5. Availability of the Systems goes down. When using distributed transactions,the completion of a distributed transaction is now a product of the availability of two different systems and there by the the total availability of the system goes down.
6. XA/Distributed transaction configuration is complicated and is difficult to test to make sure the configuration is configured correctly. (Many Java Developers tend to believe that using JTA implementation of transaction manager will take care of a XA transactions and tend to forget configuring the resources as XA resources. Many a time people end up using JTA even while dealing with single resource. ). Many people get XA enlisted resources when they don't have to and many a time the applications would perform better if they weren't unnecessarily using XA.
   
   Based on the various issues mentioned above it is very much recommended that the use of XA transactions be avoided as much as we can. So how do we maintain the ACID of a transaction that goes against multiple resources !!!? In most cases intelligent recovery business scenarios supported by the system's management interface will eliminate a need for XA. The possible ways to circumvent the need for distributed transactions could be a seperate post in itself...stay tuned..:).