How to generate Events ?

Event Driven Architecture or EDA is the handling of events as they occur and the transfer of events among the systems in a loosely coupled way. The usage of events and EDA can replace the need for integrating the systems at the database level. With the usage of EDA's the systems are loosely coupled and information can be passed between the systems in real time, rather than relying on ETL processes or data transfers between the systems. The hardest part of the implementing an EDA is capturing the events
Events can be generated by tapping into web sites, files, databases, message oriented middleware, legacy applications, networks and so on.Events are generally triggered from one of the business transactions. This requires the need for event generation and the business transaction to happen in the same transaction and moreover the event generation should not impair the performance of the business transaction. This is going to be a challenge in itself.

Most of the time the events are generated in one of the 2 ways

• Application Generated events
• Database Generated events

Database generated events are fine grained and App generated events are more complex or coarse grained.Various challenges with EDA include

• Need to have the business operation and the event generation happen with in the same transaction
• Reliable event delivery
• Events might need to be delivered in order
• Events might need to be delivered exactly once (at least once, at most once)
• Highly Scalable
• Granularity of the events
• Support for multiple platforms/languages
• Catalog for events
• How to fix the bad events that might have got created because of bad code, bad data.

To make sure that we do not lose any events, event generation and the corresponding business transaction need to be part of the same transaction. By having a queue that is backed by the same DB on which the business transaction occurs we make sure that the events are generated with the same transaction as the business activity. If our database is oracle then using OracleAQ might be a good option. Moreover by using Oracle AQ we can have the applications generate the events with in the same business transaction with out having to deal with the using of distributed transactions.
Options for capturing events at the Database end include

• DBMS data event adapters – Golden Gate
• Trigger generate events

For the data that might be modified by multiple applications, firing events from database triggers might be more reliable than firing events from applications. On the other end, database triggers may be too fine grained to capture some business events. Applications should fire those events. Also, for the data with clearly defined application ownership, firing events from application might be a good option

CAP theorem: Cap theorem is applicable to the current scenario of having to make the event generation and business transaction into a single transaction. Goals for a shared data system include.

Strong Consistency: all clients see the same view, even in presence of updates

High Availability: all clients can find some replica of the data, even in the presence of failures

Partition-tolerance: the system properties hold even when the system is partitioned

the theorem states that you can always have only two of the three CAP properties at the same time. The first property, Consistency, has to do with ACID systems, usually implemented through the two-phase commit protocol (XA transactions).

Systems that handle an incredibly huge number of transactions and data, always need some kind of system partitioning and must provide high availability. For example consider the case of a organizations like Amazon, Ebay.. the third and second CAP properties (Availability and Partitioning) are fixed, they need to sacrifice Consistency. It means they prefer to compensate or reconcile inconsistencies instead of sacrificing high availability, because their primary need is to scale well to allow for a smooth user experience/event generation.

The consistency of a two phase commit with out any performance costs can be obtained with out using distributed transactions by designing the event consumption operations to be idempotent and having the business logic in place to make sure order of events is not important. By using BASE transactions over ACID highly reliable and scalable systems can be designed.

BASE – BASICALLY AVAILABLE -SOFT STATE – EVENTUALLY CONSISTENCY

References

http://wiki/download/attachments/22454961/IEEE_Software_Design_2PC.pdf

http://camelcase.blogspot.com/2007/08/cap-theorem.html

http://activemq.apache.org/should-i-use-xa.html