Internet of Things,Big Data, Cloud Computing : The Perfect Match

What Is Internet of Things and How Does It Work?

Big Data is quickly becoming the next big asset for many organizations.  It would not be surprising for organizations to begin selling data of all types, including the metrics, knowledge, and insights gained from the data accumulated and analyzed. Riding on this wave of Big Data is Internet of Things (IoT). 

Technology, along with low storage costs, is making it easy for organizations to process large amounts of data; as a result, a new trend is emerging known as “capture it all.”  Capture it all means collecting as much data about your customers’ product usage and behaviors as possible because the data collected may be useful in the future.  At a time when organizations are seeing the benefits of Big Data, IoT provides innovative ways of capturing data that can enhance these benefits.

Internet of Things is the concept that things (animals, people or objects) with a unique identifier can automatically transfer data over a network without human interaction.   There are a number of devices that can be connected to the Internet to create a network of ‘things’ that communicate with each other to make intelligent decisions.  This is nothing new; the concept of Ubiquitous computing and sensor networks has been in use for a long time.

Why the Buzz Now?

Four technology trends are fueling the IoT revolution and renewing interest, they include:

1.     Big Data: Big Data’s success is making people realize the value of data, including the ways to identify valuable insights from data once considered junk.   IoT deployments can produce huge amounts of data, as sensors are constantly sensing stimuli and triggering real-time events.  It becomes relatively easy for the data to get accumulated over-time and the big data ecosystem or platform makes it easy to process these huge amounts of data. 

2.     Cloud Computing: With the advent of Cloud computing, computing power and data storage has become cheaper and easier to store and process data in Cloud.

3.     Ubiquitous connectivity: With the increase in the usage of smart phones with data plans along with the demand for connectivity to the Internet over smart phones, the infrastructure has been upgraded. Many IoT architectures are piggy backing on connectivity of the smart phone.

4.     Low cost sensors: The cost of Wi-Fi sensors and devices is in gradual decline. Standards like Near Field Communication (NFC), and iBeacon are becoming mainstream and supported by smart phones. As a result, App Developers can use them to creatively build IoT use cases. These improvements in the Bluetooth technology, Bluetooth low energy or BLE, are also becoming a catalyst to the IoT revolution.  In addition to the above, the ability of improved sensors to discreetly capture data is also a stimulus.  

 

How Internet of Things Is Used Today

Internet of Things deployments implemented right have the potential to become Big Data's killer Application.  Refer to figure 1 for the architecture used for typical IoT deployments.

 

Sensors that allow sensing of events are delivered to the mother ship on the cloud via servers connected to the Internet.  Data from these sensors is communicated via BLE and temporarily stored on the smartphone.  The use cases and potential for IoT and Big Data is endless as well as incredible.

A number of products based on IoT are getting launched and also receiving overwhelming response and adoption.  Wearable devices like Fitbit, Basis, Smartwatch from Samsung, and Qualcomm are playing an increasing part in IoT awareness.   Google became an early adaptor of IoT with its’ acquisition of the Nest Labs – Smart Thermostat.  Also, Google is gradually getting into home automation with their set-top boxes, NEST, Google Fiber, and Smart watches powered by Android OS.  Currently devices like Fitbit continuously and discreetly capture activity levels and sleep quality and then transmit the data to cloud using the smart phone. The sensors in turn communicate the data to smartphone through BLE.  Just imagine if there is a way to start tracking blood pressure, anxiety levels, stress levels, and heart rate in a similar discreet manner.  We can have personal data that is collected about ones self and then used by physicians in predicting a change in daily routine that can be causing current health issues.  For example, if a patient is unable to sleep properly, having the data collected historically could be invaluable in predicting what could have caused the current problem.  Tools like GOOGLE Nest Smart thermostat, Smart Smoke and CO alarm constantly track or monitor the environment in the house including information about lighting, humidity, daily behavior of the home's residents, temperature, and air quality.  Smartphone apps like Easily Do also discreetly record day to day activities of the owner by simply keeping track of the GPS on the cell phone. 

Potential Uses for Internet of Things

Imagine a world where devices can talk to each other, communicate and exchange information, and make intelligent decisions based on the data collected. For example, if you are coming home from a workout and, based on the data from your Fitbit and Easily Do, information is communicated to Google Nest thermostat that you would be home in a few minutes.  This information can be used to make your house more comfortable and cooler upon your arrival.  Taking this example a little further. What if data is captured from many people and made available to researchers (After anonymizing of Personal Identification Information).  This captured data can provide researchers and scientists with valuable data to study and find correlations between activities/actions that cause people to be susceptible to diseases.  This data can then provide feedback to users who may be susceptible to a disease and allow preventive measures to begin. In addition, trends/patterns may be identified that enable researchers to identify the reasons for diseases like heart attacks and Parkinson’s.  In addition, many of us may have seen sci-fi movies where an Artificial Intelligence system talks back and gives advice by analyzing a situation. Those days may not be far off due to the way technology trends in regards to IoT, Cloud computing, and Big Data are coming together.

Internet of Things’ Future

Gartner predicts that Internet of Things will affect every industry.  As a result, finding top analytics talent qualified to manage massive amounts of data will be difficult in the years ahead.  A yearlong research project conducted by Accenture shows that the United States is projected to create nearly 39,000 new jobs for analytics experts through 2015.  Only 23 percent if these jobs will be filled by qualified candidates. Cisco’s CEO, John Chambers, predicts that during the next decade the impact of Internet of Things will be 5 to 10 times greater than the Internet was on society and believes that IoT opens up a $19 Trillion opportunity during the same period.

IoT is here to stay and will make Big Data even bigger.  Our challenge, as IT professionals, is to discover innovative ways to use this technology that will enhance the general population’s lifestyle as well as benefit companies bottom line. 

Software development principles for an engineering manager,architect and developers

1. YAGNI - You Aren’t Gonna Need It :
   The YAGNI principle says that do not build something just because it might be needed later. Basically it says that do not build any code/features in the product that are not currently needed. This prevents the team from spending a lot of time planning for some grand, imaginary and unknown future scenarios. This will save time as most likely you do not need it or what you actually need is quite different from what we foresaw. This prevents teams from over engineering something based on what we think we might need later on. It is based on the idea that things will change with time. The principle saves you time twice. First, you don’t spend it now on something that you might don’t need. Second, when you reach the time when you need it, you don’t spend time in rewriting and refactoring but write once and well.By no means YAGNI is against planning and thinking in advance, it only encourages taking stuff into consideration NOW but postponing the implementation to later on
   Ask the YAGNI question

   "What are the chances that You Aren't Gonna Need It?"
   
   

2. KISS - Keep It Simple, Stupid:
   KISS is a principle that states that everything should be done as simple as possible. Applying KISS principle helps teams from over complicating problems. In my experience in the world of software development, developers tend to relieve the boredom of routine jobs by implementing an over complicating solution even when there exists a simple working solution. It feels great and exhilarating to do that until someone else needs to fix, maintain or modify the implementation. I must confess that I myself have done that many a time. One of my mentors once gave me a good analogy that I will not forget...an expert is some one who makes the job at hand look how simple it is and not vice versa. When ever we get back to our own code and start scratching our heads that sure sign that something got complicated than it had to be. Do not use all the fancy OOP, threading, frameworks just because you can.
   It is very simple to Implement the KISS principle. Whenever we come across a solution to a problem ask yourself " is it really the simplest way to do it? "
   Just remember
   
   "Some of the world's greatest algorithms are always the ones with the fewest lines of code. And when we go through the lines of code, we can easily understand them. The innovator of that algorithm broke down the problem until it was so easy to understand that he/she could implement it.Many great problem solvers were not great coders, but yet they produced great code! "
   
   For further reading on KISS
   http://people.apache.org/~fhanik/kiss.html
   
   

3. DRY - Don't Repeat Yourself:
   DRY is a principle that encourages to automate/extract tasks/code that we seem to be repeating again and again. DRY is aimed at reducing duplication and having a single point of maintenance. This principle can be applied in also not generalizing an implementation as a framework before we come across the need to repeat it again and again.
   
   
4. Premature optimization:
   Premature optimization is a term coined for the practice of trying to optimize the code to run faster when the code/functionality to be implemented is in a fluid state. Do not forget the well know statement " get it right then make it faster”. Do not get hung up on writing a fast-optimized code before getting the functionality right. The rationale is that it is most likely that the optimization that might be done prematurely might account to less than 3% of the actual bottleneck.The following quote from Donald Knuth in "Structure programming with go to statements" summarizes the problem perfectly
   " Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all-evil. Yet we should not pass up our opportunities in that critical 3%."
   
   
5. Principle of Least effort :
   The principle says that people and even well designed machines stop looking for better solutions once a solution that minimally matches the acceptance criteria is met. Applying the principle helps us to find the most effortless paths to solve problems. It promotes a quick and simple design over elaborate systems designed by committees.
   
   
6. The partew law of 80/20:
   The law in the field of software states that 80% of the user will use only 20% of your features. By understanding the rule and identifying the 20% of the features that 80% of the users use we can concentrate the resources and time on those 20% of the features to produce a better product.

Many senior developers and architects even though well versed with these principles tend to be applying them only when they feel like it and miss the spirit of these principles. As one of my mentor used to say the word is "Resume Driven development" , teams tend to over complicate , try new api's , frameworks and do a lot of wastage disregarding the business goals and objectives. This is the very reason and importance for the management team to be well versed in the field of the development to be able avoid this kind of wastage.

Best Leadership style

In our lives as a part of work , family, friends we come across leaders with different styles of operations and leadership qualities. Is there a one best leadership style ? ..Not really ..a leadership style in one situation might not go well in other situations. For example a crisis situation calls for a leader who is commanding and pace setting , where as the same leadership in other situations will be construed as a being a bully. Experts in psychology and organizational culture agree that a mix of leadership styles is the approach that gets the best results from the employees. Situational leadership is generally considered as the best way to go...Situational leadership calls for understanding the situation needs and dealing with appropriate style of leadership.

Richard Boyatzi and Daniel Goleman in their book Primal Leadership classify the leadership styles in

1. THE COMMANDING LEADER

Style: Gives clear directions and expects compliance.

Best used: In times of crisis, to kick-start a turnaround, and in dealing with "problem employees" who don't respond to other methods.

Most likely to say: "Do as I tel! you!"

2. THE PACE-SETTING LEADER

Style: Sets goals and expects them to be achieved.

Best used: For getting results from a motivated and competent team.

Most likely to say: "Go for itl"

3. THE VISIONARY LEADER

Style: Moves people towards a shared vision.

Best used: When a new strategic direction is needed.

Most likely to say: "Come with me."

4.THE COACHING LEADER

Style: Develops peopie for the future.

Best used: To help employees improve performance and develop long-term strengths.

Most likely to say: "Try this."

5.THE AFFILIATIVE LEADER

Style: Creates harmony and builds emotional bonds based on loyalty and trust.

Best used: To heal rifts in a team or to motivate people during stressful periods.

Most likely to say: "People come first."

6.THE DEMOCRATIC LEADER

Style: Forges consensus through participation.

Best used: To build "buy-in" and encourage Input.

Most likely to say: "What do you think?"

So what differentiates a leader from a great leader? It is the wisdom to decide on the best leadership style to employ in a given situation that differentiates a good leader from a great leader.

Access to ApplicationContext in Spring based Web applications

Recently in a Spring WS application , i had to get access to the Spring ApplicationContext and it turns out that it is not so well documented. Spring ApplicationContext can be created either by using
ClassPathXmlApplicationContext:
ClassPathXmlApplicationContext appContext = new ClassPathXmlApplicationContext(new String[] {"applicationContext.xml", "applicationContext-part2.xml"})

WebApplicationContext:

If we have access to the servletContext 

WebApplicationContext wac = WebApplicationContextUtils.getRequiredWebApplicationContext(servletContext);

In a web/Spring WS app, if we do not have access to the servletContext and if we would to like use the existing ApplicationContext , the following approach might be the way to go:


Create a class ApplicationContextContainer that implements ApplicaionContextAware and have it be defined as a
bean in Spring context xml file.

import org.springframework.beans.BeansException;

import org.springframework.context.ApplicationContext;

import org.springframework.context.ApplicationContextAware;

public class ApplicationContextContainer implements ApplicationContextAware {

private ApplicationContext appContext;         

 
@Override   

public void setApplicationContext(ApplicationContext applicationContext)
throws BeansException {

       this.appContext = applicationContext;     
 
} 
        

public ApplicationContext getAppContext(){

        return appContext;   

} 

}

Create a bean definition for the above class in the spring context file


 <bean id="applicationContextProvider"  class="com.test.ApplicationContextProvider"/>



And by injecting the bean applicationContextProvider, the spring ApplicationContext is accessible in the rest of the application.


Asynchronous Messaging Architecure as an Integration Solution

Fundamental challenges of integration solutions include most of the fallacies of distributed application
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

We can summarize the major challenges of integration solutions into to the following

• Networks are unreliable
• Networks are slow
• Any two applications are different
• Change is inevitable
  

When we are trying to integrate two Apps A and B ,there are 4 main approaches to do this and overcome the above mentioned challenges

1. File Transfer : One application writes a file and another application reads from the file. The applications will need to agree on filename,location and format
2. Shared Database: Multiple applications share the same database schema, the integration happens through the database.
3. Remote Procedure Invocation/ Web services: One application exposes one sort of functionality that can be accessed remotely by other application as remote procedure. This could as well be a web service. Communication occurs in real time and synchronously.
4. Messaging: One application publishes a message to a common message channel. Apps need to agree upon message channel and format. Communication is asynchronous.

The issues with approaches in 1,2,3 include

• Apps are tightly coupled.
• Apps A and B need to scaled together
• If A needs high availability then even B needs to made highly available.
• The model of communication involves the applications to be polling for the availability of the data from the other application.

Messaging is more immediate than file transfer, better encapsulated than a shared database and more reliable than a web service/RPI.Advantages of going with the messaging approach include:

• Apps involved in integration (A,B) are loosely coupled .
• Apps can be scaled independently of each other
• Apps can functionally be available independent of each other. App A can be up and functional even when B is down. App A will queue up the data that is required by B , When B is up it start consuming the events from A.
  
• Apps do not have to be build to support peak load as the asynchronous messaging allows the processing to be spread out over time. Asynchronous dampens the load.
• Throttling Ability.
  
• Reliable Communication - reliable and guaranteed delivery
• Platform/Language agnostic integration

Asynchronous Messaging Architectures have in turn have its own challenges.

1. It is a complex programming model. Asynchronous messaging requires developers to work with an event driven programming model.
2. Message sequence issues - Message channels guarantee message delivery but they do not guarantee when message will be delivered. Might require some logic to re-sequence the events at the consuming end if IN ORDER Delivery is a required SLA.
3. Synchronous Scenarios- There might be scenarios where synchronous is required for better user experience. Intelligent workaround schemes might be required to support it.
4. Performance-- Messaging is best suited to keep the system in sync after the initial data replication. Messaging is not suited from the performance aspect to be used as mechanism to replicate data.
5. Limited platform support. This methodology is not yet mainstream and the support and best practices are in limited availability.
6. Vendor Lock-in: Messaging brokers used might result in a vendor-lockin problem. But as along as we stick to to the standards like JMS and stay away from using the broker specific features most of the vendor lock problems can be mitigated.
   

Why should we avoid distributed transactions ?

A distributed transaction is a transaction that spans multiple resources. A distributed transactions are also at times referred to as XA transactions. XA transaction is X/Open group specification on distributed transactions. XA transaction is required to guarantee the ACID (Atomicity,Consistency,Isolation,Durability) properties of a transaction when the transaction spans multiple resources - be it multiple databases,database and JMS connection, database and file system or any resource. XA transactions require transaction manager to co-ordinate the multiple resource managers involved in the transaction. Transaction Managers co-ordinate the resource managers involved using 2-Phase Commit protocol (2-PC).
Usage of distributed/XA transactions in turn come with its own issues and complexities.Some of the problems/complexities with distributed transactions include

1. 2-PC protocol is very chatty protocol and does a lot of logging to be able to recover from any failure scenario.
2. Too much overhead to 99.9% of the cases to handle less than 0.1% of the failure/exception cases
   
3. Increases the amount of time the locks are held on the databases. This increases the chances for deadlocks on the database. This also lowers the overall performance of the system.
4. Distributed transactions are sort of the bane of scalability. It sort of grinds the entire system to halt by adding overhead to each and every transaction.
   
5. Availability of the Systems goes down. When using distributed transactions,the completion of a distributed transaction is now a product of the availability of two different systems and there by the the total availability of the system goes down.
6. XA/Distributed transaction configuration is complicated and is difficult to test to make sure the configuration is configured correctly. (Many Java Developers tend to believe that using JTA implementation of transaction manager will take care of a XA transactions and tend to forget configuring the resources as XA resources. Many a time people end up using JTA even while dealing with single resource. ). Many people get XA enlisted resources when they don't have to and many a time the applications would perform better if they weren't unnecessarily using XA.
   
   Based on the various issues mentioned above it is very much recommended that the use of XA transactions be avoided as much as we can. So how do we maintain the ACID of a transaction that goes against multiple resources !!!? In most cases intelligent recovery business scenarios supported by the system's management interface will eliminate a need for XA. The possible ways to circumvent the need for distributed transactions could be a seperate post in itself...stay tuned..:).

How to generate Events ?

Event Driven Architecture or EDA is the handling of events as they occur and the transfer of events among the systems in a loosely coupled way. The usage of events and EDA can replace the need for integrating the systems at the database level. With the usage of EDA's the systems are loosely coupled and information can be passed between the systems in real time, rather than relying on ETL processes or data transfers between the systems. The hardest part of the implementing an EDA is capturing the events
Events can be generated by tapping into web sites, files, databases, message oriented middleware, legacy applications, networks and so on.Events are generally triggered from one of the business transactions. This requires the need for event generation and the business transaction to happen in the same transaction and moreover the event generation should not impair the performance of the business transaction. This is going to be a challenge in itself.

Most of the time the events are generated in one of the 2 ways

• Application Generated events
• Database Generated events

Database generated events are fine grained and App generated events are more complex or coarse grained.Various challenges with EDA include

• Need to have the business operation and the event generation happen with in the same transaction
• Reliable event delivery
• Events might need to be delivered in order
• Events might need to be delivered exactly once (at least once, at most once)
• Highly Scalable
• Granularity of the events
• Support for multiple platforms/languages
• Catalog for events
• How to fix the bad events that might have got created because of bad code, bad data.

To make sure that we do not lose any events, event generation and the corresponding business transaction need to be part of the same transaction. By having a queue that is backed by the same DB on which the business transaction occurs we make sure that the events are generated with the same transaction as the business activity. If our database is oracle then using OracleAQ might be a good option. Moreover by using Oracle AQ we can have the applications generate the events with in the same business transaction with out having to deal with the using of distributed transactions.
Options for capturing events at the Database end include

• DBMS data event adapters – Golden Gate
• Trigger generate events

For the data that might be modified by multiple applications, firing events from database triggers might be more reliable than firing events from applications. On the other end, database triggers may be too fine grained to capture some business events. Applications should fire those events. Also, for the data with clearly defined application ownership, firing events from application might be a good option

CAP theorem: Cap theorem is applicable to the current scenario of having to make the event generation and business transaction into a single transaction. Goals for a shared data system include.

Strong Consistency: all clients see the same view, even in presence of updates

High Availability: all clients can find some replica of the data, even in the presence of failures

Partition-tolerance: the system properties hold even when the system is partitioned

the theorem states that you can always have only two of the three CAP properties at the same time. The first property, Consistency, has to do with ACID systems, usually implemented through the two-phase commit protocol (XA transactions).

Systems that handle an incredibly huge number of transactions and data, always need some kind of system partitioning and must provide high availability. For example consider the case of a organizations like Amazon, Ebay.. the third and second CAP properties (Availability and Partitioning) are fixed, they need to sacrifice Consistency. It means they prefer to compensate or reconcile inconsistencies instead of sacrificing high availability, because their primary need is to scale well to allow for a smooth user experience/event generation.

The consistency of a two phase commit with out any performance costs can be obtained with out using distributed transactions by designing the event consumption operations to be idempotent and having the business logic in place to make sure order of events is not important. By using BASE transactions over ACID highly reliable and scalable systems can be designed.

BASE – BASICALLY AVAILABLE -SOFT STATE – EVENTUALLY CONSISTENCY

References

http://wiki/download/attachments/22454961/IEEE_Software_Design_2PC.pdf

http://camelcase.blogspot.com/2007/08/cap-theorem.html

http://activemq.apache.org/should-i-use-xa.html

Taking Memory and Thread dumps in JAVA

At times we will have to debug issues like Memory Leaks and deadlocks in java apps. Most of the issues related to memory leaks and thread locking can be debugged by taking the memory and thread dumps. I would like to document some of tools and tips that have been quite useful for me and for my fellow programmers who might need these.
Memory Dumps:
Most of the memory leaks result in the appserver or the java process crashing with an out of memory error. So just to be proactive and be able to get memory dump when that happens, it is a often a good idea to pass the following arguments to the JVM

 -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/path to file for heapdump/

As it might be obvious when these arguments are used when ever an java.lang.OutOfMemoryError occurs a heap dump is printed to the file mentioned in HeapDumpPath argument. If some userdefined process or command needs to be kicked of when java.lang.OutOfMemoryError occurs, then the following command is useful

-XX:OnOutOfMemoryError="<cmd args>

-XX:-PrintClassHistogram 

Prints a histogram of class instances on Ctrl-Break
Memory dumps can also be taken on live java processes using jmap command. There are subtle variations in the way it can be used on 1.5 and 1.6

1. Java 1.5: jmap -heap:format=b
   Works intermittently and definitely doesn't work more than once on the same process due to the problems with jmap in java 1.5
2. Java 1.6: jmap -dump:format=b,file=heap.bin or jmap -dump:live,format=b,file=heap.bin
   Live option only gives the heap remained after running garbage collection. Jmap will create a file called heap.bin in the directoy jmap is run from
   Memory dumps can be taken from JConsole as well from 1.6.
   

Under certain circumstances, we may not be able to dump the heap to disk. we can instead, to get a histogram of the heap using the following command.Note that memory dump using jmap is only possible if it is executed by the same user that started the java process

Memory Dump Analysis:
Once we have the memory dumps, the next step is to analyze the dumps. One tool that was really useful for this purpose is
MAT: Eclipse based tool for analyzing the memory dumps.

http://www.eclipse.org/mat/

Thread Dump:
When applications result in any deadlocks , analyzing the threads will help us in getting to the root cause of the problem. Thread dumps can be take on linux using

1. kill -3 (must be executed by the same user that started the java process)
2. For Jboss process, we can take the thread dump from jmx-console as well (jboss.system->ServerInfo->listThreadDump operation)
3. Jstack

Analyzing Thread Dumps:
Once taking thread dumps , they can be analyzed and checked for deadlocks using Lockness
Lockness is a plugin for eclipse to analyze the thread dumps.

http://lockness.plugin.free.fr/home.php

-XX:-PrintConcurrentLocks

Print java.util.concurrent locks in Ctrl-Break thread dump. The jstack -l command provides equivalent functionality.

-XX:-TraceClassLoading

Traces the loading of classes.

-XX:-TraceClassLoadingPreorder

Trace all classes loaded in order referenced