Friday, March 20, 2009

Taking Memory and Thread dumps in JAVA

At times we will have to debug issues like Memory Leaks and deadlocks in java apps. Most of the issues related to memory leaks and thread locking can be debugged by taking the memory and thread dumps. I would like to document some of tools and tips that have been quite useful for me and for my fellow programmers who might need these.
Memory Dumps:
Most of the memory leaks result in the appserver or the java process crashing with an out of memory error. So just to be proactive and be able to get memory dump when that happens, it is a often a good idea to pass the following arguments to the JVM

 -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/path to file for heapdump/

As it might be obvious when these arguments are used when ever an java.lang.OutOfMemoryError occurs a heap dump is printed to the file mentioned in HeapDumpPath argument. If some userdefined process or command needs to be kicked of when java.lang.OutOfMemoryError occurs, then the following command is useful
-XX:OnOutOfMemoryError="<cmd args>

-XX:-PrintClassHistogram
Prints a histogram of class instances on Ctrl-Break
Memory dumps can also be taken on live java processes using jmap command. There are subtle variations in the way it can be used on 1.5 and 1.6
  1. Java 1.5: jmap -heap:format=b
    Works intermittently and definitely doesn't work more than once on the same process due to the problems with jmap in java 1.5
  2. Java 1.6: jmap -dump:format=b,file=heap.bin or jmap -dump:live,format=b,file=heap.bin
    Live option only gives the heap remained after running garbage collection. Jmap will create a file called heap.bin in the directoy jmap is run from
    Memory dumps can be taken from JConsole as well from 1.6.

Under certain circumstances, we may not be able to dump the heap to disk. we can instead, to get a histogram of the heap using the following command.Note that memory dump using jmap is only possible if it is executed by the same user that started the java process

Memory Dump Analysis:
Once we have the memory dumps, the next step is to analyze the dumps. One tool that was really useful for this purpose is
MAT: Eclipse based tool for analyzing the memory dumps.

http://www.eclipse.org/mat/

Thread Dump:
When applications result in any deadlocks , analyzing the threads will help us in getting to the root cause of the problem. Thread dumps can be take on linux using

1. kill -3 (must be executed by the same user that started the java process)
2. For Jboss process, we can take the thread dump from jmx-console as well (jboss.system->ServerInfo->listThreadDump operation)
3. Jstack

Analyzing Thread Dumps:
Once taking thread dumps , they can be analyzed and checked for deadlocks using Lockness
Lockness is a plugin for eclipse to analyze the thread dumps.

http://lockness.plugin.free.fr/home.php
-XX:-PrintConcurrentLocks
Print java.util.concurrent locks in Ctrl-Break thread dump. The jstack -l command provides equivalent functionality.
-XX:-TraceClassLoading
Traces the loading of classes.
-XX:-TraceClassLoadingPreorder
Trace all classes loaded in order referenced