2. Java Multiplatform Support offering
This support offering is designed to provide production support for shipping releases of Java technology-based applications using Sun's Java 2 Runtime Environments and distributed to end users in heterogeneous environments. The support offering helps on optimizing application performance and helps reduce time spent keeping applications up and running. The highest level of this support offering can include accelerated access to an engineer and emergency software fixes. More details on this support offering are available here :http://www.sun.com/service/sunps/jdc/javamultiplatform.html

Community Support

Community support can often be obtained using the Java Technology Forums. The forums provide a way to share information and locate solutions to problems. The forums are here: http://forums.java.sun.com
1 Diagnostics Tools and Options

1.1 Introduction

This chapter introduces the various diagnostic and monitoring tools which can be used with J2SE 5.0. The tools described here include command line utilities, command line options, and log files. In almost all cases the command line utilities described in this chapter are either included in the Java 2 Platform Standard Edition Development Kit (JDK 5.0), or are operating system tools and utilities. Although the JDK 5.0 command line utilities are included in the JDK download, it is important to note that they can be used to diagnose issues and monitor applications that are deployed with the Java 2 Platform Standard Edition Runtime Environment 5.0 (JRE 5.0). In general, the diagnostic tools and options described in this chapter use various mechanisms to obtain the information they report. In many cases the mechanisms are specific to the virtual machine implementation (5.0 in the case of this document), operating system, and version of each. Consequently, there is some overlap of the information reported by some of the tools. This should be viewed in the context of the various problems and issues for which these tools are intended. In many cases only a subset of the tools will be applicable to an issue at a particular point in time.
1.1.1 Caveats and Other Notes
Some of the command line utilities described in this chapter are experimental. The jstack, jinfo, and jmap utilities are examples of utilities that are experimental. These utilities are subject to change in future JDK releases, and may not be included in future releases. The format of log files and other output from command line utilities or options is version specific. For example, if you develop a script that relies on the format of the fatal error log (hs_err_pid<pid>.log file) then this script may cease to work as expected if the format of the log file changes in the future. Command line options that are prefixed with -XX are Java HotSpotTM Virtual Machine specific options. In general, most -XX options are unsupported, undocumented, and were originally included for the purposes of testing components of the HotSpot Virtual Machine during its development. However many -XX options are important for performance tuning and diagnostic purposes, and are therefore described in this chapter. In some cases, the command tool utilities described here are not included in the JDK release on all operating systems. For example, the jstack, jinfo, and jmap command line utilities are included in the JDK release for Solaris and Linux only. In addition to operating system specific utilities we also describe a number of diagnostic features and tools that are specific to the Solaris 10 Operating Environment. Solaris 10 has many advanced diagnostic features that are usable in production environments, and many of the native tools are capable of providing information that is specific to the Java Runtime Environment.

$ jmap -heap 19846 Attaching to process ID 19846, please wait. Debugger attached successfully. Server compiler detected. JVM version is 1.5.0-rc-b63 using thread-local object allocation. Parallel GC with 2 thread(s) Heap Configuration: MinHeapFreeRatio MaxHeapFreeRatio MaxHeapSize NewSize MaxNewSize OldSize NewRatio SurvivorRatio PermSize MaxPermSize = = = = = = = = = = 536870912 (512.0MB) 2228224 (2.125MB) 4294901760 (4095.9375MB) 1441792 (1.375MB) 16777216 (16.0MB) 67108864 (64.0MB)
Heap Usage: PS Young Generation Eden Space: capacity = 9240576 (8.8125MB) used = 9159544 (8.735221862792969MB) free = 81032 (0.07727813720703125MB) 99.12308496786348% used From Space: capacity = 2555904 (2.4375MB)
used = 2431992 (2.3193283081054688MB) free = 123912 (0.11817169189453125MB) 95.1519305889423% used To Space: capacity = 2686976 (2.5625MB) used = 0 (0.0MB) free = 2686976 (2.5625MB) 0.0% used PS Old Generation capacity = 25165824 (24.0MB) used = 2426464 (2.314056396484375MB) free = 22739360 (21.685943603515625MB) 9.641901652018229% used PS Perm Generation capacity = 16777216 (16.0MB) used = 9523600 (9.082412719726562MB) free = 7253616 (6.9175872802734375MB) 56.765079498291016% used

1.3.2 Heap Histogram

The -histo option can be used to obtain a class-wise histogram of the heap. For each class, it prints the number of objects, memory size in bytes, and fully qualified class name. Note that internal classes in the HotSpot VM are prefixed with an *. The histogram is useful when trying to understand how the heap is used. To get the size of an object you need to divide the total size by the count of that object type. The following shows an example of jmap -histo output:
$ jmap -histo 19846 Attaching to process ID 19846, please wait. Debugger attached successfully. Server compiler detected. JVM version is 1.5.0-rc-b63 Iterating over heap. This may take a while. Object Histogram: Size Count Class description ------------------------------------------------------char[] * ConstMethodKlass * MethodKlass java.util.HashMap$ValueIterator * SymbolKlass java.lang.String byte[] * ConstantPoolKlass 80235240186 org.apache.catalina.Container[] 6730721535 * ConstantPoolCacheKlass 6518321761 * InstanceKlassKlass 5524564326 int[] 3482404353 java.lang.reflect.Method 3189604430 org.apache.naming.resources.FileDirContext$FileResourceAttributes
2861363274 java.lang.Object[] 2321205815 java.lang.String[] 2178009075 java.util.HashMap$Entry 2020002005 java.util.HashMap$Entry[] 1760962393 short[] 1747681986 java.lang.Class 142808495 * MethodDataKlass 1393842771 java.lang.Object[] 1070246689 java.io.File java.util.Hashtable$Entry java.util.HashMap java.lang.StringBuilder java.util.Hashtable$Entry[] java.lang.Class[] java.util.LinkedHashMap$Entry * ObjArrayKlassKlass java.util.ArrayList java.lang.ref.SoftReference org.apache.xerces.dom.DeferredAttrImpl java.lang.ref.WeakReference java.io.ExpiringCache$Entry org.apache.commons.modeler.AttributeInfo javax.management.modelmbean.DescriptorSupport$CaseIgnoreString java.util.Hashtable java.beans.MethodDescriptor java.lang.reflect.Method[] java.lang.reflect.Constructor javax.management.modelmbean.ModelMBeanAttributeInfo[] org.apache.xerces.xni.QName org.apache.xerces.dom.DeferredElementImpl java.util.Vector org.apache.catalina.core.StandardWrapper java.nio.DirectByteBuffer javax.management.ObjectName$Property org.apache.xerces.dom.DeferredTextImpl java.beans.PropertyDescriptor sun.misc.Cleaner [. more lines removed here to reduce output.] sun.reflect.GeneratedMethodAccessor31 Heap traversal tool 7.12 seconds.

1.4 jstack

The jstack command-line utility is included in the Solaris and Linux releases of the JDK. It is not included in the JDK5.0 release on Windows. The utility attaches to the specified process (or core file) and prints the stack traces of all threads that are attached to the virtual machine (this includes Java threads and VM internal threads). A stack trace of all threads can be useful when trying to diagnose a number of issues such as deadlocks or hangs. In many cases a thread dump can be obtained by pressing Ctrl-\ at the application console (standard input) or by sending the process a QUIT signal (section 1.16). Thread dumps can also be obtained programmatically using the Thread.getAllStackTraces method, or in the debugger using the debugger option to print all thread stacks (the where command in the case of the jdb sample debugger). In these examples the VM process must be in a state where it can execute code. In rare cases (for example if you encounter a bug in the thread library or HotSpot VM) this may not be possible, but it may be possible with the jstack utility as it attaches to the process using an operating system interface. Here is example output from the jstack command to demonstrate how the output looks :
$ jstack 7034 Debugger attached successfully. Server compiler detected. JVM version is 1.5.0-rc-b63 Thread t@44: (state = BLOCKED) - java.lang.Object.wait(long) (Interpreted frame) - java.lang.Object.wait(long) (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run() @bci=8, line=549 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame) Thread t@43: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) (Interpreted frame) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=50, line=450 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame) - org.apache.jk.common.ChannelSocket.accept(org.apache.jk.core.MsgContext) @bci=12, line=278 (Interpreted frame) - org.apache.jk.common.ChannelSocket.acceptConnections() @bci=71, line=572 (Interpreted frame) - org.apache.jk.common.SocketAcceptor.runIt(java.lang.Object[]) @bci=4, line=758 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=174, line=666 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame) Thread t@42: (state = BLOCKED) - java.lang.Object.wait(long) (Interpreted frame) - java.lang.Object.wait(long) (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=19, line=642 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame) Thread t@41: (state = BLOCKED) - java.lang.Object.wait(long) (Interpreted frame) - java.lang.Object.wait(long) (Interpreted frame) - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=19, line=642 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame) [. more lines removed here to reduce output.]

$ jps 16217 MyApplication 16342 jps
The utility lists the virtual machines for which the user has access rights. This is determined by operating system specific access control mechanisms. On Solaris, for example, if a non-root user executes the jps utility then it lists the virtual machines that were started with that user's uid. In addition to listing the process id the utility provides options to output the arguments passed to the application's main method, the complete list of VM arguments, and the full package name of the application's main class. jps can also list processes on a remote system if the remote system is running the jstat daemon (jstatd). The documentation for the utility can be found here : http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html The document includes examples that obtain the process status from a remote host. The utility is included in the JDK download for all operating system platforms supported by Sun. However, the HotSpot instrumentation is not accessible on Windows 98 or Windows ME. In addition the instrumentation may not be accessible on Windows if the temporary directory is on a FAT32 file system1.
1 Workarounds to this issue can be found in the FAQ at http://developers.sun.com/dev/coolstuff/jvmstat/faq.html

1.7 jstat

The jstat utility uses the built-in instrumentation in the HotSpot VM to provide information on performance and resource consumption of running applications. The tool can be used when diagnosing performance issues, and in particular issues related to heap sizing and garbage collection. The jstat utility does not require the VM to be started with any special options. The built-in instrumentation in the HotSpot VM is enabled by default. The utility is included in the JDK download for all operating system platforms supported by Sun. However, the instrumentation is not accessible on Windows 98 or Windows ME2. Following are the jstat utility options :

-class prints statistics on the behavior of the class loader. -compiler prints statistics of the behavior of the HotSpot compiler. -gc prints statistics of the behavior of the garbage collected heap. -gccapacity prints statistics of the capacities of the generations and their corresponding spaces. -gccause prints the summary of garbage collection statistics (same as -gcutil), with the cause of the last and current (if applicable) garbage collection events. -gcnew prints statistics of the behavior of the new generation. -gcnewcapacity - prints statistics of the sizes of the new generations and its corresponding spaces. -gcold prints statistics of the behavior of the old and permanent generations. -gcoldcapacity prints statistics of the sizes of the old generation. -gcpermcapacity print statistics of the sizes of the permanent generation. -gcutil prints a summary of garbage collection statistics. -printcompilation prints HotSpot compilation method statistics.

-------- Get sample cpu information every 20 millisec, with a stack depth of 3: java -agentlib:hprof=cpu=samples,interval=20,depth=3 classname - Get heap usage information based on the allocation sites: java -agentlib:hprof=heap=sites classname Notes ----- The option format=b cannot be used with monitor=y. - The option format=b cannot be used with cpu=old|times. - Use of the -Xrunhprof interface can still be used, e.g. java -Xrunhprof:[help]|[<option>=<value>,.] will behave exactly the same as: java -agentlib:hprof=[help]|[<option>=<value>,.] Warnings -------- This is demonstration code for the JVMTI interface and use of BCI, it is not an official product or formal part of the J2SE. - The -Xrunhprof interface will be removed in a future release. - The option format=b is considered experimental, this format may change in a future release.
By default, heap profiling information (sites and dump) is written out to java.hprof.txt (ASCII) in the current working directory. The output is normally generated when the VM exits, although this can be disabled by setting the dump on exit option to n (doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris and Linux a profile is also generated when a QUIT signal is received (kill -QUIT <pid>). If Ctrl-\ or Ctrl-Break is pressed multiple times then multiple profiles are generated to the one file. The output in most cases will contain ID's for traces, threads, and objects. Each type of ID will typically start with a different number than do the other ID's. For example, traces might start with 300000.
Heap Allocation Profiles (heap=sites)
Following is the heap allocation profile generated by running the Java compiler (javac) on a set of input files. Only parts of the profiler output are shown here.
$ javac -J-agentlib:hprof=heap=sites Hello.java SITES BEGIN (ordered by live bytes) Fri Feb 6 13:13:percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name 1 44.13% 44.13% 301926 java.util.zip.ZipEntry 2 8.83% 52.95% 301927 com.sun.tools.javac.util.List 3 5.18% 58.13% 300996 byte[] 4 5.18% 63.31% 300995 com.sun.tools.javac.util.Name[]
A crucial piece of information in the heap profile is the amount of allocation that occurs in various parts of the program. The SITES record above tells us that 44.13% of the total space was allocated for java.util.zip.ZipEntry objects. A good way to relate allocation sites to the source code is to record the dynamic stack traces that led to the heap allocation. Following is another part of the profiler output that illustrates the stack traces referred to by the four allocation sites in output shown above.

Each record is a ROOT, OBJ, CLS, or ARR to represent a root, an object instance, a class, or an array. The hexidecimal numbers are identifiers assigned by HPROF. They are used to show the references from an object to another object. For example, in the above sample, the java.lang.Thread instance 50000114 has a reference to its thread group (50008c6c) and other objects. In general, as the output is very large, it is necessary to use a tool to visualize or process the output of a heap dump. The Heap Analysis Tool (HAT) is one such tool (see section 1.11).
CPU Usage Sampling Profiles (cpu=samples)
HPROF can collect CPU usage information by sampling threads. Following is part of the output collected from a run of the javac compiler.
$ javac -J-agentlib:hprof=cpu=samples Hello.java CPU SAMPLES BEGIN (total = 462) Fri Feb 6 13:33:rank self accum count trace method 1 49.57% 49.57% java.util.zip.ZipFile.getNextEntry 2 6.93% 56.49% java.util.zip.ZipEntry.initFields 3 4.76% 61.26% java.lang.ClassLoader.defineClass2.81% 64.07% java.util.zip.ZipFile.freeEntry 5 1.95% 66.02% java.util.Vector.addElement 6 1.73% 67.75% java.util.zip.ZipFile.getEntry 7 1.52% 69.26% java.lang.ClassLoader.findBootstrapClass 8 0.87% 70.13% com.sun.tools.javac.main.JavaCompiler.<init> 9 0.65% 70.78% java.util.zip.ZipFile.open 10 0.65% 71.43% com.sun.tools.javac.main.JavaCompiler.<init>. CPU SAMPLES END
The HPROF agent periodically samples the stack of all running threads to record the most frequently 34
active stack traces. The count field above indicates how many times a particular stack trace was found to be active. These stack traces correspond to the CPU usage hot spots in the application.
CPU Usage Times Profile (cpu=times)
HPROF can collect CPU usage information by injecting code into every method entry and exit, keeping track of exact method call counts and the time spent in each method. This uses Byte Code Injection (BCI) and runs considerably slower than cpu=samples. Following is part of the output collected from a run of the javac compiler.
$ javac -J-agentlib:hprof=cpu=times Hello.java CPU TIME (ms) BEGIN (total = 2082665289) Fri Feb 6 13:43:rank self accum count trace method 1 3.70% 3.70% com.sun.tools.javac.Main.compile 2 3.64% 7.34% com.sun.tools.javac.main.Main.compile 3 3.64% 10.97% com.sun.tools.javac.main.Main.compile 4 3.11% 14.08% com.sun.tools.javac.main.JavaCompiler.compile 5 2.54% 16.62% com.sun.tools.javac.jvm.ClassReader.listAll 6 2.53% 19.15% com.sun.tools.javac.jvm.ClassReader.list 7 2.03% 21.18% com.sun.tools.javac.comp.Enter.main 8 2.03% 23.21% com.sun.tools.javac.comp.Enter.complete 9 1.68% 24.90% com.sun.tools.javac.comp.Enter.classEnter 10 1.68% 26.58% com.sun.tools.javac.comp.Enter.classEnter. CPU TIME (ms) END
Here the count represents the true count of the times this method was entered, and the percentages represent a measure of thread CPU time spent in those methods.

Following is another exmaple of output that arises when something other than a jfieldID is provided to a JNI functions that expects a jfieldID:
FATAL ERROR in native method: Instance field not found in JNI get/set field operations at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) - locked <0xf082f290> (a java.net.PlainSocketImpl) at java.net.ServerSocket.bind(ServerSocket.java:318) at java.net.ServerSocket.<init>(ServerSocket.java:185) at jvm003a.<init>(jvm003.java:190) at jvm003a.<init>(jvm003.java:151) at jvm003.run(jvm003.java:51) at jvm003.main(jvm003.java:30)
Following are other examples of problems that -Xcheck:jni can help diagnose : 1. Cases where the JNI environment for the wrong thread is used. 2. Cases where an invalid JNI reference is used. 3. Cases when a reference to a non-array type is provided to a function that requires an array type. 4. Cases where a non-static field ID is provided to a function that expects a static field ID. 5. Cases where a JNI call is made with an exception pending. In general, all errors detected by -Xcheck:jni are fatal errors. That is, the error is printed and the VM is aborted when the error is detected. One exception to this is a non-fatal warning that is printed when a JNI call is made within a JNI critical region. When this arises the following warning is printed :
Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
A JNI critical region arises when native code uses the JNI GetPrimitiveArrayCritical or GetStringCritical functions to obtain a reference to an array or string in the Java heap. The reference is held until the native code calls the corresponding release function. In between the get and release is called a JNI critical section and during that time the HotSpot VM cannot bring the VM to a state that allows garbage collection to occur. The general recommendation is that other JNI functions should not be used when in a JNI critical section, and in particular any JNI function that blocks could potentially cause a deadlock. The warning printed by -Xcheck:jni is thus an indication of a potential issue; it does not always indicate an application bug.

1.17.4 -verbose:jni

The -verbose:jni option enables logging of JNI. Specifically, when a JNI/native method is resolved the HotSpot VM prints a trace message to the application console (standard output). It also prints a trace message when a native method is registered using the JNI RegisterNative function. The -verbose:jni option may be useful when trying to diagnose issues with applications that use native libraries.
1.17.5 JAVA_TOOL_OPTIONS Environment Variable
In many environments the command line to start the application is not readily accessible. This often arises with applications that use embedded VMs (meaning they use the JNI Invocation API to start the VM), or the startup is deeply nested in scripts. In these environments the JAVA_TOOL_OPTIONS environment variable may be useful to augment a command line. This environment variable is primarily intended to support the initialization of tools, specifically the launching of native or Java programming language agents using the -agentlib or -javaagent options. The environment variable is processed at the time of the JNI Invocation API create VM call; options processed by a launcher (such as VM selection options) will not be handled. When set, the JNI_CreateJavaVM function (in the JNI Invocation API) will prepend the value of the environment variable to the options supplied in its JavaVMInitArgs argument. In some cases this option is disabled for security reasons; for example, on Solaris the option is disabled when the effective user or group ID differs from the real ID. Here is an example where we set the environment variable so that the HPROF profiler is launched when the application is started:

2.1.3 Diagnosing Leaks in Native Code
There are several techniques used to find and isolate native code memory leaks. A very common practice is to track all allocation and free calls of the native allocations. This can be a fairly simple process or a very sophisticated one. Many products over the years have been built up around the tracking of native heap allocations and the use of that memory. Tools like Purify and Sun's dbx RTC (Run Time Checking) functionality can be used to find these leaks in normal native code situations and also find any access to native heap memory that represents assignments to uninitialized memory or accesses to freed memory. Not all these types of tools will work with Java applications that use native code, and usually these tools are very platform specific. Since the virtual machine dynamically creates code at runtime, these tools can become confused and fail to run at all, or give false information. Check with your tool vendor to make sure the version of the tool works with the version of the virtual machine you are using. Many simple and portable native memory leak detecting examples can be found at http://sourceforge.net/. Most of these libraries and tools assume you can re-compile or edit the source of the application and place wrapper functions over the allocation functions. The more powerful of these allow you to run your application unchanged by interposing over these allocation functions dynamically. This is the case with the Solaris 10 library libumem.so (2.1.3.2). In general there is no single ideal solution for all platforms. Anyone writing a JNI library would probably be wise to create some kind of localized way to make sure your library doesn't leak memory using a simple wrapper approach. 74
An easy localized allocation tracking approach for a JNI library would be something like:
#include <stdlib.h> #define malloc(n) debug_malloc(n, __FILE__, __LINE__) #define free(p) debug_free(p, __FILE__, __LINE__)
The above would need to be defined in all source files. And then you could use these functions to watch for leaks:
/* Total bytes allocated */ static int total_allocated; /* Memory alignment is important */ typedef union { double d; struct {size_t n; char *file; int line;} s; } Site; void * debug_malloc(size_t n, char *file, int line) { char *rp; rp = (char*)malloc(sizeof(Site)+n); total_allocated += n; ((Site*)rp)->s.n = n; ((Site*)rp)->s.file = file; ((Site*)rp)->s.line = line; return (void*)(rp + sizeof(Site)); } void debug_free(void *p, char *file, int line) { char *rp; rp = ((char*)p) - sizeof(Site); total_allocated -= ((Site*)rp)->s.n; free(rp); }
The JNI library would then need to periodically (or at shutdown) check total_allocated to make sure it made sense. The above code could also be expanded to actually save in a linked list the allocations that remained and report where the leaked memory was allocated. This is a localized and portable way to track memory allocations in a single set of sources. You would need to make sure debug_free() was only called with a pointer that came from debug_malloc(), and you would also need to create similar functions for realloc(), calloc(), strdup(), etc. if they were used. A more global way to look for native heap memory leaks would involve interposition of the library calls for the entire process. Most systems include some form of global allocation tracking support.

2.2.1 Format of the Fatal Error Log
Section 1.13.1 provided a high level list of the information that is written to the fatal error log. Here the error log is described in more detail. The error log is a text file consisting of a header that provides a brief description of the crash. This is followed by sections with thread, process, and system information. It should be noted that the format of the fatal error log described here is based on JDK5.0. In future releases the format may be different. If you develop scripts or tools that depend on the format then these scripts or tools may cease to work if the format is changed. 2.2.1.1 Header At the beginning of every HotSpot error log file is a brief description about the problem. It is also printed to standard output and may show up in the application's output log. Following is a sample header from a crash:
# # # # # # # # An unexpected error has been detected by HotSpot Virtual Machine: SIGSEGV (0xb) at pc=0x417789d7, pid=21139, tid=1024 Java VM: Java HotSpot(TM) Client VM (1.5.0-beta2-b37 mixed mode) Problematic frame: C [libNativeSEGV.so+0x9d7]
This example shows that the VM died on an unexpected signal. The second line describes the signal type, program counter (pc) that caused the signal, process ID and thread ID.
SIGSEGV (0xb) at pc=0x417789d7, pid=21139, tid=1024 | | | | +--- thread id | | | +------------- process id | | +--------------------------- instruction pointer | +--------------------------------------- signal number +---------------------------------------------- signal name
Next line is the VM version which indicates if the Client VM or Server VM is used, and it will also indicate if the application is run in mixed or interpreted mode, and it will indicate if class file sharing is enabled.
# Java VM: Java HotSpot(TM) Client VM (1.5.0-rc-b63 mixed mode, sharing)
The version information is followed by the function frame that caused the crash:
# Problematic frame: # C [libNativeSEGV.so+0x9d7] | +-- Same as pc, but represented as library name and offset. | For position independent libraries (JVM and most shared | libraries), it's possible to inspect the instructions | that caused the crash without a debugger or core file. | Just use a disassembler to dump instructions near the | offset. +----------------- Frame type

In this example the C indicates a native C frame. The following table lists the possible frame types: Frame C J j V v Description Native C frame Other frame types including compiled Java frames Interpreted Java frames VM frames VM generated stub frame
Internal errors (for example, guarantee() failure, assertion failure, ShouldNotReachHere()) will cause 80
the VM error handler to generate a similar error dump. However, the header format is different. Here is an example of how the header looks for an internal error :
# # An unexpected error has been detected by HotSpot Virtual Machine: # # Internal Error (4F533F4C494E55583F491418160E43505000F5), pid=10226, tid=16384 # # Java VM: Java HotSpot(TM) Client VM (1.5.0-rc-b63 mixed mode)
There is no signal name or signal number. Instead the second line now contains "Internal Error" and a long hexadecimal string. This hexadecimal string encodes the source module and line number where the error was detected. In general this error string is only useful to engineers working on the HotSpot Virtual Machine but the following should be noted:
The error string encodes a line number and therefore it changes with each code change and release. A crash with a given error string in 1.5.0 may not correspond to the same crash in 1.5.0_01 even if the strings match. Errors with the same root cause may have a different error string. Errors with the same error string may have completely different root causes. The error string should not be used as the sole criteria when duplicating bugs.
2.2.1.2 Thread This section contains information about the thread that just crashed. If multiple threads crash at the same time, only one thread is printed. The first part of the thread section shows the thread that provoked the fatal error.
Current thread (0x0805ac88): JavaThread "main" [_thread_in_native, id=21139] | | | | +-- id | | | +------------- state | | +-------------------------- name | +------------------------------------ type +-------------------------------------------------- pointer
The following should be noted :
The thread pointer is the pointer to the JVM internal Thread structure. It is generally of no interest unless you are debugging a live JVM or core file. Possible thread types: JavaThread, VMThread, CompilerThread, JVMPIDaemonthread, GCTaskThread, WatcherThread, ConcurrentMarkSweepThread. Note the list may change as the VM evolves. The important thread states include:

Note that the format is different from the Linux example. On Windows the output is the load and end address of each loaded module. After the list of libraries are the VM arguments (in this case the arguments were NativeSEGV 2) and the environment variables.
VM Arguments: java_command: NativeSEGV 2 Environment Variables: JAVA_HOME=/h/jdk PATH=/h/jdk/bin:.:/h/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/dist/local/e xe:/usr/dist/exe:/bin:/usr/sbin:/usr/ccs/bin:/usr/ucb:/usr/bsd:/usr/etc:/etc:/usr /dt/bin:/usr/openwin/bin:/usr/sbin:/sbin:/h:/net/prt-web/prt/bin USERNAME=user LD_LIBRARY_PATH=/h/jdk1.5/jre/lib/i386/client:/h/jdk1.5/jre/lib/i386:/h/jdk1.5/jr e/./lib/i386:/h/bugs/NativeSEGV
SHELL=/bin/tcsh DISPLAY=:0.0 HOSTTYPE=i386-linux OSTYPE=linux ARCH=Linux MACHTYPE=i386
Note that the environment list is not the full list of environment variables but rather the sub-set of the environment variables that are applicable to the JVM.
2.2.1.4 System The final section in the error log is the system information. The output is operating system specific but in general includes the operating system version, CPU information, and summary information about the memory configuration. Here is an example from a Windows XP system :
--------------S Y S T E M ---------------
OS: Windows XP Build 2600 Service Pack 1 CPU:total 1 family 6, cmov, cx8, fxsr, mmx, sse, sse2 Memory: 4k page, physical 1047024k(564864k free), swap 2520436k(2038092k free) vm_info: Java HotSpot(TM) Client VM (1.5.0-beta3-b61) for windows-x86, built on Aug 02:43:02 by "java_re" with MS VC++ 6.0
Here is an example from a Solaris 9 system :
--------------OS: S Y S T E M ---------------
Solaris 9 12/03 s9s_u5wos_08b SPARC Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 21 November 2003
uname:SunOS 5.9 Generic_112233-10 sun4u (T2 libthread) rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity load average:0.41 0.14 0.09 CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3 Memory: 8k page, physical 2097152k(1394472k free) vm_info: Java HotSpot(TM) Client VM (1.5-internal) for solaris-sparc, built on Aug 10:22:32 by unknown with unknown Workshop:0x550
On Solaris and Linux the operating system information obtained by reading /etc/*release. In general it tells you the kind of system the application is running on, and in some cases the information string may include patch level. Some system upgrades are not reflected in the /etc/*release file. This is especially true on Linux where the user can rebuild any part of the system. On Solaris the uname system call is used to get the name for the kernel, and the thread library (T1 or T2 see section 2.3.3) is also printed. On Linux the uname system call is also used to get the kernel name, and is printed along with the libc version and the thread library type.

data sharing is only enabled when the HotSpot Client VM is used. In addition sharing is only supported with the serial garbage collector. The fatal error log will print the version string in the header of the log. If sharing is enabled it should be obvious as shown in the following example :
An unexpected error has been detected by HotSpot Virtual Machine: EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x08083d77, pid=3572, tid=784 Java VM: Java HotSpot(TM) Client VM (1.5-internal mixed mode, sharing) Problematic frame: V [jvm.dll+0x83d77]
Although the sharing feature was thoroughly tested for 5.0 it may be possible that a crash stems from this new feature. Sharing can be disabled by providing the -Xshare:off option on the command line. If the crash cannot be duplicated with sharing disabled but can be duplicated with sharing enabled then it is possible that you have encountered a bug. In that case gather as much information as possible and submit a bug.

2.2.4 Visual C++ Version

JDK5.0 is compiled on Windows using Microsoft Visual C++ 6.0 Service Pack 3. This is an old but stable release of the compiler. If you experience a crash with a J2SE-based application and if you have native/JNI libraries that are compiled with a newer release of the compiler (VC.++ NET 2002 or VC++.NET 2003 for example) then you need to be aware of compatibility issues between the 6.0 and newer runtimes. Specifically, you only have a supported environment if you follow the Microsoft guidelines when dealing with multiple runtimes. For example, if you allocate memory using one runtime then you must release it using the same runtime. Unpredictable behavior or crashes can arise if you release a resource using a different library than the one that allocated it.
2.3 Hangs and Looping Processes
Issues may arise involving hangs or looping processes. A hang can arise for many reasons but often stems from a deadlock in application code, API/library code, or even a bug in the HotSpot Virtual Machine. Sometimes an apparent hang turns out not to be a hang but rather that the VM process is consuming all available CPU cycles most likely caused by a bug that causes one or more threads to go into an infinite loop. An initial step when diagnosing a hang is to find out if the VM process is idle or consuming all available CPU cycles. To do this requires using an operating system utility. If the process appears to be busy and is consuming all available CPU cycles then it is likely that the issue is a looping thread rather than a deadlock. On Solaris, for example, prstat -L -p <pid> can be used to report the statistics for all LWPs in the target process and thus will identify the thread(s) that are consuming a lot of CPU cycles.
2.3.1 Diagnosing a Looping Process

library before libc/libthread/libpthread. This library ensures that calls such as signal(), sigset(), and sigaction() are intercepted so that they do not actually replace the Java HotSpot VM's signal handlers if the handlers conflict with those already installed by the Java HotSpot VM (2.). Instead, these calls save the new signal handlers, or "chain" them behind the VMinstalled handlers. Later, when any of these signals are raised and found not to be targeted at the Java HotSpot VM, the pre-installed handlers are invoked. libjsig.so is not needed if (2.) is not required. To use libjsig.so, either link it with the application that creates/embeds a HotSpot VM, for example:
cc -L <libjvm.so dir> -ljsig -ljvm java_application.c
or use the LD_PRELOAD environment variable, for example:
export LD_PRELOAD=<libjvm.so dir>/libjsig.so; java_application(ksh) setenv LD_PRELOAD <libjvm.so dir>/libjsig.so; java_application(csh)
The interposed signal()/sigset()/sigaction() return the saved signal handlers, not the signal handlers installed by the Java HotSpot VM and which are seen by the OS. Note that the SIGUSR1 signal cannot be chained. If an application attempts to chain this signal on Solaris then the HotSpot VM will terminate with the fatal error: Signal chaining detected for VM interrupt signal, try -XX:+UseAltSigs. In addition the SIGQUIT, SIGTERM, SIGINT and SIGHUP signals cannot be chained. If the application needs to handle these signals then the -Xrs option should be considered. On Solaris, the SIGUSR2 signal can be chained but only for non-Java and non-VM threads; that is, it can only be used for native threads created by the application that do not attach to the virtual machine.
2.4.2 Exception Handling on Windows
On Windows, an exception is an event that occurs during the execution of a program. There are two kinds of exceptions: hardware exceptions and software exceptions. Hardware exceptions are comparable to signals such as SIGSEGV and SIGILL on Solaris and Linux. Software exceptions are initiated explicitly by applications or the operating system using the RaiseException() API. On Windows, the mechanism for handling both hardware and software execptions is called Structured Exception Handling (SEH). This is stack frame-based exception handling similar to the C++, Java exception handling mechanism. In C++ the __try and __except keywords are used to guard a section of code that may result in an exception : __try { // guarded body of code } __except (filter-expression) { // exception-handler block } 108