Liferay Support does not recommend or endorse specific third-party products over others. Liferay is not responsible for any instructions herein or referenced regarding these products. Any implementation of these principles is the responsibility of the subscriber.
A thread dump is a snapshot of the JVM's activity at a given time, so looking at a single thread dump does not provide useful information. When investigating threads, it is recommended to take 5-10 thread dumps at 20-30 seconds apart while the application server is slow or unresponsive to get the most useful dumps, as this will tell us how long each process takes and how many times it is executed so we can determine if there are any processes that are taking too long or executed too many times.
Resolution
Step 1
The first piece of information you will need to be able to obtain a thread dump is the Java process's PID. The java JDK ships with the jps command which lists all active java process ids. You can run this command like this: jps -v. The pid is listed as well as the JVM arguments, which will help you to identify which PID belongs to the application server running Liferay Portal.
Here's an example output:
7652 Jps -Denv.class.path=. -Dapplication.home=C:\java\jdk1.6.0_18 -Xms8m 2936 -Dexe4j.semaphoreName=c:_program files (x86)_smartsvn 6_bin_smartsvn.exe0 -Dexe4j.moduleName=C:\Program Files (x86)\SmartSVN 6\bin\smartsvn.exe -Dexe4j.tempDir=C:\Users\liferay\AppData\Local\Temp\e4j8FA1.tmp_dir20624 -Dexe4j.unextractedPosition=211749 -Dexe4j.consoleCodepage=cp0 -Dsun.java2d.noddraw=true -Xms24m -Xmx256m 4752 Bootstrap -Dfile.encoding=UTF8 -Djava.net.preferIPv4Stack=true -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -Duser.timezone=GMT -Xmx1024m -XX:MaxPermSize=256m -Djava.util.logging.config.file=E:\projects\temp\liferay-portal-6.0-ee-sp2\tomcat-6.0.32\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=E:\projects\temp\liferay-portal-6.0-ee-sp2\tomcat-6.0.32\endorsed -Dcatalina.base=E:\projects\temp\liferay-portal-6.0-ee-sp2\tomcat-6.0.32 -Dcatalina.home=E:\projects\temp\liferay-portal-6.0-ee-sp2\tomcat-6.0.32 -Djava.io.tmpdir=E:\projects\temp\liferay-portal-6.0-ee-sp2\tomcat-6.0.32\temp
4752 is running Liferay Portal based on the JVM arguments.
Note: Elasticsearch Sidecar runs in a separate JVM, and it will have its own PID listed. Check the description for references to 'sidecar' to identify it.
Note: In Linux and UNIX, you may have to run this command as sudo -u user jps -l
, where "user" is the username of the user that the java process is running as.
Step 1 Alternatives
- Windows
Ctrl+Shift+Esc to open the task manager and find the PID under 'Processes'.
From the command prompt, execute netstat command: netstat -noa
This command will return a list of Active connections. Look for the connection that is using the port your application server is using. As an example, Tomcat uses two connector ports: 8080, 8005, and 8009. After executing netstat -noa command:
127.0.0.1:8005 0.0.0.0:0 LISTENING 4752
127.0.0.1:8080 127.0.0.1:63796 ESTABLISHED 4752
Compare 4752 to the other lines and yes, this looks like the PID assigned to Tomcat.
- UNIX, Linux and Mac OSX:
ps -el | grep java
Step 2
Request a thread dump from the JVM. Use Jstack (packaged with later versions of JDK 1.5) to create a thread dump. Execute this command in the command prompt: jstack <pid> >> threaddumps.txt
This full command requires the pid of your application server process and will print the stacktrace to the threaddumps log file.
Troubleshooting
If you see an error message, Unable to attach to 32-bit process running under WOW64, this is because you are trying to use the jstack tool from JDK 1.6 to take thread dumps from a process running on JDK 1.5. The bundled version of Liferay + Tomcat/Jboss comes with a bundled 32-bit version JRE. In a 64-bit environment, you need to remove the bundled JRE and allow the application server to run on the system installed 64-bit JRE in order for the 64-bit jstack tool to work successfully.
If you see an error message:
unable to open socket file: target process not responding or HotSpot VM not loaded
this indicates that you will need to run the jstack command as the user that the java process is running as. To do so, you may need to run this command as:
sudo -u <user> jstack <pid> >> threaddumps.log
where "<user>" is the user that the java process is running as. Even running as root will not work if the user that is running the JVM isn't root.
Additional Information
- Taking Heap Dumps from a JVM
- JConsole Documentation
- Oracle Troubleshooting Guide for Java SE 6 with HotSpot VM
- Adobe Take Thread Dumps from a JVM
- Critical Issue Checklist (Liferay Documentation)
- Before Opening a Liferay Performance Ticket (Liferay Documentation)