How to generate multiple Java Thread Dumps in Linux and Windows

Issue

• In case of a throughput issue, the Liferay support service usually asks to take some java thread dumps from the Liferay server, in an interval of 1-2 seconds.
• Taking several thread dumps, allows you to detect if a Java thread is stuck or taking too long to complete.
  
• How can I easily generate these thread dump files with that 1-2 seconds interval, in an efficient way?

Environment

• All Liferay versions

Resolution

The easiest way to get the multiple java thread dumps is to use an operating system script that will generate the dumps on regular intervals.

You can also execute a Groovy script from the Server Administration script tab.

1. Using an operating system script

This article includes two scripts attached for both Linux/MacOS and Windows operating systems. You can use them to generate the thread dumps.

Steps to generate the thread dumps:

1. Get the process PID: The first step is to obtain the PID of your application server:
   • This is usually obtained using the java jps -l command line tool or using an operating system command (for example ps or netstat)
   • The article Taking Thread Dumps From a JVM contains more information on how to get the PID of the application server
2. Execute the script: Once you have the PID, execute the attached script that matches your operating system:
   
   • Linux/MacOS: threaddumps.sh [pid] => download here the threaddumps.sh script 
   • Windows: threaddumps.cmd [pid] => download here the threaddumps.cmd script

These scripts will generate 30 thread dumps in an interval of 2 seconds using the java jstack command.

2. Using a Groovy script

This article includes a Groovy script that you can execute from the Server Administration script tab.

Steps to generate the thread dumps:

1. Go to Control Panel => Server Administration => script tab
2. Copy the threaddumps.groovy script to the script tab
3. Update the folder variable with the output folder, the default value is the tomcat temp folder.
4. Execute it, following message will be displayed:

   • Generating threaddumps in a background thread
     Threaddumps will be generated in the folder: /liferay/tomcat-9.0.56/temp

 

3. Generated files

The generated files will be named: threaddump-<pid>-<time>-<number>.txt

Example:

threaddump-23065-151811-1.txt
threaddump-23065-151813-2.txt
threaddump-23065-151816-3.txt
threaddump-23065-151818-4.txt
threaddump-23065-151820-5.txt
threaddump-23065-151823-6.txt

 

4. Troubleshooting

In case of problems with the operating system script:

• Error with the PID: 999 no such process or 999 process is not running 
  
  • Cause: You have specified the wrong PID. Solution: double-check and specify the correct one

• Permission error: 9999: Operation not permitted or unable to open socket file: target process not responding or HotSpot VM not loaded
  • Cause: You have executed the script with a different user to the user that is executing the application server
  • Solution: Execute the script with the same user that is running Liferay
  • or execute this with the root user:
    

    •  sudo -u [user] ./threaddumps.sh [pid]

• Error executing jstack: jstack: command not found or 'jstack' is not recognized as an internal or external command
  • Cause: The jstack command is not available in the current user PATH
  • Solution: you have to:
    • option 1: change the PATH configuration of your user
    • option 2: edit the threaddumps.sh / threaddumps.cmd script and replace "jstack" with the full path (for example: /usr/lib/jvm/java-11-openjdk-amd64/bin/jstack)
    • option 3: use the Groovy script alternative

•  

Additional Information

• More information on how to get a single thread dump: Taking Thread Dumps From a JVM
• External Oracle documentation:
  • jstack: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.html
  • jps: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jps.html