Troubleshooting Performance Issues in Liferay PaaS

Note: please note that Liferay has renamed its Liferay Experience Could offerings to Liferay SaaS (formerly LXC) and Liferay PaaS (formerly LXC-SM).

Performance issues can manifest in various different symptoms, from 503 Service Unavailable errors to slow page loads. This article provides guidelines to help you identify and resolve these issues effectively.

Environment

• Liferay PaaS

Common Signs of Performance Issues:

• Container restarts
  • Occasional 503 errors when the service is unavailable due to the restart
  • To confirm container restarts, you can take the following steps
    1. Navigate to the Logs tab on the Cloud Console
    2. Under Service select Liferay
    3. Select Status Logs
    4. Look for probe failures: Container liferay failed liveness probe, will be restarted
• Slow navigation in Liferay DXP
  • If users experience sluggish performance when navigating your Liferay DXP instance, it could be due to performance-related problems

Troubleshooting Steps:

Initial Assessment

Begin by understanding when the performance issues started occurring. Review recent changes made to your environment, particularly those related to custom code modifications. If the issues began after a deployment, investigate these changes closely. If no deployment occurred, check for slowness across all pages and examine template changes and any new data additions (e.g., Web Contents, Documents).

If possible, take the following steps:

1. Create a backup
2. Revert changes from production to stabilize the environment
3. Deploy problematic build to a lower environment
4. Restore the created backup to a lower environment

This can allow for troubleshooting in a lower environment without affecting the production environment.

Data Collection

Generate thread dumps

1. If you are using a Liferay DXP docker image with version 5.0.18 or newer, you can execute the following command in the Liferay service's shell on the Cloud Console to generate thread dumps: generate_thread_dump.sh
1. Thread dumps will be saved to the /opt/liferay/data/sre/thread_dumps/{current_date} folder, which can be analyzed locally and shared with Liferay Support.
2. If you have an older image version, follow these steps to generate thread dumps:
1. Create a directory with the command: mkdir /opt/liferay/data/thread_dumps
2. Find the process ID (PID) with: ps aux | grep java
3. Execute thread dump generation with: jstack -l ${PID} /opt/liferay/data/thread_dumps/thread_dump1.txt
4. Ensure you create 5-10 thread dumps with a 5-second interval between each dump.

After taking these steps, you can generate a manual backup from the Backups page on the Cloud Console, and download the Document Library backup to your machine to be able to analyze thread dumps locally.

Please note that thread dumps have to be generated at the time of the slowness, otherwise they cannot be used to analyze performance issues.

Generate heap dumps

Heap dumps can help diagnose OutOfMemoryErrors (OOM). To generate a heap dump using a Liferay DXP docker image with version 5.0.18 or newer, execute the command: generate_heap_dump.sh from the Liferay service’s shell.

The heap dump will be saved to /opt/liferay/data/sre/heap_dump/{current_date}, which you can download for analysis by part of the Document Library backup and share with Liferay Support.

NOTE: Due to the nature of using a script to generate heap dumps, the generated heap dumps will only snapshot what is currently within the JVM at the time the script is run. For OOM errors, having the heap dump automatically created during an event will offer better insight. This can be done by configuring your JVM options as follows.

Ensure the following JVM options are always set in the LIFERAY_JVM_OPTS environment variable:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/liferay/data/heap-dump/

This configuration generates heap dumps when OutOfMemoryErrors occur.

Liferay logs

Thoroughly analyze Liferay logs for insights into performance issues. Use the LCP CLI to retrieve logs from a specific timeframe. Example command:

lcp log -p projectId -e envName -s serviceName --type application --since 
"2023-09-15T10:00:00+02:00" --until "2023-09-15T14:00:00+02:00"

Opening a Support Ticket

When dealing with performance issues, if you reach out to Liferay Support, please provide us with the following information:

• A clear description of the issue
• Details of changes made before the issue started
• The date and time when the issue first occurred
• Liferay logs
• portal-ext.properties
• Reproduction steps, if possible
• Thread dumps and heap dumps (Taken during the time of the incident or when slowness occurs)
• Affected Environment URL(s)
• Liferay service LCP.json
• Gradle.properties

Note that performance tuning and resolving custom code-related issues fall outside the scope of standard Support Services. If you need help with such cases, you can reach out to our Global Services Team through your Sales Representative.