Stuck reindex: How to check the progress of a reindex process and remove blocked BackgroundTask and Lock records

Symptom

Sometimes a reindex will take a long time and seem like nothing is happening. You simply see the reindex icon in the Control Panel bar and/or the percentage does not change. 

Environment

• DXP 7.4
• DXP 7.3
• DXP 7.2
• DXP 7.1
• DXP 7.0

Resolution

Since this resolution requires modification of database entries using Liferay API, ensure that a database backup is created beforehand and continue to maintain adequate backups in case a rollback is required.

Checking the BackgroundTask table

Query records from the BackgroundTask table where taskExecutorClassName column contains ReindexPortalBackgroundTaskExecutor or ReindexSingleIndexerBackgroundTaskExecutor.

• Records with 0 as status refer to new tasks that are not queued yet. Normally, you won't find entries like this for reindex.
• Records with 1 as status mean they are still in progress.
• Records with 2 as status refer to failed reindex operations. Look for the dates in the createDate and modifiedDate fields if they are close to the time when the problem encountered with reindexing.
• Records with 3 as status refer to successful reindex tasks.
• Records with 4 as status mean they are queued, but may be never picked-up so you can try to delete them and invoke a reindex again.
• Records with 5 as status refer to cancelled jobs.

To delete a BackgroundTask, we recommend to use Liferay APIs:

1. Obtain the backgroundTaskId of the record
2. Perform the delete operation from a Groovy-script through the Script console:

import com.liferay.portal.background.task.service.BackgroundTaskLocalServiceUtil;

BackgroundTaskLocalServiceUtil.deleteBackgroundTask(backgroundTaskId);

If you delete records manually from the database, make sure you clear the database cache through the Server Administration in the Control Panel.

Checking the Lock_ table

It may happen that an unreleased lock is preventing the reindex operation. You can query records with ReindexPortalBackgroundTaskExecutor* as key_. If there are any even though there are no running background tasks (recall the previous section), you can try deleting the record in question and clear the database cache through the Server Administration in the Control Panel.

Important: always start with the removal of blocked BackgroundTasks, otherwise the tasks may be continued once you delete the associated Lock record.

To delete a Lock, we recommend to use Liferay APIs:

1. Obtain the lockId of the record
2. Perform the delete operation from a Groovy-script through the Script console:

import com.liferay.portal.lock.service.LockLocalServiceUtil;

LockLocalServiceUtil.deleteLock(lockId);

If you delete records manually from the database, make sure you clear the database cache through the Server Administration in the Control Panel.

Enabling logging

Go to Control Panel > Server Administration > Log Levels. Add these two classes and set them to DEBUG:

com.liferay.portal.search.internal.background.task.ReindexStatusMessageSenderImpl

This log will print out messages when it indexes content.

com.liferay.portal.background.task.internal.messaging.RemoveOnCompletionBackgroundTaskStatusMessageListener

This log will tell you the status of the index job, such as if it is still proceeding or if it completes.

Also add the following class with INFO level:

com.liferay.portal.search.internal.SearchEngineInitializer

Verifying if the reindex has successfully completed

• No exceptions related to indexing in the DXP server log from the time-frame of the reindex operation
• No records with status 1 or 4 in BackgroundTask table for the mentioned executor class names