Issue
-
When we set the Search Service scale to "3" on the UAT environment (on lcp.json file), the elasticsearch cluster is not initialized correctly.
On the service logs we see the following error:
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: This node previously joined a cluster with UUID [1GN4bshcQ7GJNJgTQuHxBw] and is now trying to join a different cluster with UUID [cM0aLSEdSKeXXQTaoPNYlw]. This is forbidden and usually indicates an incorrect discovery or cluster bootstrapping configuration. Note that the cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path [/data/data] which will also remove any data held by this node
On the shell of each search instance we notice the following:
elasticsearch@search-0:~$ curl search-0:9200
{
"name" : "search-0",
"cluster_name" : "liferay_cluster",
"cluster_uuid" : "_na_",
"version" : {
"number" : "7.17.18",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "8682172c2130b9a411b1bd5ff37c9792367de6b0",
"build_date" : "2024-02-02T12:04:59.691750271Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}on search-1 & search-2, we see:
"cluster_uuid" : "cM0aLSEdSKeXXQTaoPNYlw"
We have tried to set the scale back to 1 and then again back to 3, but every time we end up with this misconfiguration. We have also tried to restart the service.
Environment
- Liferay PaaS
Resolution
-
We were able to solve this with the following procedure:
(search-0 was the node with the different uuid)
shell search-0: rm -rf /data/data
shell search-1: elasticsearch restart
shell search-2: elasticsearch restart
shell search-0: elasticsearch restart
Problem fixed!