Introduction
The Search Product Team often receives questions about Liferay DXP's compatibility with Elasticsearch. This seemingly simple question belies surprising complexity: it depends on multiple factors, some of which are external dependencies for Liferay.
The answer to your simple compatibility question is usually dependent on a combination of these sub-questions:
- Which Major version of Elasticsearch (e.g., 7.x, 8.x) is my version of Liferay DXP compatible with?
- What is the Minor version range of Elasticsearch (e.g., 7.16-7.17) I can run my cluster on to connect with my version of Liferay DXP?
- What is the minimum DXP Update Version/Quarterly Release Version or Patch Level (e.g., 7.4 U120, DXP 2024.Q2 or 7.2 SP3/FP8) I need to connect to these Elasticsearch versions?
- Is it mandatory to install the connector from Marketplace (and if so, which version---e.g., 3.0.0, 3.0.1, or 3.1.0) or can I use the default bundled connector?
This article is meant to demystify these underlying compatibility questions without getting too lost in the technical details. After reading this, you will better understand why keeping things simple while simultaneously providing compatibility with a wide range of Elasticsearch versions is a challenging endeavor.
Environment
- Liferay DXP 7.0-7.4 and DXP Quarterly Releases
Target Audience
- Liferay DXP Server Administrators/DevOps Engineers
- Support Engineers
- Consultants/Sales Engineers
Table of Contents
Sources of Compatibility Information
These resources can help answer your questions about compatibility with Elasticsearch:
- Liferay DXP Compatibility Matrix
- Search Engine Compatibility Matrix
- Liferay's Elasticsearch connectors Marketplace pages, like:
- DXP 7.0 => Liferay Connector to Elasticsearch 6
- DXP 7.1-7.2 => Liferay Connector to Elasticsearch 7
General take-away: Before installing a new connector from Marketplace or moving to a newer DXP patch level, always consult with these resources to see if the Elasticsearch compatibility has changed.
What Influences the Compatibility
- Liferay DXP version and patch level: determines the list of available Elasticsearch connectors and thus, the supported major versions
- Liferay Elasticsearch connector version (if it is installed from Marketplace)
- Elasticsearch client version: determines the minimum compatible minor version
- Elasticsearch client type (e.g., Transport or HTTP client)
- Compatibility tests: to test compatibility with newer minor version(s), which might be different than the client version
- Product needs: dictates what minimum (minor) version of the client the connector must use to provide the required capabilities and features
Please continue reading if you are interested in the details of these factors including how they affect Fix Pack/Service Pack and Marketplace releases.
Liferay DXP Version and Patch Level
Each Liferay DXP version is bundled with a connector to Elasticsearch that is compatible with a specific Elasticsearch Major version:
- Liferay DXP 7.0 => Elasticsearch 2
- Liferay DXP 7.1-7.2 => Elasticsearch 6
- Liferay DXP 7.3-7.4 => Elasticsearch 7 and 8*
A connector is made up of OSGi bundles with several purposes:
- Implement Liferay’s indexing & search APIs
- Translate generic documents and queries to the actual search engine dialect
- Provide other (low-level) Liferay search framework capabilities to communicate with Elasticsearch
- Send and receive data
- Leverage other Elasticsearch features.
As these bundled connectors evolve and receive bug fixes and other changes throughout the product life of the given DXP version, the compatible Elasticsearch minor versions may shift. For example, Liferay DXP 7.2 was initially compatible with Elasticsearch 6.5.x, but at the time of writing Liferay 7.2 is now compatible with 6.8.x due to modifications introduced in one or more Fix Pack/Service Packs. The technical reasons will be explained in later sections.
Take-away: Before moving to a newer patch level, always consult with the Release Notes/Important Changes of the Fix Pack you are planning to install and the Compatibility Matrix to find related information regarding Elasticsearch compatibility.
* The Elasticsearch 8.x compatibility is provided through the bundled Elasticsearch 7 connector and the REST API Compatibility of Elasticsearch 8.
Liferay Elasticsearch Connector Version
(Referred to as “Elasticsearch connector” from now on.)
To add support for newer major Elasticsearch versions for released DXP versions, Liferay releases new connectors through Marketplace. These connectors can be used as drop-in replacements for the out-of-the-box (bundled) connectors:
- DXP 7.0 => Liferay Connector to Elasticsearch 6
- DXP 7.1-7.2 => Liferay Connector to Elasticsearch 7
Most of the time, the last released connector on Marketplace for one DXP version becomes the out-of-the-box bundled connector in the next DXP version. For example, DXP 7.3 and 7.4 come with the Liferay Connector to Elasticsearch 7 bundled, and this connector is available for Liferay 7.2 on Marketplace.
Connectors can be downloaded from Marketplace as apps (distributed as LPKG files) that you can deploy onto Liferay DXP. Each marketplace app has its own version where the major version determines which Liferay DXP version they are compatible with as follows:
| DXP Version | Connector App Name | Connector App Marketplace Version |
|---|---|---|
| DXP 7.0 | Liferay Connector to Elasticsearch 6 | 1.x |
| DXP 7.1 | Liferay Connector to Elasticsearch 7 | 2.x |
| DXP 7.2 | Liferay Connector to Elasticsearch 7 | 3.x |
To deliver bug fixes and other updates, Liferay releases minor and micro version releases for these apps from time-to-time. As the Elasticsearch connectors depend on Liferay DXP APIs that can change in Fix Packs/Service Packs, a given minor/micro version of the connector app may require a minimum Liferay DXP Patch Level, for example:
| Connector App Name | Connector App Marketplace Version | Required DXP Patch Level |
|---|---|---|
| Liferay Connector to Elasticsearch 6 | 1.0.0 |
DXP 7.0 FP42+ |
| Liferay Connector to Elasticsearch 6 | 1.0.1 |
DXP 7.0 FP70+ |
| Liferay Connector to Elasticsearch 6 | 1.1.0 |
DXP 7.0 FP79+ |
| Liferay Connector to Elasticsearch 6 | 1.2.0 |
DXP 7.0 FP96+ |
| Liferay Connector to Elasticsearch 7 | 2.0.0 |
DXP 7.1 SP5+/FP20+ |
| Liferay Connector to Elasticsearch 7 | 3.0.0 |
DXP 7.2 SP1+/FP2+ through FP4 |
| Liferay Connector to Elasticsearch 7 | 3.0.1 |
DXP 7.2 SP2+/FP5+ |
| Liferay Connector to Elasticsearch 7 | 3.1.0 |
DXP 7.2 SP3+/FP8+ |
| Liferay Connector to Elasticsearch 7 | 3.2.0 |
DXP 7.2 FP9+ |
Despite the efforts to keep the compatibility as wide as possible (so a given app version stays compatible with a series of patch levels on a DXP version), there are changes that require us to bump the minimum patch level.
Take-away: We do our best to document any "breaking changes" at multiple locations. Before installing a new connector from Marketplace or moving to a newer patch level, always consult with the Release Notes/Important Changes of the Fix Pack you are currently using (or planning to install), the Search Engine Compatibility Matrix, and also check the Marketplace page of the app to see if the Elasticsearch compatibility has changed.
Elasticsearch Client Version and Type
This is the most important factor.
The connectors use Elasticsearch’s Java clients. These clients correspond to a given Elasticsearch release, so the connectors use client versions like 6.5.0, 6.8.16, 7.9.0 and 7.14.1. These versions determine the minimum compatible minor version of Elasticsearch. (Please refer to the Elasticsearch Connectors: Technical Data Sheet for more details.)
Besides the version, there is another client attribute that influences the compatibility: the client type.
Liferay's Elasticsearch connectors use two types of Java clients and each defines the compatibility with higher minor Elasticsearch versions differently:
| DXP Version | DXP 7.0-7.2 | DXP 7.3-7.4 |
| Client Type | Transport Client (2.x, 6.x, 7.x) | REST Client (7.x) |
| Client Compatibility | Elastic says: The client must have the same major version (e.g. 2.x, or 5.x) as the nodes in the cluster. Clients may connect to clusters which have a different minor version (e.g. 2.3.x) but it is possible that new functionality may not be supported. Ideally, the client should have the same version as the cluster. |
From the Elastic's documentation: The High Level Client is guaranteed to be able to communicate with any Elasticsearch node running on the same major version and greater or equal minor version. It doesn’t need to be in the same minor version as the Elasticsearch nodes it communicates with, as it is forward compatible meaning that it supports communicating with later versions of Elasticsearch than the one it was developed for.* |
* We mostly use the High Level REST Client, however, there are some currently unimplemented features that DXP needs so we obtain the Low Level REST Client from the High Level client to build the necessary JSON for the requests. Regarding the Low Level Client, Elastic's documentation states that "it is compatible with all Elasticsearch versions".
Take-away:
As you can see, the client version used by the connector determines the major and the minimum compatible minor version of Elasticsearch in all cases. The REST client allows us to define a more flexible compatibility with newer minor releases of Elasticsearch in DXP 7.3 and beyond.
The Transport Client is more strict, requiring us to perform a series of tests to mark a newer minor version as compatible.
The exercise for you is the same: before moving to a new DXP patch level or installing a new version of the connector from Marketplace, always check the compatibility matrix to make sure you don't miss an important detail so you can run your DXP-Elasticsearch stack using compatible versions.Note: Liferay DXP running a connector whose client is using newer minor version (e.g., 7.9.0) may still be able to connect to older minor versions of Elasticsearch (e.g., 7.4.0) without issues, however these combinations are not tested by Liferay.
Technical note: The client version also defines the built-in Elasticsearch server's version (use for development/testing purposes only!) that is started in Embedded or Sidecar mode when the portal first starts. At your earliest convenience, configure the Remote/Production mode connection.
Compatibility Tests
Liferay conducts ad-hoc testing of the Elasticsearch connectors with newer minor releases of Elasticsearch. These minor releases sometimes introduce breaking changes at the low-level Java API (which can affect applications using the Transport Client). When this happens we usually need to update the client's minor version and the implementation inside the connector. When we do this, we must deliver the updated connectors in Fix Packs/Service Packs or through Marketplace releases.
Take-away: in short, these test iterations help us determine the highest compatible minor versions of Elasticsearch, which are reflected in the compatibility matrix. In addition to the Search Engine Compatibility Matrix, read the Elasticsearch Connectors: Technical Data Sheet article. This article provides more detail about the client, compatible Elasticsearch (minor) versions for each Liferay DXP version, and the available Marketplace connectors.
Product Needs
To add support for new functionalities like Cross-Cluster Replication (CCR) or to fix complex issues, sometimes we need to introduce changes to the Liferay Search API/Framework which is part of the core Liferay DXP distribution. As the connectors depend on these capabilities and APIs, newer Fix Packs or Service Packs may require you to install a newer connector version from Marketplace, if the connector you need is not bundled on that DXP version.
In addition, newer Elasticsearch minor versions, new features (like support for CCR which requires 6.7+) or bug fixes (like LPS-120240) may require us to bump the client version inside the connector, which again, shifts the range of the minimum compatible Elasticsearch minor version. When it happens, the next Fix Pack/Service Pack or a new Marketplace release of the connector will include these changes.
Take-away: We try to keep these types of changes to a minimum, but there are situations when it's unavoidable. Therefore consulting with the above-mentioned resources before updating your stack is a very important step.
Related Resources
- Search Engine Compatibility Matrix
- Liferay DXP Elasticsearch Connectors: Technical Data Sheet
- Liferay DXP Compatibility Matrix for Hosted Elasticsearch Services
- Liferay DXP Compatibility with Elasticsearch Service on Elastic Cloud
* Elastic, Elasticsearch, and X-Pack are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.
** At the time of writing. Please refer to the Search Engine Compatibility Matrix for up-to-date information.