Elasticsearch users often work with one primary cluster for all operations, even if they actually require a second backup cluster. If a backup cluster exists, it will usually sit idle while the primary cluster handles all load and often becomes overloaded. This leads to a decrease in performance across the entire operation.
What’s needed is to spread and balance the search load between the 2 clusters. Ideally, the division would work in such a way that both clusters will actively work and serve requests at the same time. This is considered an active-active solution, as opposed to active-passive, where indexing and search are all directed to 1 cluster and if it falls, you activate a switch failover to the other.
To ensure active-active high availability and data recovery on your own, you would need to write a routing component that is programmed into the application itself.
This component would need to be aware of multiple clusters at the same time and aware of which cluster each user is located on. Every request to Elasticsearch would have to go through that component to route the request to the relevant cluster.
Without a backup cluster that is actively being synchronized in real-time with the indexing and update requests, you wouldn’t be able to run searches in a consistent way.
If the backup cluster itself is up to date, you might be able to change DNS records in order to switch over and achieve proper utilization of it. The downside of this solution is that DNS takes time to propagate once availability changes, as DNS routing takes time.
Opster’s Multi-Cluster Load Balancer provides indexing and mirroring in an active-active manner and balances searches in a consistent way between the synced clusters.
Instead of having one loaded cluster and one underutilized one, the Load Balancer ensures that if one cluster becomes overloaded and slow, queries will be transferred to the other cluster automatically. In this way, both clusters are utilized to handle resources and respond to requests as efficiently as possible.
When using a backup cluster, the question arises of how to route searches when identical information exists in two places. In Elasticsearch, there is no option to control the routing of searches between your different clusters. Because of this, searches could be slow even when there are resources available in other clusters.
Ideally, searches should be routed according to business logic, geographical considerations, system requirements and more. It would be very difficult and complex to create this kind of logic-based routing solely on DNS.
Opster’s Multi-Cluster Load Balancer provides a simple configuration that allows you to choose a strategy for your search routing. These strategies include hardcoded definitions of default and failover clusters, latency-based routing and more.
You can learn more about the Multi-Cluster Load Balancer here, and book a demo to try it out.
