Elasticsearch guides

Elasticsearch scripts can place heavy loads on clusters if they are not written carefully. It is a best practice to limit the type of..

Elasticsearch carries out "bootstrap checks" to ensure that important settings have been set correctly. If any of these fail, ES won't start.

Elasticsearch can be configured to prevent memory swapping on its host machine by adding bootstrap memory_lock true. If bootstrap checks...

A read-only delete block can be applied automatically by the cluster because of a disk space issue. It can also be applied manually by...

Cluster name and data path are default settings that could be destructive for proper Elasticsearch function if handled incorrectly. If you...

Adaptive replica selection is a process that prevents a distressed Elasticsearch node from delaying the response to queries. To enable it...

Cluster shard rebalancing and allocation are often confused with each other. If cluster shard rebalancing isn't enabled, then...

By default, Elasticsearch expensive queries are allowed to run. By setting search.allow_expensive_queries to false, you can prevent users...

File descriptors are required to keep track of all the files Elasticsearch has open at any given time, as well as all network...

It's possible to reduce the risk of accidental deletion of indices by preventing the use of wildcard for destructive operations. To check...

Rest-high-level is built on top of low-level rest-client and is a method of communicating with Elasticsearch based on HTTP REST endpoints...

Script regex is disabled in Elasticsearch by default, but you can decide to enable it. Regex must be used with care in painless scripts...

Elasticsearch split brain occurs when there is more than one master in the cluster. By setting the quorum of minimum master nodes...

Zen discovery settings for cluster formation were deprecated in Elasticsearch V.7 and should be removed from version 7 and above due to...

If the ratio of memory to number of shards in the cluster is low, it suggests that you have insufficient memory compared to the volume...

There are various watermark thresholds on an Elasticsearch cluster. As the disk fills up on a node, the 1st threshold to be crossed is...

The different categories in the Elastic Pricing Calculator can impact your final cost. Here's how to efficiently use the pricing calculator.

Elasticsearch clusters need to maintain the cluster state in memory on each and every nodes, which requires a large amount of resources...

Here are 12 tips to reduce and Optimize your Elasticsearch costs. First, plan data retention: carefully adjust your...

When cluster state becomes too large it poses many challenges. In order to determine the size of your cluster state and reduce it, you...

In Elasticsearch’s multi-tier architecture, the tiers are named hot, warm, cold & frozen. This Elasticsearch architecture allows better...

When the “disk flood stage” threshold is exceeded on an Elasticsearch cluster, it will start to block core actions. To resolve this issue...

A high heap size in Elasticsearch will give your node more memory for indexing and search operations. However, your node also requires...

High disk watermark is one of the various thresholds on your Elasticsearch cluster. Passing this threshold is a warning and you should not...

Finding the right number of shards for your Elasticsearch indices, and the right size for each shard depends on many factors, including...

If you don’t have enough disk space available, Elasticsearch will stop allocating shards to the node. This will eventually prevent you from...

When you have too many shards in your Elasticsearch cluster, there are a few steps you can take in order to reduce the number of shards...

Sometimes you can observe that the CPU and load on some of your data nodes is higher than on others. This can occasionally be caused by...

Low disk watermark is one of the various thresholds on your Elasticsearch cluster. Here are possible actions you can take to resolve...

The Elasticsearch process is very memory intensive. Here are the memory requirements and some tips to reduce your Elasticsearch memory usage.

A large number of shards on an Elasticsearch cluster requires extra resources. Learn key ways to avoid and correct oversharding...

Shard allocation is an algorithm by which Elasticsearch decides which unallocated shards should go on which nodes. To resolve unbalanced...

It is a best practice that Elasticsearch shard size should not go above 50GB for a single shard. If you go above this limit...

There are various methods for retrieving fields in Elasticsearch, including: _source, stored_fields, fields & docvalue_fields. To retrieve...

The Elasticsearch data stream is an abstraction layer between the names used by applications to facilitate ingestion and search operations...

The 3 main methods in Elasticsearch to calculate the storage size of specific fields in an index are: using the _disk_usage API, creating...

Terms aggregations rely on an internal data structure known as global ordinals. The eager_global_ordinals parameter is used to...

The new match_only_text feature in Elasticsearch can save up to 10% of disk space on logging datasets. This field type will set a flat...

Elasticsearch searchable snapshots allow data exploration. They can be controlled with ILM Policies or be manually mounted. To implement...

By using the Split Index API in Elasticsearch, an existing index can be split to create a new index with extra primary shards. To do this...

The text analysis process is tasked with two functions: tokenization and normalization and is carried out by employing analyzers. When you...

A tokenizer decides how Elasticsearch will take a set of words and divide it into separated terms called “tokens”. To work with synonyms...

Follow these steps to configure all Elasticsearch node role types (master, data, coordinating, ingest, machine learning, remote eligible...

Mappings are the core element of index creation in Elasticsearch. Defining them correctly can improve performance. Mapping types include...

Ingest pipelines sit within the Elasticsearch node and will perform a set of alterations on your data that you...

The join data type field allows users to establish parent-child relationships between documents in Elasticsearch. To use it, you need to...

Elasticsearch has many methods for defining relationships between documents, such as nested documents. To use the nested field type...

Object types in Elasticsearch can be used to define relationships between documents. Here's how to use the object field type for that purpose.

When you have too many shards in your Elasticsearch cluster, there are a few steps you can take in order to reduce the number of shards...

Index lifecycle management helps automate the creation, management & removal of an Elasticsearch index. Define the index lifecycle policy...

Elasticsearch index templates allow us to create indices with user defined configuration. An index can pull the configuration from these...

Nested is a special object type that is indexed as a separate document. To demonstrate the use of Elasticsearch nested VS. object fields...

There are at least three use cases where you should consider using transforms instead of aggregations in Elasticsearch. First, when the...

Many clusters use coordinating or ingest nodes, while others leave the ingest and coordination functions to the data nodes. In order to...

Once an Elasticsearch cluster reaches a certain size, it's recommended to create 3 dedicated master nodes. Here is how you can create...

A coordinating node is a node that handles HTTP(S) requests for the cluster, especially indexing & search requests. A coordinating only...

When looking at Shard View for the index, it was clear that the index in question wasn’t carrying out the highest indexing rate and wasn’t...

This error occurs when the Elasticsearch cluster doesn't have a quorum of nodes with voting rights to elect a new master node. To resolve...

The node concurrent recoveries setting determines the max number of shards that can be recovered at once from each node. It's important to...

Master nodes are responsible for actions such as creating or deleting indices. If you don't have enough master nodes, it could lead to...

Setting up zone awareness for shard allocation ensures high availability in the case of several servers going down. Here's how to...

There are various approaches for autocomplete in Elasticsearch. Here are some tips & examples for choosing the approach best suited to your...

Cross-cluster search enables users to execute a query across multiple Elasticsearch or OpenSearch clusters. To perform cross-cluster search...

Elasticsearch delete by query is an API, which provides functionality to delete all documents based on the matching query. If you don't...

Here's how to set up Elasticsearch monitoring using Prometheus and Grafana. To monitor Elasticsearch with Prometheus you first need to...

Elasticsearch currently provides 3 different techniques for fetching many results: Pagination, Search-After and Scroll. To learn how to...

Rollup jobs in Elasticsearch reduce old data storage costs by storing summaries of data for a given time period. Rollup examples include...

An Elasticsearch runtime field is a field evaluated at query time instead of indexing time, which allows to modify our schema at the...

Elasticsearch offers three types of suggesters: term suggesters, phrase suggesters & completion suggesters (autocomplete). Suggesters work...

By analyzing your slow logs, you can understand why searches are slow and how to optimize them. To enable slow logging in Elasticsearch...

Analyzing search slow logs can provide users with advanced insights like the number of costly queries, reasons why queries were costly, so...

There are multiple ways to improve your Elasticsearch aggregation performance. First, you should limit the scope by filtering documents...

By optimizing and maintaining Elasticsearch search speed, you can improve your product’s user experience. Here's how to speed up search...

One of the most difficult issues to manage and resolve in Elasticsearch is poor search performance. This blog goes through clear steps to...

Follow these steps to list and restore dangling indices in Elasticsearch: (1) Run the dangling indices API & copy the...

To create & restore snapshots, you need to register a snapshot repository with every Elasticsearch node in the cluster. Here are the steps...

This guide explores how to reduce Elasticsearch search latency based on a key study. The first lesson is to always...

An Elasticsearch snapshot is a backup of an index taken from a running cluster. It's better to use snapshots instead of disk backups due...

The cluster concurrent rebalance setting determines the maximum number of shards the cluster can move to rebalance the distribution of...

When you try to retrieve a document by ID, Elasticsearch will count the number of times that it searches for an ID which doesn't exist...

Circuit breaker exceptions are thrown to alert us that something needs to be fixed in Elasticsearch in order to reduce memory usage. To fix...

When Elasticsearch detects that the merge process cannot keep up with the rate of indexing, then it will start to throttle indexing...

"Hotspots" refers to a situation in which a cluster with multiple nodes is not balanced - some nodes are handling more load than others...

One way to evaluate whether your resources are cost efficient it check the ratio of disk usage to the memory allocated...

Here's how to set up Elasticsearch monitoring using Prometheus and Grafana. To monitor Elasticsearch with Prometheus you first need to...

By executing Elasticsearch rolling restarts with the help of the API, you can maintain high cluster availability & avoid downtime. To do..

Elasticsearch searchable snapshots allow data exploration. They can be controlled with ILM Policies or be manually mounted. To implement...

By using the Split Index API in Elasticsearch, an existing index can be split to create a new index with extra primary shards. To do this...

When upgrading to a new Elasticsearch version, you can use the feature migration APIs to avoid deprecation issues. These APIs simplify...

When the “disk flood stage” threshold is exceeded on an Elasticsearch cluster, it will start to block core actions. To resolve this issue...

Heavy merges use CPU, memory and disk resources, which can slow down the cluster’s response speed. In order to fix...

Elasticsearch cluster pending tasks are updates to the cluster state that were initiated by a user or the cluster. To resolve, list the...

High CPU is often a symptom of other underlying issues. It should be fixed because a distressed node will slow query response time and...

High disk watermark is one of the various thresholds on your Elasticsearch cluster. Passing this threshold is a warning and you should not...

A high number of tasks in management queue can cause Elasticsearch cluster instability which could result in data loss. To resolve...

Follow these steps to configure all Elasticsearch node role types (master, data, coordinating, ingest, machine learning, remote eligible...

When facing recurring red status events in Elasticsearch, AutoOps can be used to debug the issue. Many pending tasks in the cluster were...

There are 2 methods to increase the primary shard count in Elasticsearch: _reindex API & the _split API. Before using either method, you…

The 2 approaches for upgrading Elasticsearch versions are full cluster restarts & rolling restarts. Before making an Elasticsearch upgrade...

Once an indexing queue exceeds the maximum size, the Elasticsearch node will start rejecting index requests. To resolve this, check the...

A saturated coordinating node could cause an increase in search or indexing response latency. This can be fixed by putting a load balancer...

An overloaded master node may cause instability in the cluster. There are 3 ways to fix loaded master nodes: (1) Checking for...

Low disk watermark is one of the various thresholds on your Elasticsearch cluster. Here are possible actions you can take to resolve...

An Elasticsearch cluster requires a master node to be identified in the cluster. Reasons why a master node is not discovered yet include...

If the max of shards per node is exceeded in Elasticsearch, shards can't be allocated. It is crucial to check if the limit is set at a...

An Elasticsearch node can disconnect from a cluster for several reasons, including: excessive garbage collection from JVM, configuration...

Elasticsearch red status indicates not only that the primary shard has been lost, but also that a replica has not been promoted...

There are a number of reasons why a search request can be rejected by the Elasticsearch cluster. To resolve the issue, you need to...

When scaling down Elasticsearch resources, it's important to consider several factors. Here's how to effectively plan to scale down.

There are a number of possible causes for slow searches on particular nodes. To correct the issue and improve search performance, you...

An Elasticsearch cluster can start to reject search requests for several reasons. To resolve this, check the state of the thread pool and..

If the indexing queue is high/causes timeouts, it hints that Elasticsearch nodes can't keep up with the indexing rate. To fix slow indexing...

Yellow status indicates that one or more of the replica shards on the Elasticsearch cluster are not allocated to a node. This could occur...

In Elasticsearch, the combined_fields query allows you to search several text fields as though their indexed values have been indexed into...

There are 4 types of Elasticsearch boolean clauses: filter, must, should & must_not. A single bool query can contain a mix of them. To use...

Elasticsearch boosting query is used to return only documents that match a positive query while minimizing the score of documents that...

An Elasticsearch composite aggregation allows to paginate every bucket from a multi-level aggregation effectively. An example of....

In Elasticsearch, the constant score query wraps other queries by executing them in a filter context. To implement constant_score query...

The exists query is used for returning the documents that have an indexed value for a specific field, which means it returns the documents...

Elasticsearch Intervals query provides control over the words & their positions in a text that is required for a document to match a...

Elasticsearch keyword vs. text vs. wildcard vs. text field types. All have different features and are ideal for different use cases

Match, Multi-Match & Match Phrase are all types of Elasticsearch queries, used to search for matching documents in an index. To use them...

Elasticsearch runtime fields with a type of lookup can retrieve field values from the associated indices using the fields parameter on...

The Elasticsearch async search API retrieves many data in a stream fashion instead of a single request. To limit the maximum response size...

Named queries allow you to label your queries with a name. Named queries can be utilized in a variety of use cases such as...

There are several potential reasons for a slow query in Elasticsearch. Slow logs can be used to detect & troubleshoot slow queries issues...

In Elasticsearch, the Terms enum API looks for similarities in the index based on partial matches. To use the terms_enum API...

Role & user mapping is the process that links users from an external system to roles in Elasticsearch. The role mapping API can be...

Securing an Elasticsearch cluster and creating TLS certificates will require some downtime on your cluster. Here's how to create...

The popularity of Elasticsearch has made it a target for hackers. It's important to protect your cluster by enabling X-Pack Security...