Source shard ” + routingEntry + ” is not an active primary – How to solve this Elasticsearch error

Overview

In Elasticsearch, recovery refers to the process of recovering an index or shard when something goes wrong. There are many ways to recover an index or shard, such as by re-indexing the data from a backup / failover cluster to the current one, or by restoring from an Elasticsearch snapshot. Alternatively, Elasticsearch performs recoveries automatically, such as when a node restarts or disconnects and connects again. There is an API to check the updated status of index / shard recoveries.

GET /<index>/_recoveryGET /_recovery

In summary, recovery can happen in the following scenarios:

• Node startup or failure (local store recovery)
• Replication of primary shards to replica shards
• Relocation of a shard to a different node in the same cluster
• Restoring a snapshot

Examples

Getting recovery information about several indices:

GET my_index1
GET my_index2/_recovery

Notes and good things to know

• When a node is disconnected from the cluster, all of its shards go to an unassigned state. After a certain amount of time, the shards will be allocated somewhere else on other nodes. This setting determines the number of concurrent shards per node that will be recovered.

PUT _cluster/settings{"transient":{"cluster.routing.allocation.node_concurrent_recoveries":3}}

• You can also control when to start recovery after a node disconnects. This is useful if the node just restarts, for example, because you may not want to initiate any recovery for such transient events.

PUT _all/_settings{"settings":{"index.unassigned.node_left.delayed_timeout":"6m"}}

• Elasticsearch limits the speed that is allocated to recovery in order to avoid overloading the cluster. This setting can be updated to make the recovery faster or slower, depending on your requirements.

PUT _cluster/settings{"transient":{"indices.recovery.max_bytes_per_sec":"100mb"}}

Overview

In Elasticsearch, routing refers to document routing. When you index a document, Elasticsearch will determine which shard the document should be routed to for indexing. 

The shard is selected based on the following formula:

shard = hash(_routing) % number_of_primary_shards

Where the default value of _routing is _id.
It is important to know which shard the document is routed to, because Elasticsearch will need to determine where to find that document later on for document retrieval requests. 

Examples

In twitter index with 2 primary shards, the document with _id equal to “440” gets routed to the shard number:   

shard = hash( 440 ) % 2
PUT twitter/_doc/440
{
...
}

Notes and good things to know

• In order to improve search speed, you can create custom routing. For example, you can enable custom routing that will ensure that only a single shard will be queried (the shard that contains your data).
• To create custom routing in Elasticsearch, you will need to configure and define that not all routing will be completed by default settings. ( v <= 5.0)

PUT my_index/customer/_mapping
{
   "order":{
      "_routing":{
         "required":true
      }
   }
}

•  This will ensure that every document in the “customer” type must specify a custom routing. For Elasticsearch version 6 or above you will need to update the same mapping as:

PUT my_index/_mapping
{
   "order":{
      "_routing":{
         "required":true
      }
   }
}

Overview

When a document is sent for indexing, Elasticsearch indexes all the fields in the format of an inverted index, but it also keeps the original JSON document in a special field called _source. 

Examples

Disabling source field in the index:

PUT /api-logs?pretty
{
  "mappings": {
    "_source": {
      "enabled": false
    }
  }
}

Store only selected fields as a part of _source field:

PUT api-logs
{
  "mappings": {
    "_source": {
      "includes": [
        "*.count",
        "error_info.*"
      ],
      "excludes": [
        "error_info.traceback_message"
      ]
    }
  }
}

Including only selected fields using source filtering:

GET api-logs/_search
{
  "query": {
    "match_all": {}
  },
  "_source": {
       "includes": ["api_name","status_code", "*id"]
  }
}

Notes

The source field brings an overhead of extra storage space but serves special purposes such as:

• Return as a part of the response when a search query is executed.
• Used for reindexing purpose, update and update_by_query operations.
• Used for highlighting, if the field is not stored, it means  the field is not set as “store to true” inside the mapping.
• Allows selection of fields to be returned.

The only concern with source field is the extra storage usage on disk. But this storage space used by source field can be optimized by changing compression level to best_compression. This setting is done using index.codec parameter.