Troubleshooting¶
Recovery default base indexes¶
Only applies to versions 6.1.5 and older. From version 6.1.6 and later, default indexes are created automatically
If you lost or damage following index:
|Index name | Index ID |
|----------------|-----------------------|
| .security |Pfq6nNXOSSmGhqd2fcxFNg |
| .taskmanagement|E2Pwp4xxTkSc0gDhsE-vvQ |
| alert_status |fkqks4J1QnuqiqYmOFLpsQ |
| audit |cSQkDUdiSACo9WlTpc1zrw |
| alert_error |9jGh2ZNDRumU0NsB3jtDhA |
| alert_past |1UyTN1CPTpqm8eDgG9AYnw |
| .trustedhost |AKKfcpsATj6M4B_4VD5vIA |
| .kibana |cmN5W7ovQpW5kfaQ1xqf2g |
| .scheduler_job |9G6EEX9CSEWYfoekNcOEMQ |
| .authconfig |2M01Phg2T-q-rEb2rbfoVg |
| .auth |ypPGuDrFRu-_ep-iYkgepQ |
| .reportscheduler|mGroDs-bQyaucfY3-smDpg |
| .authuser |zXotLpfeRnuzOYkTJpsTaw |
| alert_silence |ARTo7ZwdRL67Khw_HAIkmw |
| .elastfilter |TtpZrPnrRGWQlWGkTOETzw |
| alert |RE6EM4FfR2WTn-JsZIvm5Q |
| .alertrules |SzV22qrORHyY9E4kGPQOtg |
You may to recover it from default installation folder with following steps:
Stop Logstash instances which load data into cluster
systemctl stop logstash
Disable shard allocation
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "none" } }
Stop indexing and perform a synced flush
POST _flush/synced
Shutdown all nodes:
systemctl stop elasticsearch.service
Copy appropriate index folder from installation folder to Elasticsearch cluster data node folder (example of .auth folder)
cp -rf ypPGuDrFRu-_ep-iYkgepQ /var/lib/elasticsearch/nodes/0/indices/
Set appropriate permission
chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
Start all Elasticsearch instance
systemctl start elasticsearch
Wait for yellow state of Elasticsearch cluster and then enable shard allocation
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "all" } }
Wait for green state of Elasticsearch cluster and then start the Logstash instances
systemctl start logstash
To many open files¶
If you have a problem with too many open files by the Elasticsearch process, modify the values in the following configuration files:
- /etc/sysconfig/elasticsearch
- /etc/security/limits.d/30-elasticsearch.conf
- /usr/lib/systemd/system/elasticsearch.service
Check these three files for:
- LimitNOFILE=65536
- elasticsearch nofile 65537
- MAX_OPEN_FILES=65537
Changes to service file require:
systemctl daemon-reload
And changes to limits.d require:
sysctl -p /etc/sysctl.d/90-elasticsearch.conf
The Kibana status code 500¶
If the login page is displayed in Kibana, but after the attempt to login, the browser displays “error: 500”, and the logs will show entries:
Error: Failed to encode cookie (sid-auth) value: Password string too short (min 32 characters required).
Generate a new server.ironsecret with the following command:
echo "server.ironsecret: \"$(</dev/urandom tr -dc _A-Z-a-z-0-9 | head -c32)\"" >> /etc/kibana/kibana.yml