Elasticsearch Upgrade from 7.17 to 8.8

Hello,
Today I want to talk about that we faced issues during our Elasticsearch upgrade process to 8.8. This is classical upgrade process for Elasticsearch. But we got some critical issue when to do this. Because of some of these issues, we had to roll back the upgrade process. We retried the upgrade operation after making changes mentioned below.

1) Elasticsearch Timeout Error After Upgrade

Firstly, I have already wrote about “Elasticsearch Timeout Error After Upgrade”. It was easy to handle this issue because of it is a known issue. This error is not entirely related to Elasticsearch. It is related to systemd and elasticsearch’s service configuration file.

2) Preupgrade checks – Kibana doesn’t show all depracated indices !!

You should use upgrade assistant on your Kibana to get Elasticsearch deprecation issues. Login to kibana and navigate Management > Stack Management > Upgrade assistant. You can see the changes you need to make before upgrade. If there is a critical findings, you must take an action.

There are some findings about that created with version 6.X in our scenario. We reindex these indices, and we complete the upgrade processes. But surprise :)!! When we try to start Elasticsearch services we got error shown as below.

[ERROR][o.e.b.Elasticsearch ] [node-1] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: Cannot start this node because it holds metadata for indices created with version [6.6.2] with which this node of version [8.8.2] is incompatible. Revert this node to version [7.17.2] and delete any indices created in versions earlier than [7.0.0] before upgrading to version [8.8.2]. If all such indices have already been deleted, revert this node to version [7.17.2] and wait for it to join the cluster to clean up any older indices from its metadata.

This error message shows, there are still some indices that created older versions. But kibana doesn’t inspect them. So we deleted Elasticsearch 8.8 packages, and backed up our data folders again. And reinstalled older Elasticsearch packages with yum. (We didn’t rolling upgrade). We could start Elasticsearch again and used the curl command shown as below to inspect all indices that created older version.

GET _migration/deprecations?pretty

You must clean or reindex all deprecated indices that gives this command. So we did this and Elasticsearch started successfully.

Conclusion

There is a bug in Kibana Upgrade Assistant Page. It is not show all deprecated indices. So you should use curl command to inspect deprecations.

Have a nice days. Thank you for your time. You can follow my youtube channel from here and you can read other elasticsearch posts from here.