Hi, I am working on upgrading a Kubernetes based 3.4.14 cluster to 3.5.6.
I are hitting the issue where the new snapshot checks prevent the newly upgraded 3.5.6 cluster from starting. This usually happens in our test set-up as the 3.4.14 cluster is not very old and has not hit the snapCount limit yet. Typically only one node (presumable the leader) has a snapshot file but the other do not. I can disable the snapshot checks via the config value for the upgrade to 3.5.6, however we obviously need to re-enable the checks ASAP. The issue is making sure all of the ZK nodes have snapshots so that the checks can be re-enabled. My question is: Is there a way to force the snapshot creation / sync from the leader? As part of our upgrade process we need to restart the servers soon after the initial upgrade of the ZK container images to enable config changes. If the snapshot creation / sync hasn't happened by this point then obviously the checks will fail. If we can force a snapshot creation, after the initial upgrade, we can re-enable the checks as part of the upgrade process. If not, we will need to create some method to check if the snapshots have been created and periodically inspect the ZK server file system, re-enabling the checks once they all have snapshots. Any pointers would be appreciated. Thanks in advance, Tom Cooper
