Hi all! I have a zookeeper cluster with 3 nodes, version 3.9.1. The cluster was running OK for over 10 months, then all of a sudden the logs on all 3 nodes became flooded (over 3GB/day) with the following messages:
2024-11-25 09:57:04,613 [myid:] - WARN [CommitProcessor:1:o.a.z.s.RateLogger@87] - Message:Digests are not matching. Value is Zxid. Value:7726973638671 2024-11-25 09:57:04,623 [myid:] - WARN [CommitProcessor:1:o.a.z.s.RateLogger@87] - Message:Digests are not matching. Value is Zxid. Value:7726973638677 2024-11-25 09:57:04,630 [myid:] - WARN [CommitProcessor:1:o.a.z.s.RateLogger@87] - Message:Digests are not matching. Value is Zxid. Value:7726973638683 2024-11-25 09:57:04,639 [myid:] - WARN [CommitProcessor:1:o.a.z.s.RateLogger@87] - Message:Digests are not matching. Value is Zxid. Value:7726973638688 I've tried restarting the nodes one by one, rebooting the VMs one by one - all to no avail. Other than the logs that I had to turn off I'm not experiencing any noticeable issues. however I don't feel comfortable leaving it like that. Googling got me to a JIRA issue that seems to be describing the same situation, however there's no response or any other activity there (see [1]). So I'm trying my luck here. Question are: 1. What do I do with this, is there any way to "repair" without recreating the cluster from scratch? 2. What could have caused it? There were no hard stops, hardware failures or anything else that I'm aware of. The load shouldn't have changed a lot. 3. If I have to recreate the cluster - how do I avoid this in the future? Should I monitor something? Thanks in advance, Alexander
