Hello everyone, A few weeks ago our team detected that zookeeper had a volume that was consuming a lot of disk. We found out that the folder that was using too much disk was the dataDir where the data logs and the snapshots are being stored. After some research we found out that we were not rotating those logs so we added to the zoo.cfg the following variables:
autopurge.snapRetainCount=3 autopurge.purgeInterval=1 What we discovered from the logs was strange: -The purge was running (every hour). -Eventhough it was running, it wasn't doing anything and the folder would get bigger and bigger. Environment: Running 3 zookeeper pods in OpenShift using a PVC to store this type of logs (the data logs and the snapshots). Zookeeper version 3.8.4 Log snippet of purge running: 2024-12-12 12:14:12,224 [myid:] - INFO [PurgeTask:o.a.z.s.DatadirCleanupManager$PurgeTask@139] - Purge task started. 2024-12-12 12:14:12,224 [myid:1] - INFO [main:o.a.z.s.q.QuorumPeerMain@152] - Starting quorum peer, myid=1 2024-12-12 12:14:12,226 [myid:] - INFO [PurgeTask:o.a.z.s.p.FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2024-12-12 12:14:12,324 [myid:] - INFO [PurgeTask:o.a.z.s.p.SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED 2024-12-12 12:14:12,327 [myid:] - INFO [PurgeTask:o.a.z.s.DatadirCleanupManager$PurgeTask@145] - Purge task completed. Log folder: $ cd /var/lib/zookeeper/data/log/version-2 $ ls -la total 608 drwxrwsr-x. 2 9994 zookeeper 12288 Dec 12 12:15 . drwxrwsr-x. 3 9994 zookeeper 4096 Apr 9 2024 .. -rw-rw-r--. 1 zookeeper zookeeper 67108880 Dec 5 17:58 log.1e900000001 -rw-rw-r--. 1 zookeeper zookeeper 67108880 Dec 6 08:23 log.1ea00000001 -rw-rw-r--. 1 zookeeper zookeeper 67108880 Dec 7 15:59 log.1eb00000001 -rw-rw-r--. 1 zookeeper zookeeper 67108880 Dec 9 10:23 log.1ec00000001 -rw-rw-r--. 1 zookeeper zookeeper 67108880 Dec 10 18:34 log.1ed00000001 -rw-r--r--. 1 zookeeper zookeeper 67108880 Dec 12 12:15 log.1ee00000001 Can someone help me identify what is happening? Thank you all in advance!
