>Without a snapshot, we cannot delete the log files, as we would have no >means of recovery. txn logs applied to the snapshot gives us back the >state. Without snapshot, all txn logs needs to be "replayed" in a recovery. >And you need all the log files created since your last snapshot (in this >case, all the txn logs as there were no snapshots yet).
Makes sense. In the heaviest environment I'm hitting around 200 requests per second, which all have to get data from zookeeper. Not sure what the impact is in terms of snapCount, I didn't set up the system myself and don't fully grasp the internals. As for the snapCount, that hasn't been touched, so that will be the default in my environments. I've read that the log files are preallocated. I see them as being 65MB a piece. Which makes me wonder: how many of those until the process hits 100000 snaps? auto-cleaning is setup, retainCount=3, purgeInterval=1 Restarting the zookeeper process shouldn't affect this count, I think? It doesn't happen often, though it might on test environments. On Wed, Aug 14, 2019 at 2:06 PM Norbert Kalmar <nkal...@cloudera.com.invalid> wrote: > Hi, > > Without a snapshot, we cannot delete the log files, as we would have no > means of recovery. txn logs applied to the snapshot gives us back the > state. Without snapshot, all txn logs needs to be "replayed" in a recovery. > And you need all the log files created since your last snapshot (in this > case, all the txn logs as there were no snapshots yet). > > As for why there is no snapshot. What is your load? Per the admin guide: > > "snapCount > (Java system property: zookeeper.snapCount) > ZooKeeper logs transactions to a transaction log. After snapCount > transactions are written to a log file a snapshot is started and a new > transaction log file is created. The default snapCount is 100,000." > > By default there will be no auto-cleaning of the snapshot and log files. > Check the autopurge.snapRetainCount and autopurge.purgeInterval settings > for this. > > Regards, > Norbert > > On Wed, Aug 14, 2019 at 1:21 PM Koen De Groote < > koen.degro...@limecraft.com> > wrote: > > > Greetings all. > > > > I was debugging something an ran into this bit of code: > > > > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/PurgeTxnLog.java#L81 > > > > If I understand it correctly, it seems that this means log.xxxxx files > will > > only get deleted if there's a snapshot. > > > > Which is troublesome, as my dataDir is filling up with log files but not > a > > single snapshot in sight. > > > > 1: Is this correct behavior? Both the logic of needing a snapshot and the > > fact that not snapshots are being generated? > > > > 2: While not having a fix for this, what would be useful to know is: can > > these log files be freely deleted? Or does the most recent one need to be > > kept, or how does it go with this files? > > > > I thought that running "bin/zkCleanup.sh /data -n 3" would clean up both > > the snapshots and the logs, but it appears that if there are not > snapshots, > > the logs aren't cleaned either. > > > > What are my options here? > > > > Kind regards, > > Koen De Groote > > >
