Ah, sorry I wasn't clear. There's currently 3 log/snapshot pairs in that dir, because I cleared it out recently. If I keep starting and stopping solr/zookeeper, they'll continue to pile up.
$ ls -lah drwxr-xr-x 14 ryanfox staff 476B Nov 4 17:25 . drwxr-xr-x 4 ryanfox staff 136B Nov 4 09:30 .. -rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1 -rw-r--r-- 1 ryanfox staff 64M Nov 4 17:25 log.112 -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48 -rw-r--r-- 1 ryanfox staff 64M Nov 4 17:22 log.7b -rw-r--r-- 1 ryanfox staff 64M Nov 4 17:22 log.ad -rw-r--r-- 1 ryanfox staff 64M Nov 4 17:25 log.e1 -rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0 -rw-r--r-- 1 ryanfox staff 60K Nov 4 17:25 snapshot.111 -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47 -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a -rw-r--r-- 1 ryanfox staff 60K Nov 4 17:22 snapshot.ac -rw-r--r-- 1 ryanfox staff 60K Nov 4 17:23 snapshot.e0 Thanks for the info on preAllocSize, that may prove useful as well. Nothing showed up in the logs today despite running for ~8 hours. Best, Ryan On Mon, Nov 4, 2013 at 5:13 PM, Patrick Hunt <[email protected]> wrote: > On Mon, Nov 4, 2013 at 7:46 AM, Ryan Fox <[email protected]> wrote: > >> > >> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old, > >> 4.5.1 is out now, perhaps try upgrading?) > >> > >> > > > Zookeeper version is 3.4.5. This is part of a large software package, so > > upgrading the solr library would be painful, but possible. > > > > 3.4.5 is fine, I just wasn't sure what version solr was using, if it > was old it might be related but I don't think that's the case here. > > >> "do not get autopurged" - could you provide more insight. 1) are you > >> sure the configuration options are being applied? 2) what does the > >> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in > >> this thread) 3) anything autopurge related in the zk server logs? > >> > > > Sure, the datadir contains only logs and snapshots. I would expect the > > autoPurge function to delete logs and snapshots older than the number I > > have set in zoo.cfg, but nothing gets deleted, and they just pile up, > > eating up my hard drive. > > 1) The zookeeper instance is picking up my settings for clientPort, so I > > assume I have the purge settings in the correct place as well. The > > relevant section of zoo.cfg: > > autopurge.snapRetainCount=3 > > autopurge.purgeInterval=1 > > > > 2) ls -lah > > -rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1 > > -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48 > > -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:37 log.7b > > -rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0 > > -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47 > > -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a > > > > You have our snapRetainCount set to 3. That's why three snaps (and > their related logs) are saved. > > Try setting it to 2. > > > A new log and snapshot gets created every time I stop and start solr. As > > an aside, every log file appears to be 64MB, no matter how much/little is > > in my index. Is that correct? Seems like a lot for an empty solr > instance. > > > > We pre-allocate the log files for performance. See "preAllocSize" here: > http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html > > If you want smaller files try setting that to something like 10k or > somesuch. > > > 3) I haven't seen any smoking gun in the logs, but haven't specifically > > looked, either. I'll let it go until it should have triggered the > > auto-purge and report back. > > See if the above suggestions/info help. > > Obviously don't carry over these changes to production. ;-) > > Regards, > > Patrick >
