Re: old searchers not closing after optimize or replication

2011-04-21 Thread Bernd Fehling
Hi Erik, deletionPolicy class=solr.SolrDeletionPolicy str name=maxCommitsToKeep1/str str name=maxOptimizedCommitsToKeep0/str /deletionPolicy Due to 44 minutes optimization time we do an optimization once a day during the night. I will try with an smaler index on my development system. Best

Re: old searchers not closing after optimize or replication

2011-04-21 Thread Trey Grainger
Hey Bernd, Checkout https://issues.apache.org/jira/browse/SOLR-2469.  There is a pretty bad bug in Solr 3.1 which occurs if you have  str name=replicateAfterstartup/str set in your replication configuration in solrconfig.xml.  See the thread between Yonik and myself from a few days ago titled

old searchers not closing after optimize or replication

2011-04-20 Thread Bernd Fehling
Hello list, we have the problem that old searchers often are not closing after optimize (on master) or replication (on slaves) and therefore have huge index volumes. Only solution so far is to stop and start solr which cleans up everything successfully, but this can only be a workaround. Is the

Re: old searchers not closing after optimize or replication

2011-04-20 Thread Erick Erickson
Does this persist? In other words, if you just watch it for some time, does the disk usage go back to normal? Because it's typical that your index size will temporarily spike after the operations you describe as new searchers are warmed up. During that interval, both the old and new searchers are

Re: old searchers not closing after optimize or replication

2011-04-20 Thread Bernd Fehling
Hi Erik, Am 20.04.2011 13:56, schrieb Erick Erickson: Does this persist? In other words, if you just watch it for some time, does the disk usage go back to normal? Only after restarting the whole solr the disk usage goes back to normal. Because it's typical that your index size will

Re: old searchers not closing after optimize or replication

2011-04-20 Thread Erick Erickson
H, this isn't right. You've pretty much eliminated the obvious things. What does lsof show? I'm assuming it shows the files are being held open by your Solr instance, but it's worth checking. I'm not getting the same behavior, admittedly on a Windows box. The only other thing I can think of

Re: old searchers not closing after optimize or replication

2011-04-20 Thread Bernd Fehling
Hi Erik, Am 20.04.2011 15:42, schrieb Erick Erickson: H, this isn't right. You've pretty much eliminated the obvious things. What does lsof show? I'm assuming it shows the files are being held open by your Solr instance, but it's worth checking. Just commited new content 3 times and

Re: old searchers not closing after optimize or replication

2011-04-20 Thread Erick Erickson
It looks OK, but still doesn't explain keeping the old files around. What is your deletionPolicy in your solrconfig.xml look like? It's possible that you're seeing Solr attempt to keep around several optimized copies of the index, but that still doesn't explain why restarting Solr removes them