I think that argument requires auto commit to be on and opening readers after the optimize starts? Otherwise, the optimized version is not put into place until a commit is called, and a Reader won't see the newly merged segments until then - so the original index is kept around in either case - having a Reader open on it shouldn't affect the space requirements?
Yonik Seeley wrote: > On Wed, Oct 7, 2009 at 12:51 PM, Phillip Farber <pfar...@umich.edu> wrote: > >> In a separate thread, I've detailed how an optimize is taking > 2x disk >> space. We don't use solr distribution/snapshooter. We are using the default >> deletion policy = 1. We can't optimize a 192G index in 400GB of space. >> >> This thread in lucene/java-user >> >> http://www.gossamer-threads.com/lists/lucene/java-user/43475 >> >> suggests that an optimize should not take > 2x unless perhaps an IndexReader >> is holding on to segments. This could be our problem since when optimization >> runs out of space, if we stop tomcat, a number of files go away and space is >> recovered. >> >> But we are not searching the index so how could a Searcher/IndexReader have >> any segments open? >> >> I notice in the logs that as part of routine commits or as part of optimize >> a Searcher is registered and autowarmed from a previous searcher (of course >> there's nothing in the caches -- this is just a build machine). >> >> INFO: registering core: >> Oct 6, 2009 2:16:20 PM org.apache.solr.core.SolrCore registerSearcher >> INFO: [] Registered new searcher searc...@2e097617 main >> >> Does this means that there's always a lucene IndexReader holding segment >> files open so they can't be deleted during an optimize so we run out of disk >> space > 2x? >> > > Yes. > A feature could probably now be developed now that avoids opening a > reader until it's requested. > That wasn't really possible in the past - due to many issues such as > Lucene autocommit. > > -Yonik > http://www.lucidimagination.com > -- - Mark http://www.lucidimagination.com
