On Wed, Oct 7, 2009 at 12:51 PM, Phillip Farber <pfar...@umich.edu> wrote: > > In a separate thread, I've detailed how an optimize is taking > 2x disk > space. We don't use solr distribution/snapshooter. We are using the default > deletion policy = 1. We can't optimize a 192G index in 400GB of space. > > This thread in lucene/java-user > > http://www.gossamer-threads.com/lists/lucene/java-user/43475 > > suggests that an optimize should not take > 2x unless perhaps an IndexReader > is holding on to segments. This could be our problem since when optimization > runs out of space, if we stop tomcat, a number of files go away and space is > recovered. > > But we are not searching the index so how could a Searcher/IndexReader have > any segments open? > > I notice in the logs that as part of routine commits or as part of optimize > a Searcher is registered and autowarmed from a previous searcher (of course > there's nothing in the caches -- this is just a build machine). > > INFO: registering core: > Oct 6, 2009 2:16:20 PM org.apache.solr.core.SolrCore registerSearcher > INFO: [] Registered new searcher searc...@2e097617 main > > Does this means that there's always a lucene IndexReader holding segment > files open so they can't be deleted during an optimize so we run out of disk > space > 2x?
Yes. A feature could probably now be developed now that avoids opening a reader until it's requested. That wasn't really possible in the past - due to many issues such as Lucene autocommit. -Yonik http://www.lucidimagination.com
