Thanks Hoss. Yes, in a separate thread on the list I reported that
doing a multi-stage optimize worked around the out of space problem. We
use mergefactor=10, maxSegments = 16, 8, 4, 2, 1 iteratively starting at
the closest power of two below the number of segments to merge. Works
nicely so far. And compoundFile=false.
Regards,
Phil
Chris Hostetter wrote:
: That theory did not work because the error log showed that solr was trying to
: merge into the _1j37 segment files showing as deleted in the lsof above when
: it ran out of space so those are a symptom not a cause of the lost space:
Right, you have to keep in mind Solr is always maintaining an open
searcher, so even as files get deleted because of a merge, the disk space
is still being used because Solr is maintaining open file handles to those
deleted files (for searching) so the os can't free up that space.
have you tried doing iterative partial optimizes so Solr can re-open the
index more frequently (and release some of those file handles sooner) ?
Also: what does your "useCompoundFile" option look like in solrconfig.xml,
becuase if it's "true" then you (aparently) need 3X not 2X for an
optimize...
http://www.nabble.com/How-much-disk-space-does-optimize-really-take-to25790344.html#a25792748
-Hoss
