Not sure but a quick search turned up:
http://www.walkernews.net/2007/07/13/df-and-du-command-show-different-used-disk-space/
Using upto 2x the index size can happen. Also check if there is a
snapshooter script running through cron which is making hard links to files
while a merge is in progress.
Do let us know if you make any progress. This is interesting.
On Tue, Oct 6, 2009 at 5:28 PM, Phillip Farber <pfar...@umich.edu> wrote:
> I am attempting to optimize a large shard on solr 1.4 and repeatedly get
> java.io.IOException: No space left on device. The shard, after a final
> commit before optimize, shows a size of about 192GB on a 400GB volume. I
> have successfully optimized 2 other shards that were similarly large without
> this problem on identical hardware boxes.
>
> Before the optimize I see:
>
> % df -B1 .
> Filesystem 1B-blocks Used Available Use% Mounted on
> /dev/mapper/internal-solr--build--2
> 435440427008 205681356800 225335255040 48%
> /l/solrs/build-2
>
> slurm-4:/l/solrs/build-2/data/index % du -B1
> 205441486848 .
>
> There's a slight discrepancy between the du and df which appears to be
> orphaned inodes. But the du says there should be enough space to handle the
> doubling in size during optimization. However, for the second time we run
> out of space and the du and df are wildly different at that point and the
> volume is at 100%
>
>
> % df -B1 .
>
> Filesystem 1B-blocks Used Available Use% Mounted on
> /dev/mapper/internal-solr--build--2
> 435440427008 430985760768 30851072 100%
> /l/solrs/build-2
>
> slurm-4:/l/solrs/build-2/data/index % du -B1
> 252552298496 .
>
> At this point it appears orphaned inodes are consuming space and not being
> freed-up. Any clue as to whether this is a lucene bug a solr bug or some
> other problem. Error traces follow.
>
> Thanks!
>
> Phil
>
> ---
>
> Oct 6, 2009 2:12:37 AM org.apache.solr.update.processor.LogUpdateProcessor
> finish
> INFO: {} 0 9110523
> Oct 6, 2009 2:12:37 AM org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: background merge hit exception: _ojl:C151080
> _169w:C141302 _1j36:C80405 _1j35:C2043 _1j34:C192 into _1j37 [optimize]
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2737)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2658)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:401)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: No space left on device
> at java.io.RandomAccessFile.writeBytes(Native Method)
> at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:719)
> at
> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
> at
> org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:85)
> at
> org.apache.lucene.store.BufferedIndexOutput.seek(BufferedIndexOutput.java:124)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.seek(FSDirectory.java:744)
> at
> org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:220)
> at
> org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:493)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4838)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4429)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>
> Oct 6, 2009 2:12:37 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/build-2 path=/update params={} status=500 QTime=9110523
>
>
>
--
Regards,
Shalin Shekhar Mangar.
