After a much closer look at the logs, I found these errors:
Looks like the errors started from the 3rd day after my server was running.
After the initial Java heap space error Jul 15, 2010 9:46:34 AM
Solr did run for a certain time till Jul 15, 2010 8:13:33 PM when it
started failing due to the missing file....
What could cause this?
catalina.2010-07-15.log:SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 9:46:34 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.search.FieldCacheImpl$StringIndexCache.createValue(FieldCacheImpl.java:658)
at
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:205)
at
org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:646)
at
org.apache.lucene.search.FieldComparator$StringOrdValComparator.setNextReader(FieldComparator.java:667)
at
org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:91)
at
org.apache.solr.search.DocSetDelegateCollector.setNextReader(DocSetHitCollector.java:140)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:251)
at org.apache.lucene.search.Searcher.search(Searcher.java:173)
at
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:174)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
catalina.2010-07-15.log:SEVERE: Exception invoking periodic operation:
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
catalina.2010-07-15.log:SEVERE: auto commit error...
Jul 15, 2010 7:10:31 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:35 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:39 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:42 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 2039
Jul 15, 2010 7:10:43 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:47 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 14741
Jul 15, 2010 7:10:53 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:55 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:10:49 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:11:10 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 13379
Jul 15, 2010 7:11:10 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:11:13 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:11:15 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: Java heap space
Jul 15, 2010 7:11:17 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 672
Jul 15, 2010 7:11:19 PM org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Invalid chunk header
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:73)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.ctc.wstx.exc.WstxIOException: Invalid chunk header
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:548)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:604)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:660)
at
com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:68)
... 17 more
Caused by: java.io.IOException: Invalid chunk header
at
org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:133)
at
org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:710)
at org.apache.coyote.Request.doRead(Request.java:428)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
at
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:405)
at
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at
com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:245)
at
com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:132)
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:543)
... 21 more
Jul 15, 2010 7:11:19 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=javabin&version=1} status=500
QTime=672
Jul 15, 2010 7:11:19 PM org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Invalid chunk header
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:73)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.ctc.wstx.exc.WstxIOException: Invalid chunk header
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:548)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:604)
at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:660)
at
com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:68)
... 17 more
Caused by: java.io.IOException: Invalid chunk header
at
org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:133)
at
org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:710)
at org.apache.coyote.Request.doRead(Request.java:428)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
at
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:405)
at
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at
com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:245)
at
com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:132)
at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:543)
... 21 more
SEVERE: auto commit error...
Jul 15, 2010 8:13:33 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
Jul 15, 2010 8:13:33 PM
org.apache.solr.update.DirectUpdateHandler2$CommitTracker run
SEVERE: auto commit error...
Jul 15, 2010 8:13:33 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
Jul 15, 2010 8:13:33 PM
org.apache.solr.update.DirectUpdateHandler2$CommitTracker run
SEVERE: auto commit error...
Jul 15, 2010 8:13:33 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 6622
Jul 15, 2010 8:13:33 PM org.apache.solr.common.SolrException log
SEVERE: java.io.FileNotFoundException: /solr_data/data/index/segments_jes
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:77)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:107)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:93)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:69)
at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:671)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:236)
at
org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:309)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:538)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:306)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1572)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1397)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:181)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:140)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
catalina.2010-07-15.log:SEVERE: java.io.FileNotFoundException:
/solr_data/data/index/segments_jes (No such file or directory)
catalina.2010-07-15.log:SEVERE: java.io.FileNotFoundException:
/solr_data/data/index/segments_jes (No such file or directory)
catalina.2010-07-15.log:SEVERE: java.io.FileNotFoundException:
/solr_data/data/index/segments_jes (No such file or directory)
catalina.2010-07-15.log:SEVERE: java.io.FileNotFoundException:
/solr_data/data/index/segments_jes (No such file or directory)
catalina.2010-07-15.log:SEVERE: java.io.FileNotFoundException:
/solr_data/data/index/segments_jes (No such file or directory)
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-Index-corruption-tp980425p980472.html
Sent from the Solr - User mailing list archive at Nabble.com.
