I'm wondering if the timeout occurs due to a JVM garbage collection due to a
large number of Lucene segments to merge.
What is the JVM heap usage like, compared to the total heap space available?
In other words, maybe the JVM needs more heap memory.
-- Jack Krupansky
-----Original Message-----
From: Markus Jelsma
Sent: Tuesday, August 07, 2012 5:39 PM
To: solr-user@lucene.apache.org ; Markus Jelsma
Subject: RE: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
A signicant detail is the batch size which we set to 64 documents due to
earlier memory limitations. We index segments of roughly 300-500k records
each time. Lowering the batch size to 32 lead to an early internal server
error and the stack trace below. Increasing it to 128 allowed us to index
some more records but it still throws the error after 200k+ indexed records.
Increasing it even more to 256 records per batch allowed us to index an
entire segment without errors.
Another detail is that we do not restart the cluster between indexing
attempts so it seems that something only builds up during indexing (nothing
seems to leak afterwards) and throws an error.
Any hints?
Thanks,
Markus
-----Original message-----
From:Markus Jelsma <markus.jel...@openindex.io>
Sent: Tue 07-Aug-2012 20:08
To: solr-user@lucene.apache.org
Subject: null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
Hello,
We sometimes see the error below in our `master` when indexing. Our master
is currently the node we send documents to - we've not yet implemented
CloudSolrServer in Apache Nutch. This causes the indexer to crash when
using Nutch locally, the task is retried when running on Hadoop. We're
running it locally in this test set up so there's only one indexing
thread.
Anyway, for me it's quite a cryptic error because i don't know what
connection has timed out, i assume a connection from the indexing node to
some other node in the cluster when it passes a document to the correct
leader? Each node of the 10 node cluster has the same configuration,
Tomcat is configured with maxThreads=512 and a time out of one second.
We're using today's trunk in this test set up and we cannot reliably
reproduce the error. We've seen the error before so it's not a very recent
issue. No errors are found in the other node's logs.
2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] -
[http-8080-exec-6] - : null:java.lang.RuntimeException: [was class
java.net.SocketTimeoutException] null
at
com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
at
com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
at
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
at
org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229)
at
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
at
io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException
at
org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
at
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229)
at
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210)
at
org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643)
at
org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945)
at
org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR
ead(InternalNioInputBuffer.java:969)
at
org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268)
at
org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167)
at
org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916)
at org.apache.coyote.Request.doRead(Request.java:427)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
at
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419)
at
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at
com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
at
com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
at
com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
at
com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
... 24 more
Any thoughts to share? Is this a bug? A misconfiguration?
Thanks,
Markus
