I've increased the connection time out on all 10 Tomcats from 1000ms to 5000ms. Indexing a larger amount of batches seems to run fine now. This, however, does not really answer the issue. What is exactly timing out here and why? I assume its the forwarding of documents from the `indexing node` to the correct shard leader but with 512 maxThreads it should be fine.
Any hints? Thanks -----Original message----- > From:Markus Jelsma <markus.jel...@openindex.io> > Sent: Wed 08-Aug-2012 00:10 > To: solr-user@lucene.apache.org > Subject: RE: null:java.lang.RuntimeException: [was class > java.net.SocketTimeoutException] null > > Jack, > > There are no peculiarities in the JVM graphs. Only increase in used threads > and GC time. Heap space is collected quickly and doesn't suddenly increase. > There's only 256MB available for the heap but it's fine. > > > Yonik, > > I'll increase the time out to five seconds tomorrow and try to reproduce it > with a low batch size of 32. Juding from what i've seen it should throw an > error quickly with such a low batch size. However, what is timing out here? > My client connection to the indexing node or something else that i don't see? > > Unfortunately no Jetty here (yet). > > Thanks > Markus > > > -----Original message----- > > From:Yonik Seeley <yo...@lucidimagination.com> > > Sent: Tue 07-Aug-2012 23:54 > > To: solr-user@lucene.apache.org > > Subject: Re: null:java.lang.RuntimeException: [was class > > java.net.SocketTimeoutException] null > > > > Could this be just a simple case of a socket timeout? Can you raise > > the timout on request threads in Tomcat? > > It's a lot easier to reproduce/diagnose stuff like this when people > > use the stock jetty server shipped with Solr. > > > > -Yonik > > http://lucidimagination.com > > > > > > On Tue, Aug 7, 2012 at 5:39 PM, Markus Jelsma > > <markus.jel...@openindex.io> wrote: > > > A signicant detail is the batch size which we set to 64 documents due to > > > earlier memory limitations. We index segments of roughly 300-500k records > > > each time. Lowering the batch size to 32 lead to an early internal server > > > error and the stack trace below. Increasing it to 128 allowed us to index > > > some more records but it still throws the error after 200k+ indexed > > > records. > > > > > > Increasing it even more to 256 records per batch allowed us to index an > > > entire segment without errors. > > > > > > Another detail is that we do not restart the cluster between indexing > > > attempts so it seems that something only builds up during indexing > > > (nothing seems to leak afterwards) and throws an error. > > > > > > Any hints? > > > > > > Thanks, > > > Markus > > > > > > > > > > > > -----Original message----- > > >> From:Markus Jelsma <markus.jel...@openindex.io> > > >> Sent: Tue 07-Aug-2012 20:08 > > >> To: solr-user@lucene.apache.org > > >> Subject: null:java.lang.RuntimeException: [was class > > >> java.net.SocketTimeoutException] null > > >> > > >> Hello, > > >> > > >> We sometimes see the error below in our `master` when indexing. Our > > >> master is currently the node we send documents to - we've not yet > > >> implemented CloudSolrServer in Apache Nutch. This causes the indexer to > > >> crash when using Nutch locally, the task is retried when running on > > >> Hadoop. We're running it locally in this test set up so there's only one > > >> indexing thread. > > >> > > >> Anyway, for me it's quite a cryptic error because i don't know what > > >> connection has timed out, i assume a connection from the indexing node > > >> to some other node in the cluster when it passes a document to the > > >> correct leader? Each node of the 10 node cluster has the same > > >> configuration, Tomcat is configured with maxThreads=512 and a time out > > >> of one second. > > >> > > >> We're using today's trunk in this test set up and we cannot reliably > > >> reproduce the error. We've seen the error before so it's not a very > > >> recent issue. No errors are found in the other node's logs. > > >> > > >> 2012-08-07 17:52:05,260 ERROR [solr.servlet.SolrDispatchFilter] - > > >> [http-8080-exec-6] - : null:java.lang.RuntimeException: [was class > > >> java.net.SocketTimeoutException] null > > >> at > > >> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) > > >> at > > >> com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) > > >> at > > >> org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:376) > > >> at > > >> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:229) > > >> at > > >> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:157) > > >> at > > >> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) > > >> at > > >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > > >> at > > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > > >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) > > >> at > > >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) > > >> at > > >> io.openindex.solr.servlet.HttpResponseSolrDispatchFilter.doFilter(HttpResponseSolrDispatchFilter.java:219) > > >> at > > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > > >> at > > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > >> at > > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > >> at > > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > >> at > > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > >> at > > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > >> at > > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > >> at > > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > > >> at > > >> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > > >> at > > >> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) > > >> at > > >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) > > >> at > > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > >> at > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > >> at java.lang.Thread.run(Thread.java:662) > > >> Caused by: java.net.SocketTimeoutException > > >> at > > >> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185) > > >> at > > >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:229) > > >> at > > >> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:210) > > >> at > > >> org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:643) > > >> at > > >> org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:945) > > >> at > > >> org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doR > > >> ead(InternalNioInputBuffer.java:969) > > >> at > > >> org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:268) > > >> at > > >> org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:167) > > >> at > > >> org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:916) > > >> at org.apache.coyote.Request.doRead(Request.java:427) > > >> at > > >> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304) > > >> at > > >> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:419) > > >> at > > >> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327) > > >> at > > >> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162) > > >> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) > > >> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) > > >> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101) > > >> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) > > >> at > > >> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) > > >> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) > > >> at > > >> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) > > >> ... 24 more > > >> > > >> Any thoughts to share? Is this a bug? A misconfiguration? > > >> > > >> Thanks, > > >> Markus > > >> > > >
