If a timeout occurs, does the distributed update then go to the next
replica?
On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:
>
> Sorry I should have specified. These timeouts go inside the <solrcloud>
> section and apply for inter-shard update requests only. The socket and
> connection timeout inside the shardHandlerFactory section apply for
> inter-shard search requests.
>
> On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <peterlkee...@gmail.com>
> wrote:
>
> > Btw, are the following timeouts still supported in solr.xml, and do they
> > only apply to distributed search?
> >
> > <shardHandlerFactory name="shardHandlerFactory"
> > class="HttpShardHandlerFactory">
> > <int name="socketTimeout">${socketTimeout:0}</int>
> > <int name="connTimeout">${connTimeout:0}</int>
> > </shardHandlerFactory>
> >
> > Thanks,
> > Peter
> >
> > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <peterlkee...@gmail.com>
> > wrote:
> >
> > > No, I wasn't aware of these. I will give that a try. If I stop the Solr
> > > jetty service manually, things recover fine, but the hang occurs when I
> > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> > > 15-sec timeout from the stopped node, and expires the session, but the
> > Solr
> > > leader never gets notified. This seems like a bug in ZK.
> > >
> > > Thanks,
> > > Peter
> > >
> > >
> > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > shalinman...@gmail.com> wrote:
> > >
> > >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> > >> reasonable values in your solr.xml? These are the timeouts used for
> > >> inter-shard update requests.
> > >>
> > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <peterlkee...@gmail.com
> >
> > >> wrote:
> > >>
> > >> > We are running SolrCloud in AWS and using their auto scaling groups
> to
> > >> spin
> > >> > up new Solr replicas when CPU utilization exceeds a threshold for a
> > >> period
> > >> > of time. All is well until the replicas are terminated when CPU
> > >> utilization
> > >> > falls below another threshold. What happens is that index updates
> sent
> > >> to
> > >> > the Solr leader hang forever in both the Solr leader and the SolrJ
> > >> client
> > >> > app. Searches work fine. Here are 2 thread stack traces from the
> Solr
> > >> > leader and 2 from the client app:
> > >> >
> > >> > 1) Solr-leader thread doing a distributed commit:
> > >> >
> > >> > Thread 23527: (state = IN_NATIVE)
> > >> > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> > - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=61 (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=62, line=300 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=574, line=520 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > >> > (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=101, line=293 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > >> > @bci=127, line=226 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > >> > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> > >> > @bci=112, line=195 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > >> > @bci=174, line=1250 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > >> > @bci=61, line=157 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> > >> > (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > >> (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > >> > org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > >> > @bci=1380, line=417 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > >> line=201
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >> > -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > >> frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > >> frame)
> > >> > -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> (Compiled
> > >> > frame)
> > >> > -
> org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> (Compiled
> > >> > frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > >> frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > >> > @bci=146, line=368 (Compiled frame)
> > >> > - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > >> > @bci=323, line=489 (Compiled frame)
> > >> > - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > >> @bci=1,
> > >> > line=53 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > >> > @bci=13, line=953 (Interpreted frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > >> > @bci=5, line=1014 (Interpreted frame)
> > >> > - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> > >> > (Compiled frame)
> > >> > - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > line=240
> > >> > (Compiled frame)
> > >> > - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > >> > line=72 (Compiled frame)
> > >> > -
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > >> > @bci=100, line=264 (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > >> > @bci=1, line=608 (Interpreted frame)
> > >> > - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > >> line=543
> > >> > (Compiled frame)
> > >> > - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >> >
> > >> > 2) Solr-leader thread doing a delete by query:
> > >> >
> > >> > Thread 19580: (state = BLOCKED)
> > >> > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > >> information
> > >> > may be imprecise)
> > >> > - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > >> @bci=14,
> > >> > line=186 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > >> > @bci=1, line=834 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > >> > int) @bci=67, line=867 (Compiled frame)
> > >> > -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > >> > @bci=17, line=1197 (Compiled frame)
> > >> > - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > @bci=21,
> > >> > line=214 (Compiled frame)
> > >> > - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > >> > @bci=81, line=391 (Compiled frame)
> > >> > - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > >> > @bci=35, line=98 (Compiled frame)
> > >> > - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8,
> line=62
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=1088, line=1088 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=14, line=825 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=61, line=121 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> > >> line=150
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> > >> line=111
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse,
> > >> > org.apache.solr.common.util.ContentStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> > >> line=58
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse,
> > >> > org.apache.solr.common.util.ContentStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> > >> line=92
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> > (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > >> (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > >> > org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > >> > @bci=1380, line=417 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > >> line=201
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >> > -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > >> frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > >> frame)
> > >> > -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> (Compiled
> > >> > frame)
> > >> > -
> org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> (Compiled
> > >> > frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > >> frame)
> > >> > -
> > >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > >> > @bci=146, line=368 (Compiled frame)
> > >> > - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > >> > @bci=323, line=489 (Compiled frame)
> > >> > - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> > >> > (Compiled frame)
> > >> > - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > line=240
> > >> > (Compiled frame)
> > >> > - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > >> > line=72 (Compiled frame)
> > >> > -
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > >> > @bci=100, line=264 (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > >> > @bci=1, line=608 (Interpreted frame)
> > >> > - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > >> line=543
> > >> > (Compiled frame)
> > >> > - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >> >
> > >> > 3) SolrJ client thread doing commit:
> > >> >
> > >> > Thread 1872: (state = IN_NATIVE)
> > >> > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> > - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=5, line=115 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=62 (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=73, line=301 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=597, line=522 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > >> (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > >> > @bci=132, line=285 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=838, line=640 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > >> > @bci=17, line=117 (Interpreted frame)
> > >> > - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> > >> > @bci=16, line=168 (Interpreted frame)
> > >> > - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> > >> > (Interpreted frame)
> > >> >
> > >> > 4) SolrJ client thread doing delete by query:
> > >> >
> > >> > Thread 1873: (state = IN_NATIVE)
> > >> > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> > - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=5, line=115 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=62 (Compiled frame)
> > >> > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> > -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=73, line=301 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > >> frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=597, line=522 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > >> (Compiled
> > >> > frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > >> > @bci=132, line=285 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=838, line=640 (Compiled frame)
> > >> > -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > >> > @bci=17, line=117 (Compiled frame)
> > >> > -
> > >>
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > >> > int) @bci=21, line=285 (Compiled frame)
> > >> > -
> > >>
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > >> > @bci=3, line=271 (Compiled frame)
> > >> >
> > >> > It looks as if the Solr-leader threads are waiting on a response
> from
> > >> the
> > >> > terminated replicas, but I would have expected an IOException to
> > occur.
> > >> Any
> > >> > ideas on what may be happening?
> > >> >
> > >> > Thanks,
> > >> > Peter
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Shalin Shekhar Mangar.
> > >>
> > >
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
