We are running SolrCloud in AWS and using their auto scaling groups to spin up new Solr replicas when CPU utilization exceeds a threshold for a period of time. All is well until the replicas are terminated when CPU utilization falls below another threshold. What happens is that index updates sent to the Solr leader hang forever in both the Solr leader and the SolrJ client app. Searches work fine. Here are 2 thread stack traces from the Solr leader and 2 from the client app:
1) Solr-leader thread doing a distributed commit: Thread 23527: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71, line=166 (Compiled frame) - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) @bci=137, line=281 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=16, line=92 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=2, line=61 (Compiled frame) - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254 (Compiled frame) - org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() @bci=8, line=289 (Compiled frame) - org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() @bci=1, line=252 (Compiled frame) - org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() @bci=6, line=191 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=62, line=300 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=60, line=127 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=574, line=520 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=344, line=906 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) @bci=6, line=784 (Compiled frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395 (Interpreted frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=17, line=199 (Interpreted frame) - org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=101, line=293 (Compiled frame) - org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req) @bci=127, line=226 (Interpreted frame) - org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand, java.util.List, org.apache.solr.common.params.ModifiableSolrParams) @bci=112, line=195 (Interpreted frame) - org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand) @bci=174, line=1250 (Interpreted frame) - org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand) @bci=61, line=157 (Interpreted frame) - org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest, org.apache.solr.update.processor.UpdateRequestProcessor, org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69 (Interpreted frame) - org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled frame) - org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled frame) - org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler, org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest, org.apache.solr.request.SolrRequestHandler, org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean) @bci=1380, line=417 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame) - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame) - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame) - org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame) - org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled frame) - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame) - org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame) - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame) - org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) @bci=146, line=368 (Compiled frame) - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() @bci=323, line=489 (Compiled frame) - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() @bci=1, line=53 (Interpreted frame) - org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer) @bci=13, line=953 (Interpreted frame) - org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer) @bci=5, line=1014 (Interpreted frame) - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921 (Compiled frame) - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240 (Compiled frame) - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51, line=72 (Compiled frame) - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run() @bci=100, line=264 (Compiled frame) - org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable) @bci=1, line=608 (Interpreted frame) - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543 (Compiled frame) - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame) 2) Solr-leader thread doing a delete by query: Thread 19580: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=867 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1197 (Compiled frame) - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=214 (Compiled frame) - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290 (Compiled frame) - org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished() @bci=81, line=391 (Compiled frame) - org.apache.solr.update.StreamingSolrServers.blockUntilFinished() @bci=35, line=98 (Compiled frame) - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62 (Compiled frame) - org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand) @bci=1088, line=1088 (Compiled frame) - org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand) @bci=14, line=825 (Compiled frame) - org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand) @bci=61, line=121 (Compiled frame) - org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest, org.apache.solr.client.solrj.request.UpdateRequest, org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208, line=150 (Compiled frame) - org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse, java.io.InputStream, org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68, line=111 (Compiled frame) - org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse, org.apache.solr.common.util.ContentStream, org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18, line=58 (Compiled frame) - org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse, org.apache.solr.common.util.ContentStream, org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165, line=92 (Compiled frame) - org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled frame) - org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled frame) - org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler, org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest, org.apache.solr.request.SolrRequestHandler, org.apache.solr.request.SolrQueryRequest, org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean) @bci=1380, line=417 (Compiled frame) - org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame) - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame) - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame) - org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled frame) - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame) - org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled frame) - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame) - org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame) - org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame) - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame) - org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) @bci=146, line=368 (Compiled frame) - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() @bci=323, line=489 (Compiled frame) - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866 (Compiled frame) - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240 (Compiled frame) - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51, line=72 (Compiled frame) - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run() @bci=100, line=264 (Compiled frame) - org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable) @bci=1, line=608 (Interpreted frame) - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543 (Compiled frame) - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame) 3) SolrJ client thread doing commit: Thread 1872: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71, line=166 (Compiled frame) - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) @bci=137, line=281 (Compiled frame) - org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) @bci=5, line=115 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=16, line=92 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=2, line=62 (Compiled frame) - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254 (Compiled frame) - org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() @bci=8, line=289 (Compiled frame) - org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() @bci=1, line=252 (Compiled frame) - org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() @bci=6, line=191 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=73, line=301 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=60, line=127 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=597, line=522 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=344, line=906 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) @bci=6, line=784 (Compiled frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=17, line=199 (Compiled frame) - org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req) @bci=132, line=285 (Compiled frame) - org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=838, line=640 (Compiled frame) - org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer) @bci=17, line=117 (Interpreted frame) - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean) @bci=16, line=168 (Interpreted frame) - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146 (Interpreted frame) 4) SolrJ client thread doing delete by query: Thread 1873: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71, line=166 (Compiled frame) - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90 (Compiled frame) - org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) @bci=137, line=281 (Compiled frame) - org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) @bci=5, line=115 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=16, line=92 (Compiled frame) - org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=2, line=62 (Compiled frame) - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254 (Compiled frame) - org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() @bci=8, line=289 (Compiled frame) - org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() @bci=1, line=252 (Compiled frame) - org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() @bci=6, line=191 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=73, line=301 (Compiled frame) - org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=60, line=127 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame) - org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=597, line=522 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=344, line=906 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame) - org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) @bci=6, line=784 (Compiled frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled frame) - org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=17, line=199 (Compiled frame) - org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req) @bci=132, line=285 (Compiled frame) - org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest) @bci=838, line=640 (Compiled frame) - org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer) @bci=17, line=117 (Compiled frame) - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String, int) @bci=21, line=285 (Compiled frame) - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String) @bci=3, line=271 (Compiled frame) It looks as if the Solr-leader threads are waiting on a response from the terminated replicas, but I would have expected an IOException to occur. Any ideas on what may be happening? Thanks, Peter
