[ https://issues.apache.org/jira/browse/SOLR-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kesharee Nandan Vishwakarma updated SOLR-13793: ----------------------------------------------- Affects Version/s: master (9.0) > HTTPSolrCall makes cascading calls even when all replicas are down for a > collection > ----------------------------------------------------------------------------------- > > Key: SOLR-13793 > URL: https://issues.apache.org/jira/browse/SOLR-13793 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud > Affects Versions: 6.6, master (9.0) > Reporter: Kesharee Nandan Vishwakarma > Priority: Major > > REMOTEQUERY action in HTTPSolrCall ends up making too many cascading > remoteQuery calls when all all the replicas of a collection are in down > state. > This results in increase in thread count, unresponsive solr nodes and > eventually node (one's which have this collection) going out of live nodes. > *Example scenario*: Consider a cluster with 3 nodes(solr1, solrw1, > solr-overseer1). A collection is present on solr1, solrw1 but both replicas > are in down state. When a search request is made to solr-overseer1, since > replica is not present locally a remote query is made to solr1 (we also > consider inactive slices/coreUrls), solr1 also doesn't see an active replica > present locally, it forwards to solrw1, again solrw1 will forward request to > solr1. This goes on till both of solr1, solrw1 become unresponsive. Attached > logs for this. > This is happening because we are considering [inactive > slices|https://github.com/apache/lucene-solr/blob/68fa249034ba8b273955f20097700dc2fbb7a800/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L913 > ], [inactive coreUrl| > https://github.com/apache/lucene-solr/blob/68fa249034ba8b273955f20097700dc2fbb7a800/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L929] > while forwarding requests to nodes. > *Steps to reproduce*: > # Bring down all replicas of a collection but ensure nodes containing them > are up > # Make any search call to any of solr nodes for this collection. > > *Possible fixes*: > # Ensure we select only active slices/coreUrls before making remote queries > # Put a limit on cascading calls probably limit to number of replicas > > {noformat} > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:14.458 ERROR (qtp762152757-8772) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:14.451 ERROR (qtp762152757-8774) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:14.333 ERROR (qtp762152757-8771) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.661 INFO > (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [ ] > o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) > solrw1_1 | 2019-09-24 09:35:12.284 ERROR (qtp762152757-8767) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.275 ERROR (qtp762152757-8768) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.274 ERROR (qtp762152757-8766) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.054 ERROR (qtp762152757-8764) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.049 ERROR (qtp762152757-8763) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:12.044 ERROR (qtp762152757-8762) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:11.826 ERROR (qtp762152757-556) [ ] > o.a.s.s.HttpSolrCall > null:org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /roles.json > solrw1_1 | at > org.apache.zookeeper.KeeperException.create(KeeperException.java:127) > solrw1_1 | at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > solrw1_1 | at > org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) > solrw1_1 | at > org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) > solrw1_1 | at > org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) > solrw1_1 | at > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) > solrw1_1 | at > org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) > solrw1_1 | at > org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:82) > solrw1_1 | at > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$20(CollectionsHandler.java:664) > solrw1_1 | at > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:922) > solrw1_1 | at > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:228) > solrw1_1 | at > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215) > solrw1_1 | at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:749) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:730) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | > zook1_1 | 2019-09-24 09:35:30,491 [myid:] - INFO > [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level > KeeperException when processing sessionid:0x16d6294ad750020 type:delete > cxid:0xd zxid:0x9a5 txntype:-1 reqpath:n/a Error > Path:/solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036 > Error:KeeperErrorCode = NoNode for > /solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036 > zook1_1 | 2019-09-24 09:35:30,527 [myid:] - INFO > [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level > KeeperException when processing sessionid:0x16d6294ad750020 type:create > cxid:0x12 zxid:0x9a7 txntype:-1 reqpath:n/a Error Path:/solr/overseer > Error:KeeperErrorCode = NodeExists for /solr/overseer > solrw1_1 | 2019-09-24 09:35:30.553 INFO > (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [ ] > o.a.s.c.ZkController Register node as live in > ZooKeeper:/live_nodes/solrw1:8983_solr > solr-overseer_1 | 2019-09-24 09:35:30.574 INFO > (zkCallback-5-thread-8-processing-n:solr-overseer:8983_solr) [ ] > o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) > solrw1_1 | 2019-09-24 09:35:30.589 INFO > (zkCallback-5-thread-6-processing-n:solrw1:8983_solr) [ ] > o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) > solrw1_1 | 2019-09-24 09:35:30.726 INFO > (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [ ] > o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper > solrw1_1 | 2019-09-24 09:35:30.727 INFO > (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [ ] > o.a.s.c.c.ConnectionManager Connected:true > solrw1_1 | 2019-09-24 09:35:32.569 ERROR (qtp762152757-8797) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg3/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:35:32.945 INFO (qtp762152757-6501) [ ] > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with > params action=clusterstatus&wt=json and sendToOCPQueue=true > solrw1_1 | 2019-09-24 09:35:32.986 INFO (qtp762152757-6501) [ ] > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections > params={action=clusterstatus&wt=json} status=0 QTime=40 > solrw1_1 | 2019-09-24 09:35:33.929 INFO (qtp762152757-3634) [ ] > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with > params action=clusterstatus&wt=json and sendToOCPQueue=true > solrw1_1 | 2019-09-24 09:35:33.947 INFO (qtp762152757-3634) [ ] > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections > params={action=clusterstatus&wt=json} status=0 QTime=18 > solr1_1 | Exception in thread "solr-idle-connections-evictor" > solr1_1 | Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "qtp762152757-9435" > solrw1_1 | 2019-09-24 09:36:09.080 ERROR (qtp762152757-8806) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr-overseer:8983/solr/kg1/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solrw1_1 | 2019-09-24 09:37:04.028 ERROR (qtp762152757-8805) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg2/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 | > solr-overseer_1 | 2019-09-24 09:37:10.610 ERROR (qtp762152757-681) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solrw1:8983/solr/kg2/select > solr-overseer_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solr-overseer_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solr-overseer_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solr-overseer_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solr-overseer_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solr-overseer_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solr-overseer_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solr-overseer_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solr-overseer_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solr-overseer_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solr-overseer_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solr-overseer_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solr-overseer_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solr-overseer_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solr-overseer_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solr-overseer_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solr-overseer_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solr-overseer_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solr-overseer_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solr-overseer_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solr-overseer_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solr-overseer_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solr-overseer_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solr-overseer_1 | at java.lang.Thread.run(Thread.java:748) > solr-overseer_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solr-overseer_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solr-overseer_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solr-overseer_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solr-overseer_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solr-overseer_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solr-overseer_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solr-overseer_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solr-overseer_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solr-overseer_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solr-overseer_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solr-overseer_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solr-overseer_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solr-overseer_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solr-overseer_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solr-overseer_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solr-overseer_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solr-overseer_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solr-overseer_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solr-overseer_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solr-overseer_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solr-overseer_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solr-overseer_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solr-overseer_1 | ... 30 more > solr-overseer_1 | > solrw1_1 | 2019-09-24 09:37:10.631 ERROR (qtp762152757-8818) [ ] > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying > to proxy request for url: http://solr1:8983/solr/kg2/select > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > solrw1_1 | at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > solrw1_1 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > solrw1_1 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > solrw1_1 | at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > solrw1_1 | at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > solrw1_1 | at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > solrw1_1 | at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > solrw1_1 | at > org.eclipse.jetty.server.Server.handle(Server.java:534) > solrw1_1 | at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > solrw1_1 | at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > solrw1_1 | at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > solrw1_1 | at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > solrw1_1 | at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > solrw1_1 | at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > solrw1_1 | at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > solrw1_1 | at java.lang.Thread.run(Thread.java:748) > solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out > solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) > solrw1_1 | at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:171) > solrw1_1 | at > java.net.SocketInputStream.read(SocketInputStream.java:141) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > solrw1_1 | at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > solrw1_1 | at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > solrw1_1 | at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > solrw1_1 | at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > solrw1_1 | at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > solrw1_1 | at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > solrw1_1 | at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > solrw1_1 | at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > solrw1_1 | at > org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > solrw1_1 | at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > solrw1_1 | at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > solrw1_1 | at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > solrw1_1 | at > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) > solrw1_1 | ... 30 more > solrw1_1 > {noformat} > Logs after adding warning messages [here | > https://github.com/apache/lucene-solr/blob/68fa249034ba8b273955f20097700dc2fbb7a800/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L929] > : > {noformat} > solr1_1 | 2019-09-25 19:39:19.597 WARN (qtp762152757-4920) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.597 WARN (qtp762152757-4920) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.599 WARN (qtp762152757-4915) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.599 WARN (qtp762152757-4915) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.604 WARN (qtp762152757-4921) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.604 WARN (qtp762152757-4921) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.608 WARN (qtp762152757-4916) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.608 WARN (qtp762152757-4916) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.611 WARN (qtp762152757-4922) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.611 WARN (qtp762152757-4922) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.615 WARN (qtp762152757-4917) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.615 WARN (qtp762152757-4917) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.618 WARN (qtp762152757-4923) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.618 WARN (qtp762152757-4923) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.622 WARN (qtp762152757-4919) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.622 WARN (qtp762152757-4919) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.624 WARN (qtp762152757-4924) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.625 WARN (qtp762152757-4924) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.627 WARN (qtp762152757-4918) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.627 WARN (qtp762152757-4918) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.629 WARN (qtp762152757-4925) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.629 WARN (qtp762152757-4925) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.631 WARN (qtp762152757-4920) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.631 WARN (qtp762152757-4920) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.636 WARN (qtp762152757-4926) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.636 WARN (qtp762152757-4926) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.638 WARN (qtp762152757-4921) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.639 WARN (qtp762152757-4921) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > solr1_1 | 2019-09-25 19:39:19.641 WARN (qtp762152757-4927) [ ] > o.a.s.s.HttpSolrCall active core: null > solr1_1 | 2019-09-25 19:39:19.641 WARN (qtp762152757-4927) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solrw1:8983/solr/kg2 > solrw1_1 | 2019-09-25 19:39:19.645 WARN (qtp762152757-4922) [ ] > o.a.s.s.HttpSolrCall active core: null > solrw1_1 | 2019-09-25 19:39:19.645 WARN (qtp762152757-4922) [ ] > o.a.s.s.HttpSolrCall Will attempt to connect inactive core: > http://solr1:8983/solr/kg2 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org