Hi,
On 7.2 is regularly see this popping up:
2018-01-23 16:16:37.056 ERROR (qtp329611835-117592) [c:logs s:shard1
r:core_node1 x:logs_shard1_replica1] o.a.s.s.HttpSolrCall
null:java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 120000/120000 ms
at
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:220)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:583)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55)
at
org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)
at java.io.OutputStream.write(OutputStream.java:116)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:140)
at org.apache.solr.util.FastWriter.write(FastWriter.java:94)
at
org.apache.solr.response.TextResponseWriter.indent(TextResponseWriter.java:108)
at
org.apache.solr.response.TextResponseWriter.indent(TextResponseWriter.java:104)
at
org.apache.solr.response.JSONWriter.writeSolrDocument(JSONResponseWriter.java:364)
at
org.apache.solr.response.TextResponseWriter.writeSolrDocumentList(TextResponseWriter.java:261)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:172)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:788)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
120000/120000 ms
at
org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Just before that a while bunch of these pop up:
2018-01-23 16:11:19.851 WARN
(zkConnectionManagerCallback-72-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs
r:core_node1-SendThread(89.188.14.10:2181)) [c:logs s:shard1 r:core_node1
x:logs_shard1_replica1] o.a.z.ClientCnxn Client session timed out, have not
heard from server in 6851ms for sessionid 0x60f9e7234def33
2018-01-23 16:11:19.851 WARN
(zkConnectionManagerCallback-137-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs
r:core_node1-SendThread(89.188.14.10:2181)) [c:logs s:shard1 r:core_node1
x:logs_shard1_replica1] o.a.z.ClientCnxn Client session timed out, have not
heard from server in 6851ms for sessionid 0x60f9e7234def34
2018-01-23 16:11:19.851 WARN
(zkConnectionManagerCallback-118-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs
r:core_node1-SendThread(89.188.14.10:2181)) [c:logs s:shard1 r:core_node1
x:logs_shard1_replica1] o.a.z.ClientCnxn Client session timed out, have not
heard from server in 6851ms for sessionid 0x60f9e7234def32
2018-01-23 16:11:19.851 WARN
(zkConnectionManagerCallback-85-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs
r:core_node1-SendThread(89.188.14.12:2181)) [c:logs s:shard1 r:core_node1
x:logs_shard1_replica1] o.a.z.ClientCnxn Client session timed out, have not
heard from server in 6851ms for sessionid 0x260f9eafdf5f04d
2018-01-23 16:11:19.851 WARN
(zkConnectionManagerCallback-93-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs
r:core_node1-SendThread(89.188.14.11:2181)) [c:logs s:shard1 r:core_node1
x:logs_shard1_replica1] o.a.z.ClientCnxn Client session timed out, have not
heard from server in 6851ms for sessionid 0x160f9e723bfefe1
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-123-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
Watcher org.apache.solr.common.cloud.ConnectionManager@1679200c name:
ZooKeeperConnection
Watcher:89.188.14.10:2181,89.188.14.11:2181,89.188.14.12:2181/solr_sitesearch_suggest
got event WatchedEvent state:Disconnected type:None path:null path: null type:
None
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-123-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
zkClient has disconnected
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-81-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
Watcher org.apache.solr.common.cloud.ConnectionManager@7d950a72 name:
ZooKeeperConnection
Watcher:89.188.14.10:2181,89.188.14.11:2181,89.188.14.12:2181/solr_sitesearch_search
got event WatchedEvent state:Disconnected type:None path:null path: null type:
None
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-81-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
zkClient has disconnected
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-89-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
Watcher org.apache.solr.common.cloud.ConnectionManager@7b29a94a name:
ZooKeeperConnection
Watcher:89.188.14.10:2181,89.188.14.11:2181,89.188.14.12:2181/solr_sitesearch_search
got event WatchedEvent state:Disconnected type:None path:null path: null type:
None
2018-01-23 16:11:19.948 WARN
(zkConnectionManagerCallback-89-thread-1-processing-n:logs1.gr.nl.openindex.io:8983_solr
x:logs_shard1_replica1 s:shard1 c:logs r:core_node1-EventThread) [c:logs
s:shard1 r:core_node1 x:logs_shard1_replica1] o.a.s.c.c.ConnectionManager
zkClient has disconnected
Are these known error?
Thanks,
Markus
