Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
From the fact that someone has tried to access /etc/passwd file via
your Solr (see all those WARN messages), it seems you have it exposed to
the world, unless of course it's a security scanner you use internally.
Internet is a hostile place, and the very first thing I would do is
shield Solr from external traffic. Even if it's your own security
scanning, I wouldn't do it until you have the system stable.
Doing the above you'll reduce noise in the logs and might be able to
better identify the issue.
Losing the Zookeeper connection is typically a Java garbage collection
issue. If GC causes too long pauses, the connection may time out. So I
would recommend you start by reading
https://wiki.apache.org/solr/SolrPerformanceProblems and
https://wiki.apache.org/solr/ShawnHeisey. Also make sure that
Zookeeper's Java settings are good.
--Ere
Bhalla, Rahat kirjoitti 5.7.2017 klo 11.05:
Hi
I’m not sure if any of you have had a chance to see this email yet.
We had a reoccurrence of the Issue Today, and I’m attaching the Logs
from today as well inline below.
Please let me know if any of you have seen this issue before as this
would really help me to get to the root of the problem to fix it. I’m a
little lost here and not entirely sure what to do.
Thanks,
Rahat Bhalla
8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20
(HttpParser.java:parseNext:1391) - parse exception:
java.lang.IllegalArgumentException: No Authority for
HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
at
org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
at
org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
at
org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Unknown Source)
8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
HttpChannelOverHttp@1276{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}
8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for
HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}
8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
byte C0 in state 0
8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
byte 80 in state 4
8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
byte 80 in state 6
8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Hi
I'm not sure if any of you have had a chance to see this email yet.
We had a reoccurrence of the Issue Today, and I'm attaching the Logs from today
as well inline below.
Please let me know if any of you have seen this issue before as this would
really help me to get to the root of the problem to fix it. I'm a little lost
here and not entirely sure what to do.
Thanks,
Rahat Bhalla
8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20
(HttpParser.java:parseNext:1391) - parse exception:
java.lang.IllegalArgumentException: No Authority for
HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
at
org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Unknown Source)
8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
HttpChannelOverHttp@1276{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}
8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for
HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}
8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte
C0 in state 0
8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte
80 in state 4
8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte
80 in state 6
8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22
(URIUtil.java:decodePath:348) -
/solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte
F8 in state 0
8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for
HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for
HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44
(HttpParser.java::1740) - Illegal character 0xA in
Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Hi Solr Users, I hope this email finds you all in the best of spirits and in a mood where you'd be willing to help a young developer (me :) ) with issues that I'm facing in regards with the Solr Cloud. At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances with 13 collections spread across the 5 nodes and an ensemble of 3 zookeeper instances spread across three different nodes. Over the last one week, our leader node seems to be going down every other day and while we restart the solr instances they still go down within the next 24 Hours or more. We have tried rebooting the nodes that host the solr instances and that hasn't helped. We plan to clear out the zookeeper logs and data folders before the restart of the zookeeper instances. As of now, I'm the only one supporting Solr in my organization and any insight from you could help me a great deal to fix the issue. I'm copying the Exception stack trace from this morning. Any recommendations that you might have will be great appreciated. Below is a snapshot of one of the zoo nodes: [cid:image001.png@01D2F43C.30CA38E0] Exception Stacktrace 138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_000140] [ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345) at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191) at java.lang.Thread.run(Unknown Source) 138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 (SolrException.java:log:148) - org.apache.solr.common.SolrException: Could not load collection from ZK: feedsOutBoundToExchange at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047) at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610) at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248) at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674) at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
