Hi Solr Users,
I hope this email finds you all in the best of spirits and in a mood where
you'd be willing to help a young developer (me :) ) with issues that I'm facing
in regards with the Solr Cloud.
At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances
with 13 collections spread across the 5 nodes and an ensemble of 3 zookeeper
instances spread across three different nodes.
Over the last one week, our leader node seems to be going down every other day
and while we restart the solr instances they still go down within the next 24
Hours or more.
We have tried rebooting the nodes that host the solr instances and that hasn't
helped. We plan to clear out the zookeeper logs and data folders before the
restart of the zookeeper instances.
As of now, I'm the only one supporting Solr in my organization and any insight
from you could help me a great deal to fix the issue. I'm copying the Exception
stack trace from this morning. Any recommendations that you might have will be
great appreciated.
Below is a snapshot of one of the zoo nodes:
[cid:image001.png@01D2F43C.30CA38E0]
Exception Stacktrace
138127149
[OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140]
[ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /overseer_elect/leader
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
at
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
at
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
at java.lang.Thread.run(Unknown Source)
138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01
(SolrException.java:log:148) - org.apache.solr.common.SolrException: Could not
load collection from ZK: feedsOutBoundToExchange
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
at
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
at
org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
at
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/collections/feedsOutBoundToExchange/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
at
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
... 33 more
Thanks,
Rahat Bhalla
HealthPlan Services
Phone: (813) 289-1000 EXT: 7002249
rbha...@healthplan.com<mailto:rbha...@healthplan.com>
www.healthplan.com<http://www.healthplan.com/>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
CONFIDENTIALITY NOTICE: This email message, including any attachments, is for
the sole use of the intended recipient(s) and may contain confidential and
privileged information and/or Protected Health Information (PHI) subject to
protection under the law, including the Health Insurance Portability and
Accountability Act of 1996, as amended (HIPAA). If you are not the intended
recipient or the person responsible for delivering the email to the intended
recipient, be advised that you have received this email in error and that any
use, disclosure, distribution, forwarding, printing, or copying of this email
is strictly prohibited. If you have received this email in error, please notify
the sender immediately and destroy all copies of the original message.
