[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434430#comment-15434430
]
Forest Soup commented on SOLR-5724:
-----------------------------------
I found the similar issue in Solr v5.3.2 -
We have a solrcloud with 3 solr nodes, 80 collections are created on them with
replicateFactor=1, and shardNum=1 for each collection.
After the collections creation, all cores are active, we start first batch of
index with SolrJ client. But we found issues on all collections of one of the 3
solr nodes, and index failure due to HTTP 503:
2016-08-16 20:02:05.660 ERROR (qtp208437930-70) [c:collection4 s:shard1
r:core_node1 x:collection4_shard1_replica1]
o.a.s.u.p.DistributedUpdateProcessor ClusterState says we are the leader, but
locally we don't think so
2016-08-16 20:02:05.667 ERROR (qtp208437930-70) [c:collection4 s:shard1
r:core_node1 x:collection4_shard1_replica1] o.a.s.c.SolrCore
org.apache.solr.common.SolrException: ClusterState says we are the leader
(https://host1.domain1:8983/solr/collection4_shard1_replica1), but locally we
don't think so. Request came from null
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:619)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:381)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:314)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:665)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:143)
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:113)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:76)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:672)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:463)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:235)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:199)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
The collections on the other 2 solr nodes works fine and index succeeded.
> Two node, one shard solr instance intermittently going offline
> ---------------------------------------------------------------
>
> Key: SOLR-5724
> URL: https://issues.apache.org/jira/browse/SOLR-5724
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.6.1
> Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> Reporter: Joseph Duchesne
>
> One server is stuck in state "recovering" while the other is stuck in state
> "down". After waiting 45 minutes or so for the cluster to recover, the
> statuses were the same.
> Log messages on the "recovering" server: (Just the individual errors for
> brevity, I can provide full stack traces if that is helpful)
> {quote}
> We are not the leader
> ClusterState says we are the leader, but locally we don't think so
> cancelElection did not find election node to remove
> We are not the leader
> No registered leader was found, collection:listsC slice:shard1
> No registered leader was found, collection:listsC slice:shard1
> {quote}
> On the "down" server at the same timeframe:
> {quote}
> org.apache.solr.common.SolrException; forwarding update to
> http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
> org.apache.solr.update.StreamingSolrServers$1; error
> Error while trying to recover.
> core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> We are not the leader
> Recovery failed - trying again... (0) core=listsC
> Stopping recovery for zkNodeName=core_node2core=listsC
> org.apache.solr.update.StreamingSolrServers$1; error
> org.apache.solr.common.SolrException: Service Unavailable
> {quote}
> I am not sure what is causing this, however it has happened a 3 times in the
> past week. If there are any additional logs I can provide, or if there is
> anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
