[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[ https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434430#comment-15434430 ] Forest Soup commented on SOLR-5724: --- I found the similar issue in Solr v5.3.2 - We have a solrcloud with 3 solr nodes, 80 collections are created on them with replicateFactor=1, and shardNum=1 for each collection. After the collections creation, all cores are active, we start first batch of index with SolrJ client. But we found issues on all collections of one of the 3 solr nodes, and index failure due to HTTP 503: 2016-08-16 20:02:05.660 ERROR (qtp208437930-70) [c:collection4 s:shard1 r:core_node1 x:collection4_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor ClusterState says we are the leader, but locally we don't think so 2016-08-16 20:02:05.667 ERROR (qtp208437930-70) [c:collection4 s:shard1 r:core_node1 x:collection4_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.common.SolrException: ClusterState says we are the leader (https://host1.domain1:8983/solr/collection4_shard1_replica1), but locally we don't think so. Request came from null at org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:619) at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:381) at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:314) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:665) at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:143) at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:113) at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:76) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:672) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:463) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:235) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:199) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) The collections on the other 2 solr nodes works fine and index succeeded. > Two node, one shard solr instance intermittently going offline > --- > > Key: SOLR-5724 > URL: https://issues.apache.org/jira/browse/SOLR-5724 > Project: Solr > Issue Type: Bug >Affects Versions: 4.6.1 > Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45" > Java(TM) SE Runtime Environment (build 1.6.0_45-b06) > Java HotSpot(TM) 64-Bit
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274153#comment-15274153
]
Shawn Heisey commented on SOLR-5724:
With the recent release of 6.0.0, 4.x is effectively dead.
Only *major* security issues and showstopper problems without a workaround will
receive any attention ... and even those might not actually be fixed in 4.x.
Only 5.x (on a limited basis) and 6.x are likely to get changes now. If any
fixes *do* happen in 4.x, they will only happen in 4.10.x, not 4.7.x.
> Two node, one shard solr instance intermittently going offline
> ---
>
> Key: SOLR-5724
> URL: https://issues.apache.org/jira/browse/SOLR-5724
> Project: Solr
> Issue Type: Bug
>Affects Versions: 4.6.1
> Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
>Reporter: Joseph Duchesne
>
> One server is stuck in state "recovering" while the other is stuck in state
> "down". After waiting 45 minutes or so for the cluster to recover, the
> statuses were the same.
> Log messages on the "recovering" server: (Just the individual errors for
> brevity, I can provide full stack traces if that is helpful)
> {quote}
> We are not the leader
> ClusterState says we are the leader, but locally we don't think so
> cancelElection did not find election node to remove
> We are not the leader
> No registered leader was found, collection:listsC slice:shard1
> No registered leader was found, collection:listsC slice:shard1
> {quote}
> On the "down" server at the same timeframe:
> {quote}
> org.apache.solr.common.SolrException; forwarding update to
> http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
> org.apache.solr.update.StreamingSolrServers$1; error
> Error while trying to recover.
> core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> We are not the leader
> Recovery failed - trying again... (0) core=listsC
> Stopping recovery for zkNodeName=core_node2core=listsC
> org.apache.solr.update.StreamingSolrServers$1; error
> org.apache.solr.common.SolrException: Service Unavailable
> {quote}
> I am not sure what is causing this, however it has happened a 3 times in the
> past week. If there are any additional logs I can provide, or if there is
> anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273752#comment-15273752
]
haedrig commented on SOLR-5724:
---
Hello, I have encountered the same problem, I would like to ask you to solve
it? Solr 4.7.2
> Two node, one shard solr instance intermittently going offline
> ---
>
> Key: SOLR-5724
> URL: https://issues.apache.org/jira/browse/SOLR-5724
> Project: Solr
> Issue Type: Bug
>Affects Versions: 4.6.1
> Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
>Reporter: Joseph Duchesne
>
> One server is stuck in state "recovering" while the other is stuck in state
> "down". After waiting 45 minutes or so for the cluster to recover, the
> statuses were the same.
> Log messages on the "recovering" server: (Just the individual errors for
> brevity, I can provide full stack traces if that is helpful)
> {quote}
> We are not the leader
> ClusterState says we are the leader, but locally we don't think so
> cancelElection did not find election node to remove
> We are not the leader
> No registered leader was found, collection:listsC slice:shard1
> No registered leader was found, collection:listsC slice:shard1
> {quote}
> On the "down" server at the same timeframe:
> {quote}
> org.apache.solr.common.SolrException; forwarding update to
> http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
> org.apache.solr.update.StreamingSolrServers$1; error
> Error while trying to recover.
> core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> We are not the leader
> Recovery failed - trying again... (0) core=listsC
> Stopping recovery for zkNodeName=core_node2core=listsC
> org.apache.solr.update.StreamingSolrServers$1; error
> org.apache.solr.common.SolrException: Service Unavailable
> {quote}
> I am not sure what is causing this, however it has happened a 3 times in the
> past week. If there are any additional logs I can provide, or if there is
> anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13901564#comment-13901564
]
Mark Miller commented on SOLR-5724:
---
Anyway you could share them with me? Via Gmail or something?
Two node, one shard solr instance intermittently going offline
---
Key: SOLR-5724
URL: https://issues.apache.org/jira/browse/SOLR-5724
Project: Solr
Issue Type: Bug
Affects Versions: 4.6.1
Environment: Ubuntu 12.04.3 LTS, 64 bit, java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Reporter: Joseph Duchesne
One server is stuck in state recovering while the other is stuck in state
down. After waiting 45 minutes or so for the cluster to recover, the
statuses were the same.
Log messages on the recovering server: (Just the individual errors for
brevity, I can provide full stack traces if that is helpful)
{quote}
We are not the leader
ClusterState says we are the leader, but locally we don't think so
cancelElection did not find election node to remove
We are not the leader
No registered leader was found, collection:listsC slice:shard1
No registered leader was found, collection:listsC slice:shard1
{quote}
On the down server at the same timeframe:
{quote}
org.apache.solr.common.SolrException; forwarding update to
http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
org.apache.solr.update.StreamingSolrServers$1; error
Error while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
Recovery failed - trying again... (0) core=listsC
Stopping recovery for zkNodeName=core_node2core=listsC
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I am not sure what is causing this, however it has happened a 3 times in the
past week. If there are any additional logs I can provide, or if there is
anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13901574#comment-13901574
]
Joseph Duchesne commented on SOLR-5724:
---
I just emailed you the log files (via your posted gmail).
Two node, one shard solr instance intermittently going offline
---
Key: SOLR-5724
URL: https://issues.apache.org/jira/browse/SOLR-5724
Project: Solr
Issue Type: Bug
Affects Versions: 4.6.1
Environment: Ubuntu 12.04.3 LTS, 64 bit, java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Reporter: Joseph Duchesne
One server is stuck in state recovering while the other is stuck in state
down. After waiting 45 minutes or so for the cluster to recover, the
statuses were the same.
Log messages on the recovering server: (Just the individual errors for
brevity, I can provide full stack traces if that is helpful)
{quote}
We are not the leader
ClusterState says we are the leader, but locally we don't think so
cancelElection did not find election node to remove
We are not the leader
No registered leader was found, collection:listsC slice:shard1
No registered leader was found, collection:listsC slice:shard1
{quote}
On the down server at the same timeframe:
{quote}
org.apache.solr.common.SolrException; forwarding update to
http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
org.apache.solr.update.StreamingSolrServers$1; error
Error while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
Recovery failed - trying again... (0) core=listsC
Stopping recovery for zkNodeName=core_node2core=listsC
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I am not sure what is causing this, however it has happened a 3 times in the
past week. If there are any additional logs I can provide, or if there is
anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900513#comment-13900513
]
Yago Riveiro commented on SOLR-5724:
I have this issue too, the only way that I found to recover from this was
restart the nodes.
Two node, one shard solr instance intermittently going offline
---
Key: SOLR-5724
URL: https://issues.apache.org/jira/browse/SOLR-5724
Project: Solr
Issue Type: Bug
Affects Versions: 4.6.1
Environment: Ubuntu 12.04.3 LTS, 64 bit, java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Reporter: Joseph Duchesne
One server is stuck in state recovering while the other is stuck in state
down. After waiting 45 minutes or so for the cluster to recover, the
statuses were the same.
Log messages on the recovering server: (Just the individual errors for
brevity, I can provide full stack traces if that is helpful)
{quote}
We are not the leader
ClusterState says we are the leader, but locally we don't think so
cancelElection did not find election node to remove
We are not the leader
No registered leader was found, collection:listsC slice:shard1
No registered leader was found, collection:listsC slice:shard1
{quote}
On the down server at the same timeframe:
{quote}
org.apache.solr.common.SolrException; forwarding update to
http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
org.apache.solr.update.StreamingSolrServers$1; error
Error while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
Recovery failed - trying again... (0) core=listsC
Stopping recovery for zkNodeName=core_node2core=listsC
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I am not sure what is causing this, however it has happened a 3 times in the
past week. If there are any additional logs I can provide, or if there is
anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5724) Two node, one shard solr instance intermittently going offline
[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900598#comment-13900598
]
Mark Miller commented on SOLR-5724:
---
I think more detailed log info would def help.
Two node, one shard solr instance intermittently going offline
---
Key: SOLR-5724
URL: https://issues.apache.org/jira/browse/SOLR-5724
Project: Solr
Issue Type: Bug
Affects Versions: 4.6.1
Environment: Ubuntu 12.04.3 LTS, 64 bit, java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Reporter: Joseph Duchesne
One server is stuck in state recovering while the other is stuck in state
down. After waiting 45 minutes or so for the cluster to recover, the
statuses were the same.
Log messages on the recovering server: (Just the individual errors for
brevity, I can provide full stack traces if that is helpful)
{quote}
We are not the leader
ClusterState says we are the leader, but locally we don't think so
cancelElection did not find election node to remove
We are not the leader
No registered leader was found, collection:listsC slice:shard1
No registered leader was found, collection:listsC slice:shard1
{quote}
On the down server at the same timeframe:
{quote}
org.apache.solr.common.SolrException; forwarding update to
http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
org.apache.solr.update.StreamingSolrServers$1; error
Error while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
Recovery failed - trying again... (0) core=listsC
Stopping recovery for zkNodeName=core_node2core=listsC
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I am not sure what is causing this, however it has happened a 3 times in the
past week. If there are any additional logs I can provide, or if there is
anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
