solr 5.4.1 leader issue
Hi got two out of my three servers think they are replicas on one shard getting exceptions wondering what is the easiest way to fix this? Can I just restart zookeeper across the servers? Here are the exceptions: TY Robi ERROR null RecoveryStrategy Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.209.55.10:8983/solr: We are not the leader at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281) ... 5 more (and on the one everyone thinks is the leader) Error while trying to recover. core=custsearch_shard3_replica3:org.apache.solr.common.SolrException: Cloud state still says we are leader. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:332) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) This communication is confidential. Frontier only sends and receives email on the basis of the terms set out at http://www.frontier.com/email_disclaimer.
Re: solr 5.4.1 leader issue
I'm on zookeeper 3.4.8 From: Petersen, Robert (Contr) Sent: Monday, January 8, 2018 12:34:28 PM To: solr-user@lucene.apache.org Subject: solr 5.4.1 leader issue Hi got two out of my three servers think they are replicas on one shard getting exceptions wondering what is the easiest way to fix this? Can I just restart zookeeper across the servers? Here are the exceptions: TY Robi ERROR null RecoveryStrategy Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.209.55.10:8983/solr: We are not the leader at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281) ... 5 more (and on the one everyone thinks is the leader) Error while trying to recover. core=custsearch_shard3_replica3:org.apache.solr.common.SolrException: Cloud state still says we are leader. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:332) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) This communication is confidential. Frontier only sends and receives email on the basis of the terms set out at http://www.frontier.com/email_disclaimer.
Re: solr 5.4.1 leader issue
Perhaps I didn't explain well, three nodes live. Two are in recovering mode exception being they cant get to the Leader because the Leader replies that he is not the leader. On the dashboard it shows him as the leader but he thinks he isn't. The exceptions are below... Do I have to just restart the solr instances, the zookeeper instances, both, or is there another better way without restarting everything? Thx Robi From: Petersen, Robert (Contr) Sent: Monday, January 8, 2018 12:34:28 PM To: solr-user@lucene.apache.org Subject: solr 5.4.1 leader issue Hi got two out of my three servers think they are replicas on one shard getting exceptions wondering what is the easiest way to fix this? Can I just restart zookeeper across the servers? Here are the exceptions: TY Robi ERROR null RecoveryStrategy Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.209.55.10:8983/solr: We are not the leader at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281) ... 5 more (and on the one everyone thinks is the leader) Error while trying to recover. core=custsearch_shard3_replica3:org.apache.solr.common.SolrException: Cloud state still says we are leader. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:332) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) This communication is confidential. Frontier only sends and receives email on the basis of the terms set out at http://www.frontier.com/email_disclaimer.
Re: solr 5.4.1 leader issue
OK just restarting all the solr nodes did fix it, since they are in production I was hesitant to do that From: Petersen, Robert (Contr) Sent: Monday, January 8, 2018 12:34:28 PM To: solr-user@lucene.apache.org Subject: solr 5.4.1 leader issue Hi got two out of my three servers think they are replicas on one shard getting exceptions wondering what is the easiest way to fix this? Can I just restart zookeeper across the servers? Here are the exceptions: TY Robi ERROR null RecoveryStrategy Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader Error while trying to recover. core=custsearch_shard3_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://x.x.x.x:8983/solr: We are not the leader at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.209.55.10:8983/solr: We are not the leader at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281) ... 5 more (and on the one everyone thinks is the leader) Error while trying to recover. core=custsearch_shard3_replica3:org.apache.solr.common.SolrException: Cloud state still says we are leader. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:332) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) This communication is confidential. Frontier only sends and receives email on the basis of the terms set out at http://www.frontier.com/email_disclaimer.