Hi Eric, Sorry I did not reply earlier. I see this page cached here -
on gmane.org but the original post I posted on Solr Users list
does not show your comment -
http://lucene.472066.n3.nabble.com/Leaders-in-Recovery-Failed-state-td4180610.html
I'm on Solr 4.10.1 - The last time this had happened I removed
the replica for the affected Shards (the shards where the Leaders
were shown as Down) - deleted the Replica data directories and
then added the replica back using the Collections API - the
did the trick then (but I'n not sure if that was the right way to do it).
Also the problem seemed to have rooted from the fact that the
Zookeeper instances were on the same
machines as the Solr servlet containers and perhaps the
Zookeeper instances were starved of resource (CPU & disk) -
I have had since moved the Zookeeper instances out
to separate servers and that makes the boot time fast -
but not all shards come online when all the solr cloud instances
are reboot. A few servers from the Solr Cluster went
down again and I have the same issues where for 3 shards
the Leaders are shown as down and the logs in the log files for
these instances as below -
INFO - 2015-02-09 05:18:13.696; org.apache.solr.handler.
admin.CoreAdminHandler; In
WaitForState(recovering): collection=collection1, shard=shard10,
thisCore=collection1_shard10_replica2,
leaderDoesNotNeedRecovery=false, isLeader?
true, live=true, checkLive=true, currentState=recovering, localState=down,
nodeName=10.68.77.8:8983_solr, coreNodeName=core_node28,
onlyIfActiveCheckResult=true, nodeProps: core_node28:
{"state":"recovering","core":"collection1_shard10_replica1",
"node_name":"10.68.77.8:89
83_solr","base_url":"http://10.68.77.8:8983/solr"}
I have tried deleting the replica for these shards -
but this time the Delete Replica Async
requests are showing in "submitted" state for very long now (over 3 hours) -
last time when I did this these requests finished fairly quickly.
Any pointers are greatly appreciated.
Thanks,
Anand
