We have a 3 node solrcloud. Each solr collection has only 1 shard and 1 replica. When we restart the 3 solr nodes, we found all the cores in one solr node are at down state and not changed to other state. The solr node is shown in /live_nodes in zookeeper.
After restarting all the zk server and solr nodes, the issue is resolved. We do not see any clue from the solr.log. 2016-09-06 19:23:16.474 WARN (main) [ ] o.e.j.s.h.RequestLogHandler !RequestLog 2016-09-06 19:23:17.418 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@26837057{/solr,file:/opt/ibm/solrsearch/SolrG2Cld101/solr/server/solr-webapp/webapp/,STARTING}{/opt/ibm/solrsearch/SolrG2Cld101/solr/server/solr-webapp/webapp} has uncovered http methods for path: / 2016-09-06 19:23:18.567 WARN (main) [ ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /mnt/solrdata1/solr/home/lib). -- View this message in context: http://lucene.472066.n3.nabble.com/All-solr-cores-in-a-solr-server-are-down-Cannot-find-anything-from-log-tp4295679.html Sent from the Solr - User mailing list archive at Nabble.com.