Hi Guys
I have a 3 zookeeper ensemble and 3 solr nodes running version 7.4.0.
Recently I had to restart one node and after I did that it started throwing
this exception.
{
"error":{
"metadata":[
"error-class","org.apache.solr.core.SolrCoreInitializationException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"SolrCore 'articles' is not available due to init failure: ",
"trace":"org.apache.solr.core.SolrCoreInitializationException: SolrCore
'articles' is not available due to init failure: \n\tat
org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1590)\n\tat
org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:249)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
org.apache.solr.common.SolrException: \n\tat
org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1685)\n\tat
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1128)\n\tat
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:681)\n\tat
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t...
1 more\nCaused by: org.apache.solr.common.SolrException: No coreNodeName
for
CoreDescriptor[name=articles;instanceDir=/opt/solr/server/solr/configsets/articles]\n\tat
org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1716)\n\tat
org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1654)\n\t...
8 more\n",
"code":500}}
I tried dropping all the replicas and then adding them back but still no
luck. The remaining two replicas return results without any issue. I can
query the two working replicas without any issue. Its this 3rd node which
keeps on throwing the error. I also made sure the write.lock is not causing
any issue and the permissions on the directories are fine. This node was
working fine before this restart. Any pointers you guys can provide will be
great.
Thank you
