Solrcloud TimeoutException: Idle timeout expired
I am seeing this error in our logs. Our Solr heap is set to more than 10G. Any clues which anyone can provide will be very helpful. Thank you null:java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 12/12 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1075) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313) at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100) at org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79) at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88) at org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107) at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 12/12 ms at
Solrcloud TimeoutException: Idle timeout expired
I am seeing this error in our logs. Our Solr heap is set to more than 10G. Any clues which anyone can provide will be very helpful. Thank you null:java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 12/12 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1075) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313) at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100) at org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79) at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88) at org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107) at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 12/12 ms at
Zookeeper restart on solr basic authentication password change
I recently updated the password on our 3 node solr cloud. We have 3 zookeeper ensemble serving it. I have one question. Do I need to restart the zookeeper ensemble after password change? Thank you
Re: SolrCoreInitializationException after restart of one solr node
Sure will do thank you On Thursday, September 20, 2018, Erick Erickson wrote: > Try "bin/solr zk rm -help" > On Thu, Sep 20, 2018 at 8:51 AM Schaum Mallik > wrote: > > > > Sure Shawn will do what you suggested in the previous emails. Thank you > so > > much > > > > On Thursday, September 20, 2018, Shawn Heisey > wrote: > > > > > On 9/20/2018 9:32 AM, Schaum Mallik wrote: > > > > > >> ‘Then use "bin/solr zk rm" to get rid of it from ZK.‘ <— can you give > the > > >> full command for this one if you don’t mind > > >> > > > > > > Before doing this, try what I suggested. I am not sure that you need > to > > > mess with what you have in ZK. > > > > > > If you have core.properties files in ZK, that should not cause any > issues > > > with any current version of Solr, or the upcoming 8.0 as far as I > know. I > > > suppose it's possible that it might cause problems with a future > version, > > > but honestly I doubt that it will ever be a problem. > > > > > > The path to delete in ZK would be /configs/XXX/core.properties, where > XXX > > > is the name of the config. The name of the config can be the same as > the > > > core, but can also be different. I think you can probably use this > command: > > > > > > bin/solr zk rm /configs/XXX/core.properties -z ZKHOST > > > > > > Where XXX is the name of the collection and ZKHOST is the ZKHOST string > > > that your Solr installs are using, listing all your ZK servers. I have > > > tested this and it should work. > > > > > > Thanks, > > > Shawn > > > > > > >
Re: SolrCoreInitializationException after restart of one solr node
Sure Shawn will do what you suggested in the previous emails. Thank you so much On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 9:32 AM, Schaum Mallik wrote: > >> ‘Then use "bin/solr zk rm" to get rid of it from ZK.‘ <— can you give the >> full command for this one if you don’t mind >> > > Before doing this, try what I suggested. I am not sure that you need to > mess with what you have in ZK. > > If you have core.properties files in ZK, that should not cause any issues > with any current version of Solr, or the upcoming 8.0 as far as I know. I > suppose it's possible that it might cause problems with a future version, > but honestly I doubt that it will ever be a problem. > > The path to delete in ZK would be /configs/XXX/core.properties, where XXX > is the name of the config. The name of the config can be the same as the > core, but can also be different. I think you can probably use this command: > > bin/solr zk rm /configs/XXX/core.properties -z ZKHOST > > Where XXX is the name of the collection and ZKHOST is the ZKHOST string > that your Solr installs are using, listing all your ZK servers. I have > tested this and it should work. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
‘Then use "bin/solr zk rm" to get rid of it from ZK.‘ <— can you give the full command for this one if you don’t mind Thank you On Thursday, September 20, 2018, Erick Erickson wrote: > bq. In response to this mistake that I did of keeping the core.properties > in > the configuration directory when it was uploaded to zookeeper, how should I > go about fixing it? > > Oh my. You're saying that core.properties is _in_ ZooKeeper? Never > seen that before ;). > > OK, I think I see what's happening. Somehow you have nodes that have > the core.properties file in your ...solr/server/configsets. Your > SOLR_HOME some ancestor of that directory and the recursive search is > finding it and trying to load it. > > So just remove it from that directory on all the Solr nodes where it > exists. I'd have Solr shut down at the time, but that's probably not > strictly necessary. > > Then use "bin/solr zk rm" to get rid of it from ZK. > > If possible, I'd do all this with all the Solr instances shut down, > but that's probably not absolutely necessary, I'm just paranoid. > > On Thu, Sep 20, 2018 at 8:13 AM Schaum Mallik > wrote: > > > > In response to this mistake that I did of keeping the core.properties in > > the configuration directory when it was uploaded to zookeeper, how > should I > > go about fixing it? > > > > Thank you > > > > On Thursday, September 20, 2018, Schaum Mallik > > wrote: > > > > > Hi Eric > > > > > > I created the collection the way you detailed it in here. The > > > configuration directory for the collections was copied over from the > > > standalone solr 6. the core.properties existed in the conf directory > and I > > > didn’t realize it would cause this issue. > > > > > > Also all the nodes are solr 7.4. > > > > > > Thank you > > > > > > On Thursday, September 20, 2018, Erick Erickson < > erickerick...@gmail.com> > > > wrote: > > > > > >> I agree with Shawn that having > > >> instanceDir=/opt/solr/server/solr/configsets/articles as where your > > >> core is located is strange, how are you starting Solr? In particular, > > >> do those nodes use a "-s' parameter to redefine SOLR_HOME? > > >> > > >> The "-d" option (assuming you created the collection with bin/solr) > > >> just tells Solr where the configset you're using is located on your > > >> local disk, it does _not_ put the instanceDir there, it just looks for > > >> the configset to upload to ZK. > > >> > > >> Check if there's a "core.properties" file in > > >> "/opt/solr/server/solr/configsets/articles". The presence of that > file > > >> indicates that that's the home of the replica. > > >> > > >> Now to the very weird bit. This sounds an awful lot like > > >> https://issues.apache.org/jira/browse/SOLR-11503. Especially if your > > >> old system was Solr 6.6.1 or 6.6.2.. Short form, "core.properties" > > >> needs to contain a, you guessed it, "coreNodeName" property that > > >> corresponds to what's in ZooKeeper. > > >> > > >> However, I simply cannot reconcile this being the problem with what > > >> you're reporting. There's code in 7x that repairs this issue > > >> automatically. Is there any chance for the node in question that it's > > >> _not_ running 7.4 and still on 6.6.1/2? Maybe Shawn's latest comment > > >> would reconcile that? > > >> > > >> How to fix. Well, fixing should be automatic unless the fixer-upper > > >> code is no longer in 7.4, but on a quick check the code _is_ in 7.4. > > >> > Make very, very sure the Solr you're running on the nodes in > question > > >> is 7.4, or at least _NOT_ 6.6.1/2 > > >> > Make very, very sure that you don't specify some strange "-s" > parameter > > >> or have defined SOLR_HOME to point to /opt/solr/server/solr/ > configsets/articles, > > >> although that shouldn't really matter. > > >> > When you DELETEREPLICA, go to the node and manually see if the > > >> directory /opt/solr/server/solr/configsets/articles exists. It > shouldn't > > >> (see the deleteInstanceDir option in the DELETEREPLICA for why). > > >> > > >> If none of that makes any difference, please show us the full output > of > > >> > ps aux | grep solr > > >> > expor
Re: SolrCoreInitializationException after restart of one solr node
Thanks Shawn. Will follow your advice on this one. Thank you so much On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 9:25 AM, Schaum Mallik wrote: > >> ok so that’s the problem. The core.properties in the replicas under >> /opt/solr/server/solr. So if I remove the file from all the replica >> folders >> and also move the directories under /opt/solr/server/solr/configsets to >> some backup location and restart this node then hopefully it should start >> working correct? >> > > The cores with names that include 'shard' and 'replica' are likely fine > and should not be touched. > > It seems to be the ones under configsets that are the problem. I would > only move instanceDir locations that give you the error in the log. Any > directory with core.properties that doesn't give you an error, don't touch. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
ok so that’s the problem. The core.properties in the replicas under /opt/solr/server/solr. So if I remove the file from all the replica folders and also move the directories under /opt/solr/server/solr/configsets to some backup location and restart this node then hopefully it should start working correct? Thank you On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 9:13 AM, Schaum Mallik wrote: > >> In response to this mistake that I did of keeping the core.properties in >> the configuration directory when it was uploaded to zookeeper, how should >> I >> go about fixing it? >> > > A core.properties file in the config in ZK will not cause any problems. > It shouldn't be there, but it would be ignored. > > This problem is happening because you have a core.properties file in > directories under /opt/solr/server/solr, so Solr is seeing that and loading > the location as a core. Because you're in cloud mode, it tries to get > information about the node from ZK, and it's not there, because when those > core.properties files were created, the instance was NOT running in cloud > mode. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
In response to this mistake that I did of keeping the core.properties in the configuration directory when it was uploaded to zookeeper, how should I go about fixing it? Thank you On Thursday, September 20, 2018, Schaum Mallik wrote: > Hi Eric > > I created the collection the way you detailed it in here. The > configuration directory for the collections was copied over from the > standalone solr 6. the core.properties existed in the conf directory and I > didn’t realize it would cause this issue. > > Also all the nodes are solr 7.4. > > Thank you > > On Thursday, September 20, 2018, Erick Erickson > wrote: > >> I agree with Shawn that having >> instanceDir=/opt/solr/server/solr/configsets/articles as where your >> core is located is strange, how are you starting Solr? In particular, >> do those nodes use a "-s' parameter to redefine SOLR_HOME? >> >> The "-d" option (assuming you created the collection with bin/solr) >> just tells Solr where the configset you're using is located on your >> local disk, it does _not_ put the instanceDir there, it just looks for >> the configset to upload to ZK. >> >> Check if there's a "core.properties" file in >> "/opt/solr/server/solr/configsets/articles". The presence of that file >> indicates that that's the home of the replica. >> >> Now to the very weird bit. This sounds an awful lot like >> https://issues.apache.org/jira/browse/SOLR-11503. Especially if your >> old system was Solr 6.6.1 or 6.6.2.. Short form, "core.properties" >> needs to contain a, you guessed it, "coreNodeName" property that >> corresponds to what's in ZooKeeper. >> >> However, I simply cannot reconcile this being the problem with what >> you're reporting. There's code in 7x that repairs this issue >> automatically. Is there any chance for the node in question that it's >> _not_ running 7.4 and still on 6.6.1/2? Maybe Shawn's latest comment >> would reconcile that? >> >> How to fix. Well, fixing should be automatic unless the fixer-upper >> code is no longer in 7.4, but on a quick check the code _is_ in 7.4. >> > Make very, very sure the Solr you're running on the nodes in question >> is 7.4, or at least _NOT_ 6.6.1/2 >> > Make very, very sure that you don't specify some strange "-s" parameter >> or have defined SOLR_HOME to point to >> /opt/solr/server/solr/configsets/articles, >> although that shouldn't really matter. >> > When you DELETEREPLICA, go to the node and manually see if the >> directory /opt/solr/server/solr/configsets/articles exists. It shouldn't >> (see the deleteInstanceDir option in the DELETEREPLICA for why). >> >> If none of that makes any difference, please show us the full output of >> > ps aux | grep solr >> > export (looking for you redefining SOLR_HOME) >> > the "core.properties" file in the indicated directory. >> >> Best, >> Erick >> >> On Thu, Sep 20, 2018 at 7:36 AM Shawn Heisey wrote: >> > >> > On 9/20/2018 8:22 AM, Schaum Mallik wrote: >> > > Thanks for the response Shawn. >> > > >> > > My follow up question is how would the zookeeper ensemble know that >> the >> > > location of the indexes has changed? Also do I need to apply the same >> > > changes to the other 2 solr nodes which are working fine? >> > >> > This move is not to change the location, it's to stop Solr from trying >> > to load the indexes completely. Solr will no longer try to load them. >> > I think these are invalid indexes from when the Solr instance was NOT >> > running in cloud mode, and have nothing at all to do with your working >> > collections. But just in case I'm wrong, I'm having you move them >> > instead of delete them, so they can be put back if it turns out they >> > actually were needed. >> > >> > ZooKeeper doesn't know anything at all about Solr and the data it puts >> > in ZK. It is just a service that stores cluster data where all nodes >> > can read it and handles elections when it is asked to. >> > >> > Thanks, >> > Shawn >> > >> >
Re: SolrCoreInitializationException after restart of one solr node
Hi Eric I created the collection the way you detailed it in here. The configuration directory for the collections was copied over from the standalone solr 6. the core.properties existed in the conf directory and I didn’t realize it would cause this issue. Also all the nodes are solr 7.4. Thank you On Thursday, September 20, 2018, Erick Erickson wrote: > I agree with Shawn that having > instanceDir=/opt/solr/server/solr/configsets/articles as where your > core is located is strange, how are you starting Solr? In particular, > do those nodes use a "-s' parameter to redefine SOLR_HOME? > > The "-d" option (assuming you created the collection with bin/solr) > just tells Solr where the configset you're using is located on your > local disk, it does _not_ put the instanceDir there, it just looks for > the configset to upload to ZK. > > Check if there's a "core.properties" file in > "/opt/solr/server/solr/configsets/articles". The presence of that file > indicates that that's the home of the replica. > > Now to the very weird bit. This sounds an awful lot like > https://issues.apache.org/jira/browse/SOLR-11503. Especially if your > old system was Solr 6.6.1 or 6.6.2.. Short form, "core.properties" > needs to contain a, you guessed it, "coreNodeName" property that > corresponds to what's in ZooKeeper. > > However, I simply cannot reconcile this being the problem with what > you're reporting. There's code in 7x that repairs this issue > automatically. Is there any chance for the node in question that it's > _not_ running 7.4 and still on 6.6.1/2? Maybe Shawn's latest comment > would reconcile that? > > How to fix. Well, fixing should be automatic unless the fixer-upper > code is no longer in 7.4, but on a quick check the code _is_ in 7.4. > > Make very, very sure the Solr you're running on the nodes in question is > 7.4, or at least _NOT_ 6.6.1/2 > > Make very, very sure that you don't specify some strange "-s" parameter > or have defined SOLR_HOME to point to > /opt/solr/server/solr/configsets/articles, > although that shouldn't really matter. > > When you DELETEREPLICA, go to the node and manually see if the directory > /opt/solr/server/solr/configsets/articles exists. It shouldn't (see the > deleteInstanceDir option in the DELETEREPLICA for why). > > If none of that makes any difference, please show us the full output of > > ps aux | grep solr > > export (looking for you redefining SOLR_HOME) > > the "core.properties" file in the indicated directory. > > Best, > Erick > > On Thu, Sep 20, 2018 at 7:36 AM Shawn Heisey wrote: > > > > On 9/20/2018 8:22 AM, Schaum Mallik wrote: > > > Thanks for the response Shawn. > > > > > > My follow up question is how would the zookeeper ensemble know that the > > > location of the indexes has changed? Also do I need to apply the same > > > changes to the other 2 solr nodes which are working fine? > > > > This move is not to change the location, it's to stop Solr from trying > > to load the indexes completely. Solr will no longer try to load them. > > I think these are invalid indexes from when the Solr instance was NOT > > running in cloud mode, and have nothing at all to do with your working > > collections. But just in case I'm wrong, I'm having you move them > > instead of delete them, so they can be put back if it turns out they > > actually were needed. > > > > ZooKeeper doesn't know anything at all about Solr and the data it puts > > in ZK. It is just a service that stores cluster data where all nodes > > can read it and handles elections when it is asked to. > > > > Thanks, > > Shawn > > >
Re: SolrCoreInitializationException after restart of one solr node
Hi Shawn Sorry for being repetitive but want to make sure I understand you clearly. So my real collection data sits just outside the configsets folder. So if I move the location mentioned under instancedir to some backup location and then restart solr cloud you think this non working node will get the real collection data sitting outside the configsets dir. I really appreciate the time and effort you are taking in answering my queries. Thank you On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 8:44 AM, Schaum Mallik wrote: > >> Thank you for your detailed responses. I am still kind of confused though. >> Just to give you some more insight. When I first created the cloud I >> created to collections and the ‘-d’ option pointed to the directory where >> the config for the collection was stored which was in >> /opt/solr/server/solr/configsets/articles/conf >> >> so as you are saying if I move the replica data to another location and >> restart the node and most probably solr might complain then what happens >> next. This solr node was working fine a week back and the indexes were >> upgraded when they were moved from solr 6 to solr 7 >> > > I really think that those cores that won't initialize are remnants from a > non-cloud install and probably have *never* worked. For some reason, you > didn't notice these errors until now. > > If you take a non-cloud Solr instance that has indexes and add ZK_HOST to > make it a cloud instance, the cores that existed before cloud mode will NOT > work. > > I just completed a test where I did that exact sequence of converting a > node from non-cloud with existing indexes to cloud, and ran into the exact > same errors you're seeing. I can relay exact details of the test if > required. > > I think that moving the non-cloud cores out so Solr doesn't try to load > them will likely fix this issue. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
Thank you for your detailed responses. I am still kind of confused though. Just to give you some more insight. When I first created the cloud I created to collections and the ‘-d’ option pointed to the directory where the config for the collection was stored which was in /opt/solr/server/solr/configsets/articles/conf so as you are saying if I move the replica data to another location and restart the node and most probably solr might complain then what happens next. This solr node was working fine a week back and the indexes were upgraded when they were moved from solr 6 to solr 7 Let me know your thoughts Thank you On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 8:22 AM, Schaum Mallik wrote: > >> Thanks for the response Shawn. >> >> My follow up question is how would the zookeeper ensemble know that the >> location of the indexes has changed? Also do I need to apply the same >> changes to the other 2 solr nodes which are working fine? >> > > This move is not to change the location, it's to stop Solr from trying to > load the indexes completely. Solr will no longer try to load them. I > think these are invalid indexes from when the Solr instance was NOT running > in cloud mode, and have nothing at all to do with your working > collections. But just in case I'm wrong, I'm having you move them instead > of delete them, so they can be put back if it turns out they actually were > needed. > > ZooKeeper doesn't know anything at all about Solr and the data it puts in > ZK. It is just a service that stores cluster data where all nodes can read > it and handles elections when it is asked to. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
Thanks for the response Shawn. My follow up question is how would the zookeeper ensemble know that the location of the indexes has changed? Also do I need to apply the same changes to the other 2 solr nodes which are working fine? Thanks On Thursday, September 20, 2018, Shawn Heisey wrote: > On 9/20/2018 6:02 AM, Schaum Mallik wrote: > >> Yeah my indexes, read and write works fine on the other two solr nodes. >> Since I have this setup running in prod currently what are the steps you >> will advice I take to resolve this issue. Starting from scratch is really >> not an option since it will involve a huge cost. Is there some workaround >> that you guys can suggest. >> when you say move the instance dir outside the /op/solr are you >> suggesting, >> modifying this value 'SOLR_HOME'? >> > > No, I mean literally moving the directory somewhere else. Something like > this: > > mkdir /tmp/index-keep > mv /opt/solr/server/solr/configsets/articles /tmp/index-keep/. > mv /opt/solr/server/solr/configsets/categories /tmp/index-keep/. > # move any other non-cloud cores out > # restart Solr here > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
Yeah my indexes, read and write works fine on the other two solr nodes. Since I have this setup running in prod currently what are the steps you will advice I take to resolve this issue. Starting from scratch is really not an option since it will involve a huge cost. Is there some workaround that you guys can suggest. when you say move the instance dir outside the /op/solr are you suggesting, modifying this value 'SOLR_HOME'? Thank you On Thu, Sep 20, 2018 at 3:00 AM Shawn Heisey wrote: > On 9/19/2018 9:20 PM, Schaum Mallik wrote: > > The data and index get stored under > > /opt/solr/server/solr/articles_shard1_replica_n1. > > The config directory when the collection was created, that time the path > to > > the config was given as '/opt/solr/server/solr/configsets/articles'. I > > didn't use the service installer script. The other two solr nodes are > > working without any issue. Any ideas how I can resolve this. > > Do your indexes still work after startup, even with those initialization > errors? I think there's a good chance that they do. > > This looks to me like what you did was start with a non-cloud install > that was using the configsets feature, so instanceDir actually WAS in > configsets, and then converted it to a cloud setup, at which point those > old cores wouldn't have worked right, so you probably created the > collections again. I'm betting that if you moved those instancedirs > that are under configsets somewhere else (not under /opt/solr at all), > that everything MIGHT be fine after another restart, based on the fact > that in the core list I see xxx_shardN_replica_nM cores for articles and > categories, which would probably be the REAL indexes being used for > those names. > > Converting from cloud to non-cloud is a process that requires > understanding exactly how Solr deals with cores at startup, and isn't > something I would recommend anyone ever try do. Set your cloud up from > scratch in cloud mode. If you're using the configsets feature in > non-cloud mode, that will confuse things even more. > > Also, I strongly recommend using the service installer script. It > creates a much cleaner install that I find much easier to understand. > The data is entirely separate from the program install. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
I also want to add one other things. I had moved from a single core solr instance on solr 6.6 to the solr cloud few months back. I had ran the indexupgrader tool on the indexes before I moved them to the solr cloud. On Wed, Sep 19, 2018 at 7:29 PM Shawn Heisey wrote: > On 9/19/2018 8:22 PM, Schaum Mallik wrote: > > I have a 3 zookeeper ensemble and 3 solr nodes running version 7.4.0. > > Recently I had to restart one node and after I did that it started > throwing > > this exception. > > > Caused by: org.apache.solr.common.SolrException: No coreNodeName > > for > > > CoreDescriptor[name=articles;instanceDir=/opt/solr/server/solr/configsets/articles] > > It is VERY weird for instanceDir to be under the configsets directory. > Especially for SolrCloud. I know you're in cloud mode because the > ZK-related classes are heavily mentioned in the stacktrace. > > Can you share the entire solr.log file after a restart? If that > directory really is the instanceDir, there should be a core.properties > file. What are its contents? > > Did you use the service installer script? If you did, having an > instanceDir underneath configsets is even MORE strange. > > Thanks, > Shawn > >
Re: SolrCoreInitializationException after restart of one solr node
ategories x:categories] o.a.s.c.ZkController org.apache.solr.common.SolrException: No coreNodeName for CoreDescriptor[name=categories;instanceDir=/opt/solr/server/solr/configsets/categories] at org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1716) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1654) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1128) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:681) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ~[metrics-core-3.2.6.jar:3.2.6] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] 2018-09-20 03:01:36.002 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:solr1:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup org.apache.solr.common.SolrException: Unable to create core [categories] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1156) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:681) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ~[metrics-core-3.2.6.jar:3.2.6] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] Caused by: org.apache.solr.common.SolrException: at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1685) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1128) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] ... 7 more Caused by: org.apache.solr.common.SolrException: No coreNodeName for CoreDescriptor[name=categories;instanceDir=/opt/solr/server/solr/configsets/categories] at org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1716) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1654) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1128) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] ... 7 more On Wed, Sep 19, 2018 at 7:29 PM Shawn Heisey wrote: > On 9/19/2018 8:22 PM, Schaum Mallik wrote: > > I have a 3 zookeeper ensemble and 3 solr nodes running version 7.4.0. > > Recently I had to restart one node and after I did that it started > throwing > > this exception. > > > Caused by: org.apache.solr.common.SolrException: No coreNodeName > > for > > > CoreDescriptor[name=articles;instanceDir=/opt/solr/server/solr/configsets/articles] > > It is VERY weird for instanceDir to be under the configsets directory. > Especially for SolrCloud. I know you're in cloud mode because the > ZK-related classes are heavily mentioned in the stacktrace. > > Can you share the entire solr.log file after a restart? If that > directory really is the instanceDir, there should be a core.properties > file. What are its contents? > > Did you use the service installer
SolrCoreInitializationException after restart of one solr node
Hi Guys
I have a 3 zookeeper ensemble and 3 solr nodes running version 7.4.0.
Recently I had to restart one node and after I did that it started throwing
this exception.
{
"error":{
"metadata":[
"error-class","org.apache.solr.core.SolrCoreInitializationException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"SolrCore 'articles' is not available due to init failure: ",
"trace":"org.apache.solr.core.SolrCoreInitializationException: SolrCore
'articles' is not available due to init failure: \n\tat
org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1590)\n\tat
org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:249)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
org.apache.solr.common.SolrException: \n\tat
org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1685)\n\tat
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1128)\n\tat
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:681)\n\tat
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t...
1 more\nCaused by: org.apache.solr.common.SolrException: No coreNodeName
for
CoreDescriptor[name=articles;instanceDir=/opt/solr/server/solr/configsets/articles]\n\tat
org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1716)\n\tat
org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1654)\n\t...
8 more\n",
"code":500}}
I tried dropping
