RE: Solr 4.3 Master/Slave Issues
thanks Erick. My tomcat setup is running on Ubuntu and has nothing else deployed other than the Solr war. My suspicion is it takes a long time to de-allocate the memory it has reserved for itself, but will get dump to find better. regards, Ayush > Date: Sat, 6 Jul 2013 07:55:57 -0400 > Subject: Re: Solr 4.3 Master/Slave Issues > From: erickerick...@gmail.com > To: solr-user@lucene.apache.org > > kill -9 is evil, you aren't certain what the state is afterwards, > so the presence of the lock file is not surprising.. > > solrconfig.xml has a commented-out entry > > false > > I haven't personally used it, but it looks like it might > help if you insist on kill -9. > > Personally I'd try to figure out why your setup takes > so long to shut down, I know lots of Tomcat installations > that seem to function so this might point to something > else in your setup that needs fixing. > > Best > Erick > > > On Fri, Jul 5, 2013 at 11:54 AM, Cool Techi wrote: > > > The normal tomcat shutdown doesn't stop the server and take a long time, > > so i do issue a kill -9 command. Any other suggestion to do this without > > the locking. > > > > I would initiate a backup again and send the logs. > > > > regards, > > Ayush > > > > > Date: Fri, 5 Jul 2013 19:40:12 +0530 > > > Subject: Re: Solr 4.3 Master/Slave Issues > > > From: shalinman...@gmail.com > > > To: solr-user@lucene.apache.org > > > > > > On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi > > wrote: > > > > > > > > 1) That was my initial suspicion, but when I run ps -aux | grep > > "java", but there it doesn't show any other program running. I kill the > > process and start again and it locks. > > > > > > How are you killing the process? A SIGKILL will leave a lock file. > > > > > > > > > > > 2) When we fire backup on Slave, the whole core hangs after a while > > and also replication stops. This was not happening with our earlier setup > > of solr3.6 > > > > > > > > > > Just to confirm, are you using the backup command of replication > > > handler? Can you post a thread dump of the process to pastebin or > > > http://apaste.info/ ? > > > > > > > Regards, > > > > > > > > > > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 > > > >> Subject: Re: Solr 4.3 Master/Slave Issues > > > >> From: shalinman...@gmail.com > > > >> To: solr-user@lucene.apache.org > > > >> > > > >> This can mean multiple things: > > > >> 1. You had killed a solr process earlier which left the lock file in > > place > > > >> 2. You have more than one Solr core pointing to the same data > > directory > > > >> 3. A solr process is already running and you are trying to start > > > >> another one with the same config. > > > >> > > > >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi > > wrote: > > > >> > We have set up solr 4.3 with master/setup and are facing a couple > > of issues, > > > >> > > > > >> > Index locking, the index on slave hangs at time and when we restart > > the core the core get's locked up. I have checked the logs and there are no > > OOM error or anything else other than the error given below,Caused by: > > org.apache.solr.common.SolrException: Index locked for write for core XXX > > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > > > >> > at > > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > > > >> > at > > org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > > > >> > at > > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > > > >> > at > > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > > > >> > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > > >> > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > &
Re: Solr 4.3 Master/Slave Issues
kill -9 is evil, you aren't certain what the state is afterwards, so the presence of the lock file is not surprising.. solrconfig.xml has a commented-out entry false I haven't personally used it, but it looks like it might help if you insist on kill -9. Personally I'd try to figure out why your setup takes so long to shut down, I know lots of Tomcat installations that seem to function so this might point to something else in your setup that needs fixing. Best Erick On Fri, Jul 5, 2013 at 11:54 AM, Cool Techi wrote: > The normal tomcat shutdown doesn't stop the server and take a long time, > so i do issue a kill -9 command. Any other suggestion to do this without > the locking. > > I would initiate a backup again and send the logs. > > regards, > Ayush > > > Date: Fri, 5 Jul 2013 19:40:12 +0530 > > Subject: Re: Solr 4.3 Master/Slave Issues > > From: shalinman...@gmail.com > > To: solr-user@lucene.apache.org > > > > On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi > wrote: > > > > > > 1) That was my initial suspicion, but when I run ps -aux | grep > "java", but there it doesn't show any other program running. I kill the > process and start again and it locks. > > > > How are you killing the process? A SIGKILL will leave a lock file. > > > > > > > > 2) When we fire backup on Slave, the whole core hangs after a while > and also replication stops. This was not happening with our earlier setup > of solr3.6 > > > > > > > Just to confirm, are you using the backup command of replication > > handler? Can you post a thread dump of the process to pastebin or > > http://apaste.info/ ? > > > > > Regards, > > > > > > > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 > > >> Subject: Re: Solr 4.3 Master/Slave Issues > > >> From: shalinman...@gmail.com > > >> To: solr-user@lucene.apache.org > > >> > > >> This can mean multiple things: > > >> 1. You had killed a solr process earlier which left the lock file in > place > > >> 2. You have more than one Solr core pointing to the same data > directory > > >> 3. A solr process is already running and you are trying to start > > >> another one with the same config. > > >> > > >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi > wrote: > > >> > We have set up solr 4.3 with master/setup and are facing a couple > of issues, > > >> > > > >> > Index locking, the index on slave hangs at time and when we restart > the core the core get's locked up. I have checked the logs and there are no > OOM error or anything else other than the error given below,Caused by: > org.apache.solr.common.SolrException: Index locked for write for core XXX > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > > >> > at > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > > >> > at > org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > > >> > at > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > > >> > at > org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > > >> > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > >> > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > >> > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > >> > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > >> > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > >> > ... 1 more > > >> > Caused by: org.apache.lucene.store.LockObtainFailedException: Index > locked for write for core XXat > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > > >> > ... 13 moreWhen we fire backup on Slave, the whole core hangs > after a while and also replication stops. This was not happening with our > earlier setup of solr3.6 > > >> > > > >> > > > >> > > > >> > Regards, > > >> > Ayush > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> Shalin Shekhar Mangar. > > > > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > >
RE: Solr 4.3 Master/Slave Issues
The normal tomcat shutdown doesn't stop the server and take a long time, so i do issue a kill -9 command. Any other suggestion to do this without the locking. I would initiate a backup again and send the logs. regards, Ayush > Date: Fri, 5 Jul 2013 19:40:12 +0530 > Subject: Re: Solr 4.3 Master/Slave Issues > From: shalinman...@gmail.com > To: solr-user@lucene.apache.org > > On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi wrote: > > > > 1) That was my initial suspicion, but when I run ps -aux | grep "java", but > > there it doesn't show any other program running. I kill the process and > > start again and it locks. > > How are you killing the process? A SIGKILL will leave a lock file. > > > > > 2) When we fire backup on Slave, the whole core hangs after a while and > > also replication stops. This was not happening with our earlier setup of > > solr3.6 > > > > Just to confirm, are you using the backup command of replication > handler? Can you post a thread dump of the process to pastebin or > http://apaste.info/ ? > > > Regards, > > > > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 > >> Subject: Re: Solr 4.3 Master/Slave Issues > >> From: shalinman...@gmail.com > >> To: solr-user@lucene.apache.org > >> > >> This can mean multiple things: > >> 1. You had killed a solr process earlier which left the lock file in place > >> 2. You have more than one Solr core pointing to the same data directory > >> 3. A solr process is already running and you are trying to start > >> another one with the same config. > >> > >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: > >> > We have set up solr 4.3 with master/setup and are facing a couple of > >> > issues, > >> > > >> > Index locking, the index on slave hangs at time and when we restart the > >> > core the core get's locked up. I have checked the logs and there are no > >> > OOM error or anything else other than the error given below,Caused by: > >> > org.apache.solr.common.SolrException: Index locked for write for core XXX > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > >> > at > >> > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > >> > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> > at > >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > ... 1 more > >> > Caused by: org.apache.lucene.store.LockObtainFailedException: Index > >> > locked for write for core XXat > >> > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > >> > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > >> > ... 13 moreWhen we fire backup on Slave, the whole core hangs after > >> > a while and also replication stops. This was not happening with our > >> > earlier setup of solr3.6 > >> > > >> > > >> > > >> > Regards, > >> > Ayush > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > >> > >> -- > >> Regards, > >> Shalin Shekhar Mangar. > > > > > > -- > Regards, > Shalin Shekhar Mangar.
Re: Solr 4.3 Master/Slave Issues
On Fri, Jul 5, 2013 at 6:14 PM, Cool Techi wrote: > > 1) That was my initial suspicion, but when I run ps -aux | grep "java", but > there it doesn't show any other program running. I kill the process and start > again and it locks. How are you killing the process? A SIGKILL will leave a lock file. > > 2) When we fire backup on Slave, the whole core hangs after a while and also > replication stops. This was not happening with our earlier setup of solr3.6 > Just to confirm, are you using the backup command of replication handler? Can you post a thread dump of the process to pastebin or http://apaste.info/ ? > Regards, > > >> Date: Fri, 5 Jul 2013 16:49:24 +0530 >> Subject: Re: Solr 4.3 Master/Slave Issues >> From: shalinman...@gmail.com >> To: solr-user@lucene.apache.org >> >> This can mean multiple things: >> 1. You had killed a solr process earlier which left the lock file in place >> 2. You have more than one Solr core pointing to the same data directory >> 3. A solr process is already running and you are trying to start >> another one with the same config. >> >> On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: >> > We have set up solr 4.3 with master/setup and are facing a couple of >> > issues, >> > >> > Index locking, the index on slave hangs at time and when we restart the >> > core the core get's locked up. I have checked the logs and there are no >> > OOM error or anything else other than the error given below,Caused by: >> > org.apache.solr.common.SolrException: Index locked for write for core XXX >> > at org.apache.solr.core.SolrCore.(SolrCore.java:821) >> > at org.apache.solr.core.SolrCore.(SolrCore.java:618) >> > at >> > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) >> > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) >> > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > at >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > ... 1 more >> > Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked >> > for write for core XXat >> > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) >> > at org.apache.solr.core.SolrCore.(SolrCore.java:745) >> > ... 13 moreWhen we fire backup on Slave, the whole core hangs after a >> > while and also replication stops. This was not happening with our earlier >> > setup of solr3.6 >> > >> > >> > >> > Regards, >> > Ayush >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. > -- Regards, Shalin Shekhar Mangar.
RE: Solr 4.3 Master/Slave Issues
1) That was my initial suspicion, but when I run ps -aux | grep "java", but there it doesn't show any other program running. I kill the process and start again and it locks. 2) When we fire backup on Slave, the whole core hangs after a while and also replication stops. This was not happening with our earlier setup of solr3.6 Regards, > Date: Fri, 5 Jul 2013 16:49:24 +0530 > Subject: Re: Solr 4.3 Master/Slave Issues > From: shalinman...@gmail.com > To: solr-user@lucene.apache.org > > This can mean multiple things: > 1. You had killed a solr process earlier which left the lock file in place > 2. You have more than one Solr core pointing to the same data directory > 3. A solr process is already running and you are trying to start > another one with the same config. > > On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: > > We have set up solr 4.3 with master/setup and are facing a couple of issues, > > > > Index locking, the index on slave hangs at time and when we restart the > > core the core get's locked up. I have checked the logs and there are no OOM > > error or anything else other than the error given below,Caused by: > > org.apache.solr.common.SolrException: Index locked for write for core XXX > > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > > at > > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > ... 1 more > > Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked > > for write for core XXat > > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > > ... 13 moreWhen we fire backup on Slave, the whole core hangs after a > > while and also replication stops. This was not happening with our earlier > > setup of solr3.6 > > > > > > > > Regards, > > Ayush > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Regards, > Shalin Shekhar Mangar.
Re: Solr 4.3 Master/Slave Issues
This can mean multiple things: 1. You had killed a solr process earlier which left the lock file in place 2. You have more than one Solr core pointing to the same data directory 3. A solr process is already running and you are trying to start another one with the same config. On Fri, Jul 5, 2013 at 2:34 PM, Cool Techi wrote: > We have set up solr 4.3 with master/setup and are facing a couple of issues, > > Index locking, the index on slave hangs at time and when we restart the core > the core get's locked up. I have checked the logs and there are no OOM error > or anything else other than the error given below,Caused by: > org.apache.solr.common.SolrException: Index locked for write for core XXX > at org.apache.solr.core.SolrCore.(SolrCore.java:821) > at org.apache.solr.core.SolrCore.(SolrCore.java:618) > at > org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949) > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984) > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597) > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > ... 1 more > Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked > for write for core XXat > org.apache.solr.core.SolrCore.initIndex(SolrCore.java:484) > at org.apache.solr.core.SolrCore.(SolrCore.java:745) > ... 13 moreWhen we fire backup on Slave, the whole core hangs after a > while and also replication stops. This was not happening with our earlier > setup of solr3.6 > > > > Regards, > Ayush > > > > > > > > > > > > > > -- Regards, Shalin Shekhar Mangar.
