Re: Solr Index Lock Issue
: Subject: Solr Index Lock Issue : Date: Fri, 16 Apr 2010 22:43:57 +0530 : In-Reply-To: : References: <1271363131217-722255.p...@n3.nabble.com> : http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email. Even if you change the subject line of your email, other mail headers still track which thread you replied to and your question is "hidden" in that thread and gets less attention. It makes following discussions in the mailing list archives particularly difficult. See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking -Hoss
RE: Solr Index Lock Issue
Hi Otis, Thanks. I cleared the data folder and restarted the Solr server. We trigger solr ingestion through Java. We tried removing all incremental solr.commit() after every batch, and added a commit command using curl in the shell script file after the ingestion finishes. Content of .sh file - ### java mySolrIngestionClass curl "http://localhost:9083/solr/update?commit=true"; ### mySolrIngestionClass adds documents to solr, and does not contain any commit. The observation is that the lock file still appears in the Solr/data/index folder within 30 mins of Solr ingestion and after that solr hangs. In threaddump.jsp of Solr, it shows some of the threads in Blocked state. We checked the solr config, the autocommit is also commented out. We are using Solr 1.4 with tomcat 1.6.18. It used to run earlier without any issues in 24 hrs time, but now even after 3 days its not completed. Any pointers to the solution will be great. Thanks! Regards, Param -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Saturday, April 17, 2010 3:50 AM To: solr-user@lucene.apache.org Subject: Re: Solr Index Lock Issue Hi, What you are doing sounds fine. You don't need to commit while indexing, though, just commit/optimize at the end. I'm not saying this will solve your problem, but give it a try. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Hadoop ecosystem search :: http://search-hadoop.com/ - Original Message > From: "Sethi, Parampreet" > To: solr-user@lucene.apache.org > Sent: Fri, April 16, 2010 1:13:57 PM > Subject: Solr Index Lock Issue > > Hi All, We are facing the issue with the Solr server in the DMOZ data > migration. The Solr has 0 records when the migration starts and the data is > added into Solr in the batches of 2 records. The commit is called on > Solr after 20k records are processed. While commiting the data into > Solr, a lucene lock file is created in the /data/index > folder which is automatically released once the successful commit happens. > But after 4-5 batches, the lock file remains there and Solr just hangs and > does not add any new records. Some times the whole migration goes through > without any errors. Kindly let me know in case some setting needs to be > required on Solr side, which ensures that until the Solr commits the index, > the next set of records should not be added. Thanks, Param
Re: Solr Index Lock Issue
The commit call can return before Solr is completely finished with everything it does after a commit. This tail processing can build up with successive commits until Solr is basically stuck. Commit should use waitFlush=true. This makes the commit call wait until the disk manipulation is done. If you add 'waitSearch=true' and 'expungeDeletes=true', this will wait completely until Solr is ready to do more indexing. "Solr just hangs" is not enough to go on. Solr/Lucene have processing where it takes increasing amounts of time to index data, Is the CPU busy? Is the disk busy? Does 'jhat' or 'visualgc' show memory activity? On 4/16/10, Otis Gospodnetic wrote: > Hi, > > What you are doing sounds fine. You don't need to commit while indexing, > though, just commit/optimize at the end. I'm not saying this will solve > your problem, but give it a try. > > Otis > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Hadoop ecosystem search :: http://search-hadoop.com/ > > > > - Original Message >> From: "Sethi, Parampreet" >> To: solr-user@lucene.apache.org >> Sent: Fri, April 16, 2010 1:13:57 PM >> Subject: Solr Index Lock Issue >> >> Hi All, > > We are facing the issue with the Solr server in the DMOZ data >> migration. > The Solr has 0 records when the migration starts and the data is >> added > into Solr in the batches of 2 records. The commit is called on >> Solr > after 20k records are processed. > > While commiting the data into >> Solr, a lucene lock file is created in the > /data/index >> folder which is automatically released once the > successful commit happens. >> But after 4-5 batches, the lock file remains > there and Solr just hangs and >> does not add any new records. Some times > the whole migration goes through >> without any errors. > > Kindly let me know in case some setting needs to be >> required on Solr > side, which ensures that until the Solr commits the index, >> the next set > of records should not be added. > > Thanks, > Param > -- Lance Norskog goks...@gmail.com
Re: Solr Index Lock Issue
Hi, What you are doing sounds fine. You don't need to commit while indexing, though, just commit/optimize at the end. I'm not saying this will solve your problem, but give it a try. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Hadoop ecosystem search :: http://search-hadoop.com/ - Original Message > From: "Sethi, Parampreet" > To: solr-user@lucene.apache.org > Sent: Fri, April 16, 2010 1:13:57 PM > Subject: Solr Index Lock Issue > > Hi All, We are facing the issue with the Solr server in the DMOZ data > migration. The Solr has 0 records when the migration starts and the data is > added into Solr in the batches of 2 records. The commit is called on > Solr after 20k records are processed. While commiting the data into > Solr, a lucene lock file is created in the /data/index > folder which is automatically released once the successful commit happens. > But after 4-5 batches, the lock file remains there and Solr just hangs and > does not add any new records. Some times the whole migration goes through > without any errors. Kindly let me know in case some setting needs to be > required on Solr side, which ensures that until the Solr commits the index, > the next set of records should not be added. Thanks, Param
Solr Index Lock Issue
Hi All, We are facing the issue with the Solr server in the DMOZ data migration. The Solr has 0 records when the migration starts and the data is added into Solr in the batches of 2 records. The commit is called on Solr after 20k records are processed. While commiting the data into Solr, a lucene lock file is created in the /data/index folder which is automatically released once the successful commit happens. But after 4-5 batches, the lock file remains there and Solr just hangs and does not add any new records. Some times the whole migration goes through without any errors. Kindly let me know in case some setting needs to be required on Solr side, which ensures that until the Solr commits the index, the next set of records should not be added. Thanks, Param
