Re: SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
it loos like it works, with patch, after a couple of hours of testing under same conditions didn't see it happen (without it, approx. every 15 minutes). I do not think it will happen again with this patch. Thanks again and my respect to your debugging capacity, my bug report was really thin. On Thu, Feb 23, 2012 at 8:47 AM, eks dev eks...@yahoo.co.uk wrote: thanks Mark, I will give it a go and report back... On Thu, Feb 23, 2012 at 1:31 AM, Mark Miller markrmil...@gmail.com wrote: Looks like an issue around replication IndexWriter reboot, soft commits and hard commits. I think I've got a workaround for it: Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java === --- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344) +++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy) @@ -499,6 +499,17 @@ // reboot the writer on the new index and get a new searcher solrCore.getUpdateHandler().newIndexWriter(); + Future[] waitSearcher = new Future[1]; + solrCore.getSearcher(true, false, waitSearcher, true); + if (waitSearcher[0] != null) { + try { + waitSearcher[0].get(); + } catch (InterruptedException e) { + SolrException.log(LOG,e); + } catch (ExecutionException e) { + SolrException.log(LOG,e); + } + } // update our commit point to the right dir solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false)); That should allow the searcher that the following commit command prompts to see the *new* IndexWriter. On Feb 22, 2012, at 10:56 AM, eks dev wrote: We started observing strange failures from ReplicationHandler when we commit on master trunk version 4-5 days old. It works sometimes, and sometimes not didn't dig deeper yet. Looks like the real culprit hides behind: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Looks familiar to somebody? 120222 154959 SEVERE SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043) at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source) at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348) at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815) at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095) ... 15 more - Mark Miller lucidimagination.com
SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
We started observing strange failures from ReplicationHandler when we commit on master trunk version 4-5 days old. It works sometimes, and sometimes not didn't dig deeper yet. Looks like the real culprit hides behind: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Looks familiar to somebody? 120222 154959 SEVERE SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043) at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source) at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348) at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815) at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095) ... 15 more
Re: SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
Looks like an issue around replication IndexWriter reboot, soft commits and hard commits. I think I've got a workaround for it: Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java === --- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344) +++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy) @@ -499,6 +499,17 @@ // reboot the writer on the new index and get a new searcher solrCore.getUpdateHandler().newIndexWriter(); + Future[] waitSearcher = new Future[1]; + solrCore.getSearcher(true, false, waitSearcher, true); + if (waitSearcher[0] != null) { +try { + waitSearcher[0].get(); + } catch (InterruptedException e) { + SolrException.log(LOG,e); + } catch (ExecutionException e) { + SolrException.log(LOG,e); + } + } // update our commit point to the right dir solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false)); That should allow the searcher that the following commit command prompts to see the *new* IndexWriter. On Feb 22, 2012, at 10:56 AM, eks dev wrote: We started observing strange failures from ReplicationHandler when we commit on master trunk version 4-5 days old. It works sometimes, and sometimes not didn't dig deeper yet. Looks like the real culprit hides behind: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Looks familiar to somebody? 120222 154959 SEVERE SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043) at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source) at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348) at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815) at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095) ... 15 more - Mark Miller lucidimagination.com
Re: SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
thanks Mark, I will give it a go and report back... On Thu, Feb 23, 2012 at 1:31 AM, Mark Miller markrmil...@gmail.com wrote: Looks like an issue around replication IndexWriter reboot, soft commits and hard commits. I think I've got a workaround for it: Index: solr/core/src/java/org/apache/solr/handler/SnapPuller.java === --- solr/core/src/java/org/apache/solr/handler/SnapPuller.java (revision 1292344) +++ solr/core/src/java/org/apache/solr/handler/SnapPuller.java (working copy) @@ -499,6 +499,17 @@ // reboot the writer on the new index and get a new searcher solrCore.getUpdateHandler().newIndexWriter(); + Future[] waitSearcher = new Future[1]; + solrCore.getSearcher(true, false, waitSearcher, true); + if (waitSearcher[0] != null) { + try { + waitSearcher[0].get(); + } catch (InterruptedException e) { + SolrException.log(LOG,e); + } catch (ExecutionException e) { + SolrException.log(LOG,e); + } + } // update our commit point to the right dir solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false)); That should allow the searcher that the following commit command prompts to see the *new* IndexWriter. On Feb 22, 2012, at 10:56 AM, eks dev wrote: We started observing strange failures from ReplicationHandler when we commit on master trunk version 4-5 days old. It works sometimes, and sometimes not didn't dig deeper yet. Looks like the real culprit hides behind: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Looks familiar to somebody? 120222 154959 SEVERE SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1043) at org.apache.solr.update.DirectUpdateHandler2.commit(Unknown Source) at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:503) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:348) at org.apache.solr.handler.ReplicationHandler.doFetch(Unknown Source) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:810) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:815) at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:3984) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:254) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:233) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:223) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1095) ... 15 more - Mark Miller lucidimagination.com