The file system checked out, I also tried creating a slave on a different machine and could reproduce the issue. I logged SOLR-2329.
On Sat, Dec 18, 2010 at 8:01 PM, Lance Norskog <goks...@gmail.com> wrote: > This could be a quirk of the native locking feature. What's the file > system? Can you fsck it? > > If this error keeps happening, please file this. It should not happen. > Add the text above and also your solrconfigs if you can. > > One thing you could try is to change from the native locking policy to > the simple locking policy - but only on the child. > > On Sat, Dec 18, 2010 at 4:44 PM, feedly team <feedly...@gmail.com> wrote: >> I have set up index replication (triggered on optimize). The problem I >> am having is the old index files are not being deleted on the slave. >> After each replication, I can see the old files still hanging around >> as well as the files that have just been pulled. This causes the data >> directory size to increase by the index size every replication until >> the disk fills up. >> >> Checking the logs, I see the following error: >> >> SEVERE: SnapPull failed >> org.apache.solr.common.SolrException: Index fetch failed : >> at >> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329) >> at >> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265) >> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock >> obtain timed out: >> NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >> at org.apache.lucene.store.Lock.obtain(Lock.java:84) >> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065) >> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954) >> at >> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192) >> at >> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99) >> at >> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) >> at >> org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376) >> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471) >> at >> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319) >> ... 11 more >> >> lsof reveals that the file is still opened from the java process. >> >> I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup >> is pretty vanilla. The OS is linux, the indexes are on local >> directories, write permissions look ok, nothing unusual in the config >> (default deletion policy, etc.). Contents of the index data dir: >> >> master: >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt >> -rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii >> -rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx >> -rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >> -rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >> >> slave: >> -rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt >> -rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii >> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx >> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq >> -rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm >> -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 >> lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt >> -rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii >> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx >> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt >> -rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii >> -rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx >> -rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt >> -rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii >> -rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx >> -rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >> >> >> Any pointers on how to proceed? Thanks. >> > > > > -- > Lance Norskog > goks...@gmail.com >
