I see the file -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
was created on Dec. 15. At the end of the replication, as far as I remember, the SnapPuller tries to open the writer to ensure the old files are deleted, and in your case it cannot obtain a lock on the index folder on Dec 16, 17,18. Can you reproduce the problem if you delete the lock file, restart the slave and try replication again? Do you have any other Writer(s) open for this folder outside of this core? -Alexander On Sat, Jan 22, 2011 at 3:52 PM, feedly team <feedly...@gmail.com> wrote: > The file system checked out, I also tried creating a slave on a > different machine and could reproduce the issue. I logged SOLR-2329. > > On Sat, Dec 18, 2010 at 8:01 PM, Lance Norskog <goks...@gmail.com> wrote: >> This could be a quirk of the native locking feature. What's the file >> system? Can you fsck it? >> >> If this error keeps happening, please file this. It should not happen. >> Add the text above and also your solrconfigs if you can. >> >> One thing you could try is to change from the native locking policy to >> the simple locking policy - but only on the child. >> >> On Sat, Dec 18, 2010 at 4:44 PM, feedly team <feedly...@gmail.com> wrote: >>> I have set up index replication (triggered on optimize). The problem I >>> am having is the old index files are not being deleted on the slave. >>> After each replication, I can see the old files still hanging around >>> as well as the files that have just been pulled. This causes the data >>> directory size to increase by the index size every replication until >>> the disk fills up. >>> >>> Checking the logs, I see the following error: >>> >>> SEVERE: SnapPull failed >>> org.apache.solr.common.SolrException: Index fetch failed : >>> at >>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329) >>> at >>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265) >>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock >>> obtain timed out: >>> NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >>> at org.apache.lucene.store.Lock.obtain(Lock.java:84) >>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065) >>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954) >>> at >>> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192) >>> at >>> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99) >>> at >>> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) >>> at >>> org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376) >>> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471) >>> at >>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319) >>> ... 11 more >>> >>> lsof reveals that the file is still opened from the java process. >>> >>> I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup >>> is pretty vanilla. The OS is linux, the indexes are on local >>> directories, write permissions look ok, nothing unusual in the config >>> (default deletion policy, etc.). Contents of the index data dir: >>> >>> master: >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt >>> -rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii >>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx >>> -rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >>> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >>> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >>> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >>> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >>> -rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >>> >>> slave: >>> -rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt >>> -rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii >>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx >>> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq >>> -rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm >>> -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 >>> lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt >>> -rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii >>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx >>> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt >>> -rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii >>> -rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx >>> -rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt >>> -rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii >>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx >>> -rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >>> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >>> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >>> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >>> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >>> >>> >>> Any pointers on how to proceed? Thanks. >>> >> >> >> >> -- >> Lance Norskog >> goks...@gmail.com >> >
