[ https://issues.apache.org/jira/browse/SOLR-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edwin Khodabakchian updated SOLR-2329: -------------------------------------- Attachment: (was: solrconfig_slave.xml) > old index files not deleted on slave > ------------------------------------ > > Key: SOLR-2329 > URL: https://issues.apache.org/jira/browse/SOLR-2329 > Project: Solr > Issue Type: Bug > Components: replication (java) > Affects Versions: 4.0-ALPHA > Environment: centos 5.5 > ext3 file system > Reporter: Edwin Khodabakchian > Attachments: solrconfig.xml > > > I have set up index replication (triggered on optimize). The problem I > am having is the old index files are not being deleted on the slave. > After each replication, I can see the old files still hanging around > as well as the files that have just been pulled. This causes the data > directory size to increase by the index size every replication until > the disk fills up. > I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup > is pretty vanilla. I can reproduce this on multiple slaves. > Checking the logs, I see the following error: > SEVERE: SnapPull failed > org.apache.solr.common.SolrException: Index fetch failed : > at > org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265) > at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.lucene.store.LockObtainFailedException: Lock > obtain timed out: > NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock > at org.apache.lucene.store.Lock.obtain(Lock.java:84) > at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065) > at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954) > at > org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192) > at > org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99) > at > org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) > at > org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376) > at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471) > at > org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319) > ... 11 more > lsof reveals that the file is still opened from the java process. > Contents of the index data dir: > master: > -rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt > -rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii > -rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx > -rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji > -rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt > -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx > -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt > -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx > -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq > -rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm > slave: > -rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen > -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt > -rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii > -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx > -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq > -rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj > -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm > -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 > lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock > -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt > -rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii > -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx > -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe > -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm > -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt > -rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii > -rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx > -rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq > -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm > -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv > -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt > -rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii > -rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx > -rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx > -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt > -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx > -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt > -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx > -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org