Hi, I ran into this problem again the other night. I've looked through my log files in more detail, and nothing seems out of place (I stripped user queries out and included it below). I have the following setup: 1. Indexer has 2 cores. One core gets incremental updates, the other is for full re-syncs with a database. The last step in my full re-sync process is to swap cores (so that the searchers don't have to change their replication master URLs). 2. Searcher that is subscribed to a constant indexer URL.
I noticed this replication error occurred right after I swapped my indexer's cores. Since the index version and generation numbers are independent across the 2 cores, could the searcher's index clean up be pre-emptively deleting the active searcher index? When the error occurred, index.20100921053730 did not exist, but index.properties was pointing to it. Previous entries in the log make it seem like the directory did exist a few minutes earlier (replication + warmup succeeded pointing at that directory). I've tried to reproduce this in a development environment, but haven't been able to so far. https://issues.apache.org/jira/browse/SOLR-1822?focusedCommentId=12845175 SOLR-1822 seems to address a similar issue. I suspect that it would solve what I'm seeing, but it treats the symptom rather than the cause (and I'd like to be able to repro before trying it). Any insight/theories are appreciated. Thanks, Wojtek Sep 21, 2010 5:35:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave in sync with master. Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Master's version: 1271723727936, generation: 18616 Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave's version: 1271723727935, generation: 18615 Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Starting replication process Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Number of files in latest index in master: 118 Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller downloadIndexFiles INFO: Skipping download for /solr/data/index.20100921053730/_13n9.prx Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller downloadIndexFiles INFO: Skipping download for /solr/data/index.20100921053730/_13nx.fnm ... Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller downloadIndexFiles INFO: Skipping download for /solr/data/index.20100921053730/_13m5.fnm ... Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller downloadIndexFiles INFO: Skipping download for /solr/data/index.20100921053730/_13n9.frq Sep 21, 2010 5:37:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Total time taken for download : 0 secs Sep 21, 2010 5:37:31 PM org.apache.solr.update.DirectUpdateHandler2 __AW_commit INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false) Sep 21, 2010 5:37:31 PM org.apache.solr.search.SolrIndexSearcher <init> INFO: Opening searc...@61080339 main Sep 21, 2010 5:37:31 PM org.apache.solr.update.DirectUpdateHandler2 __AW_commit INFO: end_commit_flush Sep 21, 2010 5:37:31 PM org.apache.solr.search.SolrIndexSearcher __AW_warm INFO: autowarming searc...@61080339 main from searc...@26aebd8c main fieldValueCache{lookups=866,hits=866,hitratio=1.00,inserts=0,evictions=0,size=11,warmupTime=0,cumulative_lookups=493365,cumulative_hits=493351,cumulative_hitratio=0.99,cumulative_inserts=7,cumulative_evictions=0,item_FeaturesFacet={field=FeaturesFacet,memSize=51896798,tindexSize=56,time=988,phase1=936,nTerms=50,bigTerms=9,termInstances=5403271,uses=146},...} ... Sep 21, 2010 5:37:31 PM org.apache.solr.search.SolrIndexSearcher __AW_warm INFO: autowarming result for searc...@61080339 main documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2036931,cumulative_hits=836191,cumulative_hitratio=0.41,cumulative_inserts=1200740,cumulative_evictions=1103563} Sep 21, 2010 5:37:31 PM org.apache.solr.core.QuerySenderListener __AW_newSearcher INFO: QuerySenderListener sending requests to searc...@61080339 main Sep 21, 2010 5:37:31 PM org.apache.solr.request.UnInvertedField uninvert INFO: UnInverted multi-valued field {field=BedFacet,memSize=48178130,tindexSize=42,time=313,phase1=261,nTerms=6,bigTerms=4,termInstances=328351,uses=0} ... INFO: [] webapp=null path=null params={*:*} hits=11546888 status=0 QTime=20687 Sep 21, 2010 5:37:58 PM org.apache.solr.core.QuerySenderListener __AW_newSearcher INFO: QuerySenderListener done. Sep 21, 2010 5:37:58 PM org.apache.solr.core.SolrCore registerSearcher INFO: [] Registered new searcher searc...@61080339 main Sep 21, 2010 5:37:58 PM org.apache.solr.search.SolrIndexSearcher __AW_close INFO: Closing searc...@26aebd8c main fieldValueCache{lookups=950,hits=950,hitratio=1.00,inserts=0,evictions=0,size=11,warmupTime=0,cumulative_lookups=493449,cumulative_hits=493435,cumulative_hitratio=0.99,cumulative_inserts=7,cumulative_evictions=0,item_FeaturesFacet={field=FeaturesFacet,memSize=51896798,tindexSize=56,time=988,phase1=936,nTerms=50,bigTerms=9,termInstances=5403271,uses=160},...} Sep 21, 2010 5:37:58 PM org.apache.solr.handler.SnapPuller doCommit INFO: Force open index writer to make sure older index files get deleted Sep 21, 2010 5:37:58 PM org.apache.solr.core.SolrDeletionPolicy onInit INFO: SolrDeletionPolicy.onInit: commits:num=2 commit{dir=/solr/data/index.20100921053730,segFN=segments_ed3,version=1271723727935,generation=18615,filenames=[_13n9.prx, _13o2.fdt, _13nu_8.del, _13n9_q.del, _13nx.fnm, _13hd.prx, _13nv.tii, _13o0.frq, _13m5.frq, _13ny.fnm, _13n9.fnm, _13o0.prx, _13nv.tis, _13hd.nrm, _13nv_7.del, _13o2.fdx, _13o2.frq, _13nx.prx, _13o0.tii, _13ny.nrm, _13o2.tii, _13nu.prx, _13m5.fdt, _13o0.tis, _13o2.tis, _13m5.fdx, _13nx.nrm, _13m5_1m.del, _13o1_2.del, _13nv.frq, _13n9.fdx, _13nz.fnm, _13ny.prx, _13o2_1.del, _13o1.fnm, _13nv.fdx, _13n9.fdt, _13nv.fdt, _13nu.frq, _13nz.tis, _13nw.fnm, _13ny_5.del, _13nz.tii, _13o0.fdx, _13o2.fnm, _13hd.fnm, _13o0.fdt, _13hd.fdt, _13ny.tii, _13ny.tis, _13hd.fdx, _13nw_6.del, _13nv.fnm, _13m5.fnm, _13nz.fdx, _13o2.prx, _13ny.frq, _13m5.tis, _13nz_4.del, _13nz.fdt, _13n9.tis, _13m5.tii, _13hd.tis, _13nz.nrm, _13hd.tii, _13nx.frq, _13o0.fnm, _13n9.tii, _13nx.fdt, _13nu.tii, _13nx.fdx, _13nz.prx, _13o1.tis, _13m5.nrm, _13m5.prx, _13hd_4b.del, _13ny.fdx, _13nv.prx, _13nu.tis, _13nz.frq, _13o0_3.del, _13o2.nrm, _13ny.fdt, _13o1.tii, _13o1.frq, _13nv.nrm, _13nx.tii, segments_ed3, _13nu.fdx, _13nx.tis, _13nu.fdt, _13n9.nrm, _13o1.fdt, _13o1.fdx, _13o1.prx, _13nu.nrm, _13nw.prx, _13nw.frq, _13o0.nrm, _13nw.fdx, _13nw.fdt, _13nw.tii, _13nw.tis, _13nu.fnm, _13nw.nrm, _13o1.nrm, _13nx_6.del, _13hd.frq, _13n9.frq] commit{dir=/solr/data/index.20100921053730,segFN=segments_ed4,version=1271723727936,generation=18616,filenames=[_13n9.prx, _13o2.fdt, _13nx.fnm, _13hd.prx, _13nv.tii, _13o0.frq, _13m5.frq, _13o3.fdt, _13ny.fnm, _13nx_7.del, _13n9.fnm, _13o0.prx, _13nv.tis, _13hd.nrm, _13o3.fdx, _13o2.fdx, _13o2.frq, _13nx.prx, _13o0.tii, _13ny.nrm, _13o3_1.del, _13o2.tii, _13o3.prx, _13o1_3.del, _13nu.prx, _13m5.fdt, _13o0.tis, _13o2.tis, _13m5.fdx, _13nx.nrm, _13o3.nrm, _13nu_9.del, _13nv.frq, _13n9.fdx, _13nz.fnm, _13ny.prx, _13o3.tii, _13nw_7.del, _13o1.fnm, _13nv.fdx, _13n9.fdt, _13nv.fdt, _13nu.frq, _13nz.tis, _13nw.fnm, _13o3.tis, _13nz.tii, _13o0.fdx, _13o2.fnm, _13hd.fnm, _13o0.fdt, _13hd.fdt, _13ny.tii, _13ny.tis, _13o0_4.del, _13hd.fdx, _13o3.fnm, _13nv.fnm, _13m5.fnm, _13nz.fdx, _13o2.prx, _13ny.frq, _13m5.tis, _13o2_2.del, _13nv_8.del, _13nz_4.del, _13nz.fdt, _13n9.tis, _13m5.tii, _13hd.tis, _13nz.nrm, _13hd.tii, _13nx.frq, _13o0.fnm, _13n9.tii, _13nx.fdt, _13nu.tii, _13nx.fdx, _13nz.prx, _13o1.tis, _13m5.nrm, _13m5.prx, _13ny.fdx, _13nv.prx, _13nu.tis, _13nz.frq, _13o2.nrm, _13ny.fdt, _13o1.tii, _13o1.frq, _13nv.nrm, _13nx.tii, segments_ed4, _13nu.fdx, _13nx.tis, _13nu.fdt, _13hd_4c.del, _13n9.nrm, _13o1.fdt, _13o1.fdx, _13o1.prx, _13nu.nrm, _13nw.prx, _13n9_r.del, _13nw.frq, _13o0.nrm, _13nw.fdx, _13nw.fdt, _13ny_6.del, _13o3.frq, _13nw.tii, _13m5_1n.del, _13nw.tis, _13nu.fnm, _13nw.nrm, _13o1.nrm, _13hd.frq, _13n9.frq] Sep 21, 2010 5:37:58 PM org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: newest commit = 1271723727936 Sep 21, 2010 5:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave in sync with master. Sep 21, 2010 5:42:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Master's version: 1271723727937, generation: 18617 Sep 21, 2010 5:42:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave's version: 1271723727936, generation: 18616 Sep 21, 2010 5:42:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Starting replication process Sep 21, 2010 5:42:30 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Number of files in latest index in master: 45 Sep 21, 2010 5:56:29 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Total time taken for download : 839 secs Sep 21, 2010 5:56:29 PM org.apache.solr.handler.SnapPuller copyAFile SEVERE: Unable to move index file from: /solr/data/index.20100921054230/_13m5.fnm to: /solr/data/index.20100921053730/_13m5.fnm Sep 21, 2010 5:56:36 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Master's version: 1271723727940, generation: 18620 Sep 21, 2010 5:56:36 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave's version: 1271723727936, generation: 18616 Sep 21, 2010 5:56:36 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Starting replication process Sep 21, 2010 5:56:36 PM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Number of files in latest index in master: 73 -- View this message in context: http://lucene.472066.n3.nabble.com/SEVERE-Unable-to-move-index-file-tp812915p1609827.html Sent from the Solr - User mailing list archive at Nabble.com.
