Can IW.waitForMerges be called between 'prepareCommit' and 'commit'? That's when the app calls 'getReader' to create external data.
Peter On Fri, Feb 26, 2010 at 3:15 PM, Peter Keegan <peterlkee...@gmail.com>wrote: > Great, I'll give it a try. > Thanks! > > > On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> Note that it's a BG merge (not commit)... >> >> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API? >> If you call that, then call .getReader().getVersion(), then close & >> open the writer, I think (but you better test to be sure!) the next >> .getReader().getVersion() should always match. >> >> Mike >> >> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <peterlkee...@gmail.com> >> wrote: >> > Is there a way for the application to wait for the BG commit to finish >> > before it calls IW.close? If so, would this prevent the extra version? >> The >> > extra version causes the app. to think that the external data it >> committed >> > is out of synch with the index, which requires the app to do extra >> > processing to re-synch. >> > >> > Thanks, >> > Peter >> > >> > >> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless < >> > luc...@mikemccandless.com> wrote: >> > >> >> OK -- I can now see what happened. >> >> >> >> There was a merge still running, when you called IW.commit (Lucene >> >> Merge Thread #0). Because IW.commit does not wait for BG merges to >> >> finish, but IW.close does (by default), this means you'll pick up an >> >> extra version whenever a merge is running when you call close. >> >> >> >> Mike >> >> >> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <peterlkee...@gmail.com> >> >> wrote: >> >> > I'm pretty sure this output occurred when the version number skipped >> +1. >> >> > The line containing '============'. separates the close/open >> IndexWriter. >> >> > >> >> > IFD [Indexer]: setInfoStream >> >> > >> >> >> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@646f9dd9 >> >> > IW 9 [Indexer]: setInfoStream: >> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated >> >> > pathname>\lresumes1.search.main.1 autoCommit=false >> >> > >> >> >> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@5be44512mergescheduler >> >> >> =org.apache.lucene.index.concurrentmergeschedu...@6772cfdframbuffersizemb >> >> =16.0 >> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 >> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b >> >> _d:C1220->_d >> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >> >> > IW 9 [Indexer]: flush at getReader >> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816 >> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16 >> >> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j >> >> > docStoreOffset=0 flushDocs=true flushDeletes=false >> flushDocStores=false >> >> > numDocs=1456 numBufDelTerms=1456 >> >> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b >> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f >> >> _h:C1291->_h >> >> > _i:C703->_h >> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j >> numDocs=1456 >> >> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704 >> >> newFlushedSize=4969789 >> >> > docs/MB=307.202 new/old=29.966% >> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ; >> >> > isCommit = false] >> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ; >> >> > isCommit = false] >> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments >> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 >> segments >> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 >> segments >> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge >> >> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b >> >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j >> >> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return >> >> > IW 9 [Indexer]: prepareCommit: flush >> >> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j >> >> docStoreOffset=1456 >> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509 >> >> > numBufDelTerms=509 >> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j >> >> > IW 9 [Indexer]: flush shared docStore segment _j >> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j >> >> > numDocs=1965 >> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509 >> >> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970 >> >> > docs/MB=287.727 new/old=24.788% >> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = >> >> false] >> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted >> >> docIDs >> >> > and 0 deleted queries on 11 segments. >> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = >> >> false] >> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments >> >> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments >> >> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments >> >> > IW 9 [Indexer]: LMP: 1 to 11: add this merge >> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j [total 1 pending] >> >> > IW 9 [Indexer]: CMS: now merge >> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b >> >> _d:C1220->_d >> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >> >> _j:C1456->_j >> >> > _k:C509->_j >> >> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b >> >> _d:C1220->_d >> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >> >> _j:C1456->_j >> >> > _k:C509->_j into _l [mergeDocStores] >> >> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0] >> >> > IW 9 [Indexer]: CMS: no more merges pending; now return >> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0 >> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j changeCount=7 >> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start >> >> > IW 9 [Indexer]: now sync _k.fnm >> >> > IW 9 [Lucene Merge Thread #0]: now merge >> >> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f >> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l >> >> > [mergeDocStores] >> >> > merge=org.apache.lucene.index.mergepolicy$oneme...@5203ef34 >> >> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d >> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j >> >> _k:C509->_j >> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b >> >> _d:C1220->_d >> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >> >> _j:C1456->_j >> >> > _k:C509->_j into _l [mergeDocStores] >> >> > IW 9 [Indexer]: now sync _k.tii >> >> > IW 9 [Indexer]: now sync _k.tis >> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs >> >> > IW 9 [Indexer]: now sync _j.tis >> >> > IW 9 [Indexer]: now sync _j.tii >> >> > IW 9 [Indexer]: now sync _j.fdx >> >> > IW 9 [Indexer]: now sync _j.fdt >> >> > IW 9 [Indexer]: now sync _k.prx >> >> > IW 9 [Indexer]: now sync _k.nrm >> >> > IW 9 [Indexer]: now sync _j.nrm >> >> > IW 9 [Indexer]: now sync _j.prx >> >> > IW 9 [Indexer]: now sync _j.fnm >> >> > IW 9 [Indexer]: now sync _k.frq >> >> > IW 9 [Indexer]: now sync _j.frq >> >> > IW 9 [Indexer]: done all syncs >> >> > IW 9 [Indexer]: flush at getReader >> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j >> >> > IW 9 [Indexer]: flush shared docStore segment _k >> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k >> >> numDocs=0 >> >> > IW 9 [Indexer]: commit: start >> >> > IW 9 [Indexer]: commit: already prepared >> >> > IW 9 [Indexer]: commit: pendingCommit != null >> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c" >> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit = >> >> true] >> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b" >> >> > IFD [Indexer]: delete "segments_b" >> >> > IW 9 [Indexer]: commit: done >> >> > IW 9 [Indexer]: flush at getReader >> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j >> >> > IW 9 [Indexer]: now flush at close >> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j >> >> > IW 9 [Indexer]: CMS: now merge >> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b >> >> _d:C1220->_d >> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >> >> _j:C1456->_j >> >> > _k:C509->_j >> >> > IW 9 [Indexer]: CMS: no more merges pending; now return >> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780 >> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f >> >> _g:C858->_f >> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j >> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b >> >> _c:C717->_b >> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >> >> _i:C703->_h >> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] >> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments >> ; >> >> > isCommit = false] >> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments >> ; >> >> > isCommit = false] >> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done >> >> > IW 9 [Indexer]: now call final commit() >> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0 >> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8 >> >> > IW 9 [Indexer]: now sync _l.nrm >> >> > IW 9 [Indexer]: now sync _l.fnm >> >> > IW 9 [Indexer]: now sync _l.prx >> >> > IW 9 [Indexer]: now sync _l.frq >> >> > IW 9 [Indexer]: now sync _l.tis >> >> > IW 9 [Indexer]: now sync _l.fdx >> >> > IW 9 [Indexer]: now sync _l.fdt >> >> > IW 9 [Indexer]: now sync _l.tii >> >> > IW 9 [Indexer]: done all syncs >> >> > IW 9 [Indexer]: commit: pendingCommit != null >> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d" >> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = >> true] >> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c" >> >> > IFD [Indexer]: delete "_h.prx" >> >> > IFD [Indexer]: delete "_f.tis" >> >> > IFD [Indexer]: delete "_g.tis" >> >> > IFD [Indexer]: delete "_d.tii" >> >> > IFD [Indexer]: delete "_e.frq" >> >> > IFD [Indexer]: delete "_k.fnm" >> >> > IFD [Indexer]: delete "_c.tii" >> >> > IFD [Indexer]: delete "_h.nrm" >> >> > IFD [Indexer]: delete "_i.tis" >> >> > IFD [Indexer]: delete "_k.tii" >> >> > IFD [Indexer]: delete "_h.fdt" >> >> > IFD [Indexer]: delete "_c.tis" >> >> > IFD [Indexer]: delete "_g.tii" >> >> > IFD [Indexer]: delete "_h.fdx" >> >> > IFD [Indexer]: delete "_c.nrm" >> >> > IFD [Indexer]: delete "_i.tii" >> >> > IFD [Indexer]: delete "_d.tis" >> >> > IFD [Indexer]: delete "_f.tii" >> >> > IFD [Indexer]: delete "_e.fnm" >> >> > IFD [Indexer]: delete "_k.tis" >> >> > IFD [Indexer]: delete "_d.frq" >> >> > IFD [Indexer]: delete "_f.nrm" >> >> > IFD [Indexer]: delete "_h.fnm" >> >> > IFD [Indexer]: delete "segments_c" >> >> > IFD [Indexer]: delete "_c.prx" >> >> > IFD [Indexer]: delete "_d.fnm" >> >> > IFD [Indexer]: delete "_h.frq" >> >> > IFD [Indexer]: delete "_h.tis" >> >> > IFD [Indexer]: delete "_f.fdt" >> >> > IFD [Indexer]: delete "_j.tis" >> >> > IFD [Indexer]: delete "_f.fdx" >> >> > IFD [Indexer]: delete "_g.nrm" >> >> > IFD [Indexer]: delete "_f.prx" >> >> > IFD [Indexer]: delete "_j.tii" >> >> > IFD [Indexer]: delete "_h.tii" >> >> > IFD [Indexer]: delete "_e.prx" >> >> > IFD [Indexer]: delete "_c.frq" >> >> > IFD [Indexer]: delete "_i.prx" >> >> > IFD [Indexer]: delete "_j.fdx" >> >> > IFD [Indexer]: delete "_i.nrm" >> >> > IFD [Indexer]: delete "_j.fdt" >> >> > IFD [Indexer]: delete "_k.prx" >> >> > IFD [Indexer]: delete "_b.nrm" >> >> > IFD [Indexer]: delete "_e.tis" >> >> > IFD [Indexer]: delete "_e.tii" >> >> > IFD [Indexer]: delete "_g.fnm" >> >> > IFD [Indexer]: delete "_k.nrm" >> >> > IFD [Indexer]: delete "_j.nrm" >> >> > IFD [Indexer]: delete "_j.prx" >> >> > IFD [Indexer]: delete "_e.nrm" >> >> > IFD [Indexer]: delete "_g.prx" >> >> > IFD [Indexer]: delete "_b.prx" >> >> > IFD [Indexer]: delete "_i.frq" >> >> > IFD [Indexer]: delete "_b.tis" >> >> > IFD [Indexer]: delete "_d.prx" >> >> > IFD [Indexer]: delete "_g.frq" >> >> > IFD [Indexer]: delete "_d.fdx" >> >> > IFD [Indexer]: delete "_b.tii" >> >> > IFD [Indexer]: delete "_b.frq" >> >> > IFD [Indexer]: delete "_d.fdt" >> >> > IFD [Indexer]: delete "_f.fnm" >> >> > IFD [Indexer]: delete "_j.fnm" >> >> > IFD [Indexer]: delete "_b.fnm" >> >> > IFD [Indexer]: delete "_d.nrm" >> >> > IFD [Indexer]: delete "_b.fdt" >> >> > IFD [Indexer]: delete "_k.frq" >> >> > IFD [Indexer]: delete "_c.fnm" >> >> > IFD [Indexer]: delete "_j.frq" >> >> > IFD [Indexer]: delete "_i.fnm" >> >> > IFD [Indexer]: delete "_f.frq" >> >> > IFD [Indexer]: delete "_b.fdx" >> >> > IW 9 [Indexer]: commit: done >> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909 >> >> > >> >> > ===================================================== >> >> > >> >> > IFD [Indexer]: setInfoStream >> >> > >> >> >> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@68d349c1 >> >> > IW 10 [Indexer]: setInfoStream: >> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated >> >> > pathname>\lresumes1.search.main.1 autoCommit=false >> >> > >> >> >> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@78f53abmergescheduler >> >> >> =org.apache.lucene.index.concurrentmergeschedu...@52be036drambuffersizemb >> >> =16.0 >> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 >> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909 >> >> > IW 10 [Indexer]: flush at getReader >> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 >> >> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789 >> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16 >> >> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m >> >> > docStoreOffset=0 flushDocs=true flushDeletes=false >> flushDocStores=false >> >> > numDocs=1526 numBufDelTerms=1526 >> >> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909 >> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m >> >> numDocs=1526 >> >> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032 >> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925% >> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ; >> >> > isCommit = false] >> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ; >> >> > isCommit = false] >> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments >> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2 >> >> segments >> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1 >> >> segments >> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge >> >> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 >> _m:C1526->_m >> >> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now >> return >> >> > IW 10 [Indexer]: prepareCommit: flush >> >> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m >> >> docStoreOffset=1526 >> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501 >> >> > numBufDelTerms=501 >> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >> >> > IW 10 [Indexer]: flush shared docStore segment _m >> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m >> >> > numDocs=2027 >> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501 >> >> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342 >> >> > docs/MB=316.593 new/old=23.953% >> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = >> >> false] >> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted >> >> docIDs >> >> > and 0 deleted queries on 4 segments. >> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = >> >> false] >> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments >> >> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments >> >> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments >> >> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments >> >> > IW 10 [Indexer]: CMS: now merge >> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m >> _n:C501->_m >> >> > IW 10 [Indexer]: CMS: no more merges pending; now return >> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0 >> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m >> >> > _n:C501->_m changeCount=6 >> >> > IW 10 [Indexer]: now sync _m.frq >> >> > IW 10 [Indexer]: now sync _m.prx >> >> > IW 10 [Indexer]: now sync _n.tis >> >> > IW 10 [Indexer]: now sync _m.tis >> >> > IW 10 [Indexer]: now sync _n.frq >> >> > IW 10 [Indexer]: now sync _n.tii >> >> > IW 10 [Indexer]: now sync _m.tii >> >> > IW 10 [Indexer]: now sync _m.fnm >> >> > IW 10 [Indexer]: now sync _n.fnm >> >> > IW 10 [Indexer]: now sync _n.nrm >> >> > IW 10 [Indexer]: now sync _n.prx >> >> > IW 10 [Indexer]: now sync _m.nrm >> >> > IW 10 [Indexer]: now sync _m.fdt >> >> > IW 10 [Indexer]: now sync _m.fdx >> >> > IW 10 [Indexer]: done all syncs >> >> > IW 10 [Indexer]: flush at getReader >> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >> >> > _n:C501->_m >> >> > IW 10 [Indexer]: flush shared docStore segment _n >> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n >> >> numDocs=0 >> >> > IW 10 [Indexer]: commit: start >> >> > IW 10 [Indexer]: commit: already prepared >> >> > IW 10 [Indexer]: commit: pendingCommit != null >> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e" >> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = >> true] >> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d" >> >> > IFD [Indexer]: delete "segments_d" >> >> > IW 10 [Indexer]: commit: done >> >> > IW 10 [Indexer]: flush at getReader >> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >> >> > _n:C501->_m >> >> > IW 10 [Indexer]: now flush at close >> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >> >> docStoreOffset=0 >> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >> >> > numBufDelTerms=0 >> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >> >> > _n:C501->_m >> >> > IW 10 [Indexer]: CMS: now merge >> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m >> _n:C501->_m >> >> > IW 10 [Indexer]: CMS: no more merges pending; now return >> >> > IW 10 [Indexer]: now call final commit() >> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0 >> >> > IW 10 [Indexer]: skip startCommit(): no changes pending >> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip >> >> > IW 10 [Indexer]: commit: done >> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m >> >> > >> >> > >> >> > Peter >> >> > >> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless < >> >> > luc...@mikemccandless.com> wrote: >> >> > >> >> >> Do you know the place in the infoStream output where you got a >> reader >> >> >> with the wrong (unexplained extra +1) version? If so, can you post >> >> >> the infoStream output up to that point? >> >> >> >> >> >> Mike >> >> >> >> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan < >> peterlkee...@gmail.com> >> >> >> wrote: >> >> >> > I've reproduced this and I have a bunch of infoStream log files. >> Since >> >> >> the >> >> >> > messages have no timestamps, it's hard to tell where the relevant >> >> entries >> >> >> > are. What should I be looking for? >> >> >> > >> >> >> > Peter >> >> >> > >> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan < >> peterlkee...@gmail.com >> >> >> >wrote: >> >> >> > >> >> >> >> I'm pretty sure there are flushes and segment merges going on, >> but as >> >> >> you >> >> >> >> said, that shouldn't affect the version increment. I'll see what >> I >> >> can >> >> >> do to >> >> >> >> get infoStream output. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Peter >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless < >> >> >> >> luc...@mikemccandless.com> wrote: >> >> >> >> >> >> >> >>> Well I'm at a loss then. The version should only increment on >> >> commit. >> >> >> >>> >> >> >> >>> Can you make it all happen when infoStream is on, and post back? >> >> >> >>> >> >> >> >>> Mike >> >> >> >>> >> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan < >> >> peterlkee...@gmail.com >> >> >> > >> >> >> >>> wrote: >> >> >> >>> > Only one writer thread and one writer process. >> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean >> create, >> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false. >> >> >> >>> > >> >> >> >>> > Peter >> >> >> >>> > >> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless < >> >> >> >>> > luc...@mikemccandless.com> wrote: >> >> >> >>> > >> >> >> >>> >> That's curious. >> >> >> >>> >> >> >> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first >> >> prepare, >> >> >> >>> >> since that will call prepareCommit internally) that this >> version >> >> >> >>> >> should increase. >> >> >> >>> >> >> >> >> >>> >> Is there only 1 thread doing this? >> >> >> >>> >> >> >> >> >>> >> Oh, and, are you passing false for autoCommit? >> >> >> >>> >> >> >> >> >>> >> Mike >> >> >> >>> >> >> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan < >> >> >> peterlkee...@gmail.com >> >> >> >>> > >> >> >> >>> >> wrote: >> >> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which >> gets >> >> >> >>> repeated >> >> >> >>> >> at >> >> >> >>> >> > regular intervals: >> >> >> >>> >> > >> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File); >> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory()); >> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, >> false, >> >> >> >>> >> maxFieldLen) >> >> >> >>> >> > 4. writer.getReader().getVersion(); >> >> >> >>> >> > 5. writer.prepareCommit(); >> >> >> >>> >> > 6. writer.getReader().getVersion(); >> >> >> >>> >> > 7. writer.commit(); >> >> >> >>> >> > 8. writer.close(); >> >> >> >>> >> > >> >> >> >>> >> > I'm using the version number to keep external data in synch >> >> with >> >> >> the >> >> >> >>> >> index. >> >> >> >>> >> > Usually, the version number from (6) is 1 greater than from >> (4) >> >> >> and >> >> >> >>> the >> >> >> >>> >> > version from (4) equals the version from the previous (6). >> At >> >> >> least >> >> >> >>> once >> >> >> >>> >> a >> >> >> >>> >> > day, however, the version from (4) is 1 greater than from >> the >> >> >> >>> previous >> >> >> >>> >> (6). >> >> >> >>> >> > What would explain this sporadic behavior of version >> numbers? >> >> >> >>> >> > >> >> >> >>> >> > Thanks, >> >> >> >>> >> > Peter >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> >>> >> To unsubscribe, e-mail: >> java-user-unsubscr...@lucene.apache.org >> >> >> >>> >> For additional commands, e-mail: >> >> java-user-h...@lucene.apache.org >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> > >> >> >> >>> >> >> >> >>> >> >> --------------------------------------------------------------------- >> >> >> >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> >> >>> For additional commands, e-mail: >> java-user-h...@lucene.apache.org >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> >> >> >> >> >> > >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >
