That should be fine! Mike
On Fri, Feb 26, 2010 at 3:26 PM, Peter Keegan <peterlkee...@gmail.com> wrote: > Can IW.waitForMerges be called between 'prepareCommit' and 'commit'? That's > when the app calls 'getReader' to create external data. > > Peter > > On Fri, Feb 26, 2010 at 3:15 PM, Peter Keegan <peterlkee...@gmail.com>wrote: > >> Great, I'll give it a try. >> Thanks! >> >> >> On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless < >> luc...@mikemccandless.com> wrote: >> >>> Note that it's a BG merge (not commit)... >>> >>> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API? >>> If you call that, then call .getReader().getVersion(), then close & >>> open the writer, I think (but you better test to be sure!) the next >>> .getReader().getVersion() should always match. >>> >>> Mike >>> >>> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <peterlkee...@gmail.com> >>> wrote: >>> > Is there a way for the application to wait for the BG commit to finish >>> > before it calls IW.close? If so, would this prevent the extra version? >>> The >>> > extra version causes the app. to think that the external data it >>> committed >>> > is out of synch with the index, which requires the app to do extra >>> > processing to re-synch. >>> > >>> > Thanks, >>> > Peter >>> > >>> > >>> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless < >>> > luc...@mikemccandless.com> wrote: >>> > >>> >> OK -- I can now see what happened. >>> >> >>> >> There was a merge still running, when you called IW.commit (Lucene >>> >> Merge Thread #0). Because IW.commit does not wait for BG merges to >>> >> finish, but IW.close does (by default), this means you'll pick up an >>> >> extra version whenever a merge is running when you call close. >>> >> >>> >> Mike >>> >> >>> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <peterlkee...@gmail.com> >>> >> wrote: >>> >> > I'm pretty sure this output occurred when the version number skipped >>> +1. >>> >> > The line containing '============'. separates the close/open >>> IndexWriter. >>> >> > >>> >> > IFD [Indexer]: setInfoStream >>> >> > >>> >> >>> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@646f9dd9 >>> >> > IW 9 [Indexer]: setInfoStream: >>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated >>> >> > pathname>\lresumes1.search.main.1 autoCommit=false >>> >> > >>> >> >>> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@5be44512mergescheduler >>> >> >>> =org.apache.lucene.index.concurrentmergeschedu...@6772cfdframbuffersizemb >>> >> =16.0 >>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 >>> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b >>> >> _d:C1220->_d >>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >>> >> > IW 9 [Indexer]: flush at getReader >>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >>> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > IW 9 [UpdWriterBuild : 9]: DW: RAM: now flush @ usedMB=15.816 >>> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16 >>> >> > IW 9 [UpdWriterBuild : 9]: flush: segment=_j docStoreSegment=_j >>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false >>> flushDocStores=false >>> >> > numDocs=1456 numBufDelTerms=1456 >>> >> > IW 9 [UpdWriterBuild : 9]: index before flush _a:C9780 _b:C1204->_b >>> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f >>> >> _h:C1291->_h >>> >> > _i:C703->_h >>> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j >>> numDocs=1456 >>> >> > IW 9 [UpdWriterBuild : 9]: DW: oldRAMSize=16584704 >>> >> newFlushedSize=4969789 >>> >> > docs/MB=307.202 new/old=29.966% >>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ; >>> >> > isCommit = false] >>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments ; >>> >> > isCommit = false] >>> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments >>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.863048 to 7.613048: 1 >>> segments >>> >> > IW 9 [UpdWriterBuild : 9]: LMP: level 6.2247195 to 6.696363: 9 >>> segments >>> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge >>> >> > IW 9 [UpdWriterBuild : 9]: CMS: index: _a:C9780 _b:C1204->_b >>> >> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j >>> >> > IW 9 [UpdWriterBuild : 9]: CMS: no more merges pending; now return >>> >> > IW 9 [Indexer]: prepareCommit: flush >>> >> > IW 9 [Indexer]: flush: segment=_k docStoreSegment=_j >>> >> docStoreOffset=1456 >>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509 >>> >> > numBufDelTerms=509 >>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >>> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j >>> >> > IW 9 [Indexer]: flush shared docStore segment _j >>> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment _j >>> >> > numDocs=1965 >>> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509 >>> >> > IW 9 [Indexer]: DW: oldRAMSize=7483392 newFlushedSize=1854970 >>> >> > docs/MB=287.727 new/old=24.788% >>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = >>> >> false] >>> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted >>> >> docIDs >>> >> > and 0 deleted queries on 11 segments. >>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit = >>> >> false] >>> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments >>> >> > IW 9 [Indexer]: LMP: level 6.863048 to 7.613048: 1 segments >>> >> > IW 9 [Indexer]: LMP: level 6.2247195 to 6.696363: 10 segments >>> >> > IW 9 [Indexer]: LMP: 1 to 11: add this merge >>> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j [total 1 pending] >>> >> > IW 9 [Indexer]: CMS: now merge >>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b >>> >> _d:C1220->_d >>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >>> >> _j:C1456->_j >>> >> > _k:C509->_j >>> >> > IW 9 [Indexer]: CMS: consider merge _b:C1204->_b _c:C717->_b >>> >> _d:C1220->_d >>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >>> >> _j:C1456->_j >>> >> > _k:C509->_j into _l [mergeDocStores] >>> >> > IW 9 [Indexer]: CMS: launch new thread [Lucene Merge Thread #0] >>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return >>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0 >>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j changeCount=7 >>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: start >>> >> > IW 9 [Indexer]: now sync _k.fnm >>> >> > IW 9 [Lucene Merge Thread #0]: now merge >>> >> > merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f >>> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j into _l >>> >> > [mergeDocStores] >>> >> > merge=org.apache.lucene.index.mergepolicy$oneme...@5203ef34 >>> >> > index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d >>> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j >>> >> _k:C509->_j >>> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b >>> >> _d:C1220->_d >>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >>> >> _j:C1456->_j >>> >> > _k:C509->_j into _l [mergeDocStores] >>> >> > IW 9 [Indexer]: now sync _k.tii >>> >> > IW 9 [Indexer]: now sync _k.tis >>> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs >>> >> > IW 9 [Indexer]: now sync _j.tis >>> >> > IW 9 [Indexer]: now sync _j.tii >>> >> > IW 9 [Indexer]: now sync _j.fdx >>> >> > IW 9 [Indexer]: now sync _j.fdt >>> >> > IW 9 [Indexer]: now sync _k.prx >>> >> > IW 9 [Indexer]: now sync _k.nrm >>> >> > IW 9 [Indexer]: now sync _j.nrm >>> >> > IW 9 [Indexer]: now sync _j.prx >>> >> > IW 9 [Indexer]: now sync _j.fnm >>> >> > IW 9 [Indexer]: now sync _k.frq >>> >> > IW 9 [Indexer]: now sync _j.frq >>> >> > IW 9 [Indexer]: done all syncs >>> >> > IW 9 [Indexer]: flush at getReader >>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=_k >>> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >>> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j >>> >> > IW 9 [Indexer]: flush shared docStore segment _k >>> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment _k >>> >> numDocs=0 >>> >> > IW 9 [Indexer]: commit: start >>> >> > IW 9 [Indexer]: commit: already prepared >>> >> > IW 9 [Indexer]: commit: pendingCommit != null >>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c" >>> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit = >>> >> true] >>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b" >>> >> > IFD [Indexer]: delete "segments_b" >>> >> > IW 9 [Indexer]: commit: done >>> >> > IW 9 [Indexer]: flush at getReader >>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >>> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j >>> >> > IW 9 [Indexer]: now flush at close >>> >> > IW 9 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 9 [Indexer]: index before flush _a:C9780 _b:C1204->_b >>> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j >>> >> > IW 9 [Indexer]: CMS: now merge >>> >> > IW 9 [Indexer]: CMS: index: _a:C9780 _b:C1204->_b _c:C717->_b >>> >> _d:C1220->_d >>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h >>> >> _j:C1456->_j >>> >> > _k:C509->_j >>> >> > IW 9 [Indexer]: CMS: no more merges pending; now return >>> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780 >>> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f >>> >> _g:C858->_f >>> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j >>> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b >>> >> _c:C717->_b >>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h >>> >> _i:C703->_h >>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] >>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments >>> ; >>> >> > isCommit = false] >>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments >>> ; >>> >> > isCommit = false] >>> >> > IW 9 [Lucene Merge Thread #0]: CMS: merge thread: done >>> >> > IW 9 [Indexer]: now call final commit() >>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0 >>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8 >>> >> > IW 9 [Indexer]: now sync _l.nrm >>> >> > IW 9 [Indexer]: now sync _l.fnm >>> >> > IW 9 [Indexer]: now sync _l.prx >>> >> > IW 9 [Indexer]: now sync _l.frq >>> >> > IW 9 [Indexer]: now sync _l.tis >>> >> > IW 9 [Indexer]: now sync _l.fdx >>> >> > IW 9 [Indexer]: now sync _l.fdt >>> >> > IW 9 [Indexer]: now sync _l.tii >>> >> > IW 9 [Indexer]: done all syncs >>> >> > IW 9 [Indexer]: commit: pendingCommit != null >>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d" >>> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit = >>> true] >>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c" >>> >> > IFD [Indexer]: delete "_h.prx" >>> >> > IFD [Indexer]: delete "_f.tis" >>> >> > IFD [Indexer]: delete "_g.tis" >>> >> > IFD [Indexer]: delete "_d.tii" >>> >> > IFD [Indexer]: delete "_e.frq" >>> >> > IFD [Indexer]: delete "_k.fnm" >>> >> > IFD [Indexer]: delete "_c.tii" >>> >> > IFD [Indexer]: delete "_h.nrm" >>> >> > IFD [Indexer]: delete "_i.tis" >>> >> > IFD [Indexer]: delete "_k.tii" >>> >> > IFD [Indexer]: delete "_h.fdt" >>> >> > IFD [Indexer]: delete "_c.tis" >>> >> > IFD [Indexer]: delete "_g.tii" >>> >> > IFD [Indexer]: delete "_h.fdx" >>> >> > IFD [Indexer]: delete "_c.nrm" >>> >> > IFD [Indexer]: delete "_i.tii" >>> >> > IFD [Indexer]: delete "_d.tis" >>> >> > IFD [Indexer]: delete "_f.tii" >>> >> > IFD [Indexer]: delete "_e.fnm" >>> >> > IFD [Indexer]: delete "_k.tis" >>> >> > IFD [Indexer]: delete "_d.frq" >>> >> > IFD [Indexer]: delete "_f.nrm" >>> >> > IFD [Indexer]: delete "_h.fnm" >>> >> > IFD [Indexer]: delete "segments_c" >>> >> > IFD [Indexer]: delete "_c.prx" >>> >> > IFD [Indexer]: delete "_d.fnm" >>> >> > IFD [Indexer]: delete "_h.frq" >>> >> > IFD [Indexer]: delete "_h.tis" >>> >> > IFD [Indexer]: delete "_f.fdt" >>> >> > IFD [Indexer]: delete "_j.tis" >>> >> > IFD [Indexer]: delete "_f.fdx" >>> >> > IFD [Indexer]: delete "_g.nrm" >>> >> > IFD [Indexer]: delete "_f.prx" >>> >> > IFD [Indexer]: delete "_j.tii" >>> >> > IFD [Indexer]: delete "_h.tii" >>> >> > IFD [Indexer]: delete "_e.prx" >>> >> > IFD [Indexer]: delete "_c.frq" >>> >> > IFD [Indexer]: delete "_i.prx" >>> >> > IFD [Indexer]: delete "_j.fdx" >>> >> > IFD [Indexer]: delete "_i.nrm" >>> >> > IFD [Indexer]: delete "_j.fdt" >>> >> > IFD [Indexer]: delete "_k.prx" >>> >> > IFD [Indexer]: delete "_b.nrm" >>> >> > IFD [Indexer]: delete "_e.tis" >>> >> > IFD [Indexer]: delete "_e.tii" >>> >> > IFD [Indexer]: delete "_g.fnm" >>> >> > IFD [Indexer]: delete "_k.nrm" >>> >> > IFD [Indexer]: delete "_j.nrm" >>> >> > IFD [Indexer]: delete "_j.prx" >>> >> > IFD [Indexer]: delete "_e.nrm" >>> >> > IFD [Indexer]: delete "_g.prx" >>> >> > IFD [Indexer]: delete "_b.prx" >>> >> > IFD [Indexer]: delete "_i.frq" >>> >> > IFD [Indexer]: delete "_b.tis" >>> >> > IFD [Indexer]: delete "_d.prx" >>> >> > IFD [Indexer]: delete "_g.frq" >>> >> > IFD [Indexer]: delete "_d.fdx" >>> >> > IFD [Indexer]: delete "_b.tii" >>> >> > IFD [Indexer]: delete "_b.frq" >>> >> > IFD [Indexer]: delete "_d.fdt" >>> >> > IFD [Indexer]: delete "_f.fnm" >>> >> > IFD [Indexer]: delete "_j.fnm" >>> >> > IFD [Indexer]: delete "_b.fnm" >>> >> > IFD [Indexer]: delete "_d.nrm" >>> >> > IFD [Indexer]: delete "_b.fdt" >>> >> > IFD [Indexer]: delete "_k.frq" >>> >> > IFD [Indexer]: delete "_c.fnm" >>> >> > IFD [Indexer]: delete "_j.frq" >>> >> > IFD [Indexer]: delete "_i.fnm" >>> >> > IFD [Indexer]: delete "_f.frq" >>> >> > IFD [Indexer]: delete "_b.fdx" >>> >> > IW 9 [Indexer]: commit: done >>> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909 >>> >> > >>> >> > ===================================================== >>> >> > >>> >> > IFD [Indexer]: setInfoStream >>> >> > >>> >> >>> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@68d349c1 >>> >> > IW 10 [Indexer]: setInfoStream: >>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated >>> >> > pathname>\lresumes1.search.main.1 autoCommit=false >>> >> > >>> >> >>> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@78f53abmergescheduler >>> >> >>> =org.apache.lucene.index.concurrentmergeschedu...@52be036drambuffersizemb >>> >> =16.0 >>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 >>> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909 >>> >> > IW 10 [Indexer]: flush at getReader >>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 >>> >> > IW 10 [UpdWriterBuild : 10]: DW: RAM: now flush @ usedMB=15.789 >>> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16 >>> >> > IW 10 [UpdWriterBuild : 10]: flush: segment=_m docStoreSegment=_m >>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false >>> flushDocStores=false >>> >> > numDocs=1526 numBufDelTerms=1526 >>> >> > IW 10 [UpdWriterBuild : 10]: index before flush _a:C9780 _l:C9909 >>> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m >>> >> numDocs=1526 >>> >> > IW 10 [UpdWriterBuild : 10]: DW: oldRAMSize=16556032 >>> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925% >>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ; >>> >> > isCommit = false] >>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments ; >>> >> > isCommit = false] >>> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments >>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.8642592 to 7.6142592: 2 >>> >> segments >>> >> > IW 10 [UpdWriterBuild : 10]: LMP: level 6.2247195 to 6.6802545: 1 >>> >> segments >>> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge >>> >> > IW 10 [UpdWriterBuild : 10]: CMS: index: _a:C9780 _l:C9909 >>> _m:C1526->_m >>> >> > IW 10 [UpdWriterBuild : 10]: CMS: no more merges pending; now >>> return >>> >> > IW 10 [Indexer]: prepareCommit: flush >>> >> > IW 10 [Indexer]: flush: segment=_n docStoreSegment=_m >>> >> docStoreOffset=1526 >>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501 >>> >> > numBufDelTerms=501 >>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >>> >> > IW 10 [Indexer]: flush shared docStore segment _m >>> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment _m >>> >> > numDocs=2027 >>> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501 >>> >> > IW 10 [Indexer]: DW: oldRAMSize=6927360 newFlushedSize=1659342 >>> >> > docs/MB=316.593 new/old=23.953% >>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = >>> >> false] >>> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted >>> >> docIDs >>> >> > and 0 deleted queries on 4 segments. >>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit = >>> >> false] >>> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments >>> >> > IW 10 [Indexer]: LMP: level 6.8642592 to 7.6142592: 2 segments >>> >> > IW 10 [Indexer]: LMP: level 6.2247195 to 6.6802545: 1 segments >>> >> > IW 10 [Indexer]: LMP: level -1.0 to 6.220011: 1 segments >>> >> > IW 10 [Indexer]: CMS: now merge >>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m >>> _n:C501->_m >>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return >>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0 >>> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m >>> >> > _n:C501->_m changeCount=6 >>> >> > IW 10 [Indexer]: now sync _m.frq >>> >> > IW 10 [Indexer]: now sync _m.prx >>> >> > IW 10 [Indexer]: now sync _n.tis >>> >> > IW 10 [Indexer]: now sync _m.tis >>> >> > IW 10 [Indexer]: now sync _n.frq >>> >> > IW 10 [Indexer]: now sync _n.tii >>> >> > IW 10 [Indexer]: now sync _m.tii >>> >> > IW 10 [Indexer]: now sync _m.fnm >>> >> > IW 10 [Indexer]: now sync _n.fnm >>> >> > IW 10 [Indexer]: now sync _n.nrm >>> >> > IW 10 [Indexer]: now sync _n.prx >>> >> > IW 10 [Indexer]: now sync _m.nrm >>> >> > IW 10 [Indexer]: now sync _m.fdt >>> >> > IW 10 [Indexer]: now sync _m.fdx >>> >> > IW 10 [Indexer]: done all syncs >>> >> > IW 10 [Indexer]: flush at getReader >>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=_n >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >>> >> > _n:C501->_m >>> >> > IW 10 [Indexer]: flush shared docStore segment _n >>> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment _n >>> >> numDocs=0 >>> >> > IW 10 [Indexer]: commit: start >>> >> > IW 10 [Indexer]: commit: already prepared >>> >> > IW 10 [Indexer]: commit: pendingCommit != null >>> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e" >>> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit = >>> true] >>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d" >>> >> > IFD [Indexer]: delete "segments_d" >>> >> > IW 10 [Indexer]: commit: done >>> >> > IW 10 [Indexer]: flush at getReader >>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >>> >> > _n:C501->_m >>> >> > IW 10 [Indexer]: now flush at close >>> >> > IW 10 [Indexer]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0 >>> >> > numBufDelTerms=0 >>> >> > IW 10 [Indexer]: index before flush _a:C9780 _l:C9909 _m:C1526->_m >>> >> > _n:C501->_m >>> >> > IW 10 [Indexer]: CMS: now merge >>> >> > IW 10 [Indexer]: CMS: index: _a:C9780 _l:C9909 _m:C1526->_m >>> _n:C501->_m >>> >> > IW 10 [Indexer]: CMS: no more merges pending; now return >>> >> > IW 10 [Indexer]: now call final commit() >>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0 >>> >> > IW 10 [Indexer]: skip startCommit(): no changes pending >>> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip >>> >> > IW 10 [Indexer]: commit: done >>> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m >>> >> > >>> >> > >>> >> > Peter >>> >> > >>> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless < >>> >> > luc...@mikemccandless.com> wrote: >>> >> > >>> >> >> Do you know the place in the infoStream output where you got a >>> reader >>> >> >> with the wrong (unexplained extra +1) version? If so, can you post >>> >> >> the infoStream output up to that point? >>> >> >> >>> >> >> Mike >>> >> >> >>> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan < >>> peterlkee...@gmail.com> >>> >> >> wrote: >>> >> >> > I've reproduced this and I have a bunch of infoStream log files. >>> Since >>> >> >> the >>> >> >> > messages have no timestamps, it's hard to tell where the relevant >>> >> entries >>> >> >> > are. What should I be looking for? >>> >> >> > >>> >> >> > Peter >>> >> >> > >>> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan < >>> peterlkee...@gmail.com >>> >> >> >wrote: >>> >> >> > >>> >> >> >> I'm pretty sure there are flushes and segment merges going on, >>> but as >>> >> >> you >>> >> >> >> said, that shouldn't affect the version increment. I'll see what >>> I >>> >> can >>> >> >> do to >>> >> >> >> get infoStream output. >>> >> >> >> >>> >> >> >> Thanks, >>> >> >> >> Peter >>> >> >> >> >>> >> >> >> >>> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless < >>> >> >> >> luc...@mikemccandless.com> wrote: >>> >> >> >> >>> >> >> >>> Well I'm at a loss then. The version should only increment on >>> >> commit. >>> >> >> >>> >>> >> >> >>> Can you make it all happen when infoStream is on, and post back? >>> >> >> >>> >>> >> >> >>> Mike >>> >> >> >>> >>> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan < >>> >> peterlkee...@gmail.com >>> >> >> > >>> >> >> >>> wrote: >>> >> >> >>> > Only one writer thread and one writer process. >>> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer a, boolean >>> create, >>> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false. >>> >> >> >>> > >>> >> >> >>> > Peter >>> >> >> >>> > >>> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael McCandless < >>> >> >> >>> > luc...@mikemccandless.com> wrote: >>> >> >> >>> > >>> >> >> >>> >> That's curious. >>> >> >> >>> >> >>> >> >> >>> >> It's only on prepareCommit (or, commit, if you didn't first >>> >> prepare, >>> >> >> >>> >> since that will call prepareCommit internally) that this >>> version >>> >> >> >>> >> should increase. >>> >> >> >>> >> >>> >> >> >>> >> Is there only 1 thread doing this? >>> >> >> >>> >> >>> >> >> >>> >> Oh, and, are you passing false for autoCommit? >>> >> >> >>> >> >>> >> >> >>> >> Mike >>> >> >> >>> >> >>> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter Keegan < >>> >> >> peterlkee...@gmail.com >>> >> >> >>> > >>> >> >> >>> >> wrote: >>> >> >> >>> >> > Using Lucene 2.9.1, I have the following pseudocode which >>> gets >>> >> >> >>> repeated >>> >> >> >>> >> at >>> >> >> >>> >> > regular intervals: >>> >> >> >>> >> > >>> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File); >>> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory()); >>> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir, Analyzer, >>> false, >>> >> >> >>> >> maxFieldLen) >>> >> >> >>> >> > 4. writer.getReader().getVersion(); >>> >> >> >>> >> > 5. writer.prepareCommit(); >>> >> >> >>> >> > 6. writer.getReader().getVersion(); >>> >> >> >>> >> > 7. writer.commit(); >>> >> >> >>> >> > 8. writer.close(); >>> >> >> >>> >> > >>> >> >> >>> >> > I'm using the version number to keep external data in synch >>> >> with >>> >> >> the >>> >> >> >>> >> index. >>> >> >> >>> >> > Usually, the version number from (6) is 1 greater than from >>> (4) >>> >> >> and >>> >> >> >>> the >>> >> >> >>> >> > version from (4) equals the version from the previous (6). >>> At >>> >> >> least >>> >> >> >>> once >>> >> >> >>> >> a >>> >> >> >>> >> > day, however, the version from (4) is 1 greater than from >>> the >>> >> >> >>> previous >>> >> >> >>> >> (6). >>> >> >> >>> >> > What would explain this sporadic behavior of version >>> numbers? >>> >> >> >>> >> > >>> >> >> >>> >> > Thanks, >>> >> >> >>> >> > Peter >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >> >>> >> >> >>> --------------------------------------------------------------------- >>> >> >> >>> >> To unsubscribe, e-mail: >>> java-user-unsubscr...@lucene.apache.org >>> >> >> >>> >> For additional commands, e-mail: >>> >> java-user-h...@lucene.apache.org >>> >> >> >>> >> >>> >> >> >>> >> >>> >> >> >>> > >>> >> >> >>> >>> >> >> >>> >>> >> --------------------------------------------------------------------- >>> >> >> >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> >> >>> For additional commands, e-mail: >>> java-user-h...@lucene.apache.org >>> >> >> >>> >>> >> >> >>> >>> >> >> >> >>> >> >> > >>> >> >> >>> >> >> >>> --------------------------------------------------------------------- >>> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> >>> >> >> >>> >> > >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >>> >> >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
