[
https://issues.apache.org/jira/browse/LUCENE-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796312#action_12796312
]
Michael McCandless commented on LUCENE-2192:
--------------------------------------------
OK so it's only the 2nd problem.
>From your CheckIndex output, the 2nd segment has precisely 2X the number of
>tokens than the first segment (and the same number of documents and same
>number of unique terms).
Can you double check how you create the Document that you pass to Lucene? Is
it possible the field in the Document is just getting added twice? Can you
post the code that constructs the document?
> Memory Leak
> ------------
>
> Key: LUCENE-2192
> URL: https://issues.apache.org/jira/browse/LUCENE-2192
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Ramazan VARLIKLI
>
> Hi All ,
> I have been working on a problem with Lucene and now gave up after trying
> many different possibilites which gives me a feeling that There is a bug on
> this .
> The scenario is we have an CMS applicaton into which we add new content every
> week , instead of updating the index which is a bit tricky, I prefer to
> delete all index documents and add them again which is straightforward . The
> problem is Lucene doesn't delete the old data somehow and increase the index
> size every time during the update . I also profile it with java tools and see
> that even if I close the IndexWriter class and sent it to Garbage Collector
> it holds all the docs in the memory .
> Here is the code I use
> Directory directory = new SimpleFSDirectory(new File(path));
> writer = new IndexWriter(directory, analyzer,
> false,IndexWriter.MaxFieldLength.LIMITED);
> writer.deleteAll();
> //after adding docs close the indexwriter
> writer.close();
> The above code invoked every time we need to update the index . I tried many
> different scenario here to overcome the problem which includes physically
> removing the index directory( see how desperate I am ) , optimizing ,
> flushing, commiting indexwriter, create=true parameter and so on .
> Here is the index file size during creation. If I shutdown the application
> and restart it , index size starts with 2,458 which is correct size.
> Any help will be appreciated
> _17.cfs 2,458 KB
> _18.cfs 3,990 KB
> _19.cfs 5,149 KB
> here is the Lucene logs during creationg of index files 3 times in a row
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@6649
> IW 0 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@3b626c
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@baa6ba
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 0 [http-8080-1]: now flush at close
> IW 0 [http-8080-1]: flush: segment=_17 docStoreSegment=_17 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 0 [http-8080-1]: index before flush
> IW 0 [http-8080-1]: DW: flush postings as segment _17 numDocs=2765
> IW 0 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _17
> numDocs=2765
> IW 0 [http-8080-1]: DW: oldRAMSize=7485440 newFlushedSize=2472818
> docs/MB=1,172.473 new/old=33.035%
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_17.fdx"
> IFD [http-8080-1]: delete "_17.tis"
> IFD [http-8080-1]: delete "_17.frq"
> IFD [http-8080-1]: delete "_17.nrm"
> IFD [http-8080-1]: delete "_17.fdt"
> IFD [http-8080-1]: delete "_17.fnm"
> IFD [http-8080-1]: delete "_17.tii"
> IFD [http-8080-1]: delete "_17.prx"
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IW 0 [http-8080-1]: LMP: findMerges: 1 segments
> IW 0 [http-8080-1]: LMP: level 6.2247195 to 6.400742: 1 segments
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS: index: _17:c2765
> IW 0 [http-8080-1]: CMS: no more merges pending; now return
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS: index: _17:c2765
> IW 0 [http-8080-1]: CMS: no more merges pending; now return
> IW 0 [http-8080-1]: now call final commit()
> IW 0 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 0 [http-8080-1]: startCommit index=_17:c2765 changeCount=5
> IW 0 [http-8080-1]: now sync _17.cfs
> IW 0 [http-8080-1]: done all syncs
> IW 0 [http-8080-1]: commit: pendingCommit != null
> IW 0 [http-8080-1]: commit: wrote segments file "segments_1k"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1j"
> IFD [http-8080-1]: delete "_16.cfs"
> IFD [http-8080-1]: delete "segments_1j"
> IW 0 [http-8080-1]: commit: done
> IW 0 [http-8080-1]: at close: _17:c2765
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@fb1ba7
> IW 1 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1d49559
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1990e2d
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 1 [http-8080-1]: now flush at close
> IW 1 [http-8080-1]: flush: segment=_18 docStoreSegment=_18 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 1 [http-8080-1]: index before flush
> IW 1 [http-8080-1]: DW: flush postings as segment _18 numDocs=2765
> IW 1 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _18
> numDocs=2765
> IW 1 [http-8080-1]: DW: oldRAMSize=9517056 newFlushedSize=4042307
> docs/MB=717.242 new/old=42.474%
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_18.nrm"
> IFD [http-8080-1]: delete "_18.frq"
> IFD [http-8080-1]: delete "_18.fdx"
> IFD [http-8080-1]: delete "_18.tii"
> IFD [http-8080-1]: delete "_18.fdt"
> IFD [http-8080-1]: delete "_18.prx"
> IFD [http-8080-1]: delete "_18.fnm"
> IFD [http-8080-1]: delete "_18.tis"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IW 1 [http-8080-1]: LMP: findMerges: 1 segments
> IW 1 [http-8080-1]: LMP: level 6.2247195 to 6.6112633: 1 segments
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS: index: _18:c2765
> IW 1 [http-8080-1]: CMS: no more merges pending; now return
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS: index: _18:c2765
> IW 1 [http-8080-1]: CMS: no more merges pending; now return
> IW 1 [http-8080-1]: now call final commit()
> IW 1 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 1 [http-8080-1]: startCommit index=_18:c2765 changeCount=5
> IW 1 [http-8080-1]: now sync _18.cfs
> IW 1 [http-8080-1]: done all syncs
> IW 1 [http-8080-1]: commit: pendingCommit != null
> IW 1 [http-8080-1]: commit: wrote segments file "segments_1l"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1k"
> IFD [http-8080-1]: delete "segments_1k"
> IFD [http-8080-1]: delete "_17.cfs"
> IW 1 [http-8080-1]: commit: done
> IW 1 [http-8080-1]: at close: _18:c2765
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@7ceec1
> IW 2 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1edacc
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1ae9ba8
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 2 [http-8080-1]: now flush at close
> IW 2 [http-8080-1]: flush: segment=_19 docStoreSegment=_19 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 2 [http-8080-1]: index before flush
> IW 2 [http-8080-1]: DW: flush postings as segment _19 numDocs=2765
> IW 2 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _19
> numDocs=2765
> IW 2 [http-8080-1]: DW: oldRAMSize=11188224 newFlushedSize=5229106
> docs/MB=554.457 new/old=46.738%
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_19.tis"
> IFD [http-8080-1]: delete "_19.prx"
> IFD [http-8080-1]: delete "_19.nrm"
> IFD [http-8080-1]: delete "_19.fnm"
> IFD [http-8080-1]: delete "_19.fdx"
> IFD [http-8080-1]: delete "_19.fdt"
> IFD [http-8080-1]: delete "_19.tii"
> IFD [http-8080-1]: delete "_19.frq"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IW 2 [http-8080-1]: LMP: findMerges: 1 segments
> IW 2 [http-8080-1]: LMP: level 6.2247195 to 6.722014: 1 segments
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS: index: _19:c2765
> IW 2 [http-8080-1]: CMS: no more merges pending; now return
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS: index: _19:c2765
> IW 2 [http-8080-1]: CMS: no more merges pending; now return
> IW 2 [http-8080-1]: now call final commit()
> IW 2 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 2 [http-8080-1]: startCommit index=_19:c2765 changeCount=5
> IW 2 [http-8080-1]: now sync _19.cfs
> IW 2 [http-8080-1]: done all syncs
> IW 2 [http-8080-1]: commit: pendingCommit != null
> IW 2 [http-8080-1]: commit: wrote segments file "segments_1m"
> IFD [http-8080-1]: now checkpoint "segments_1m" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1l"
> IFD [http-8080-1]: delete "_18.cfs"
> IFD [http-8080-1]: delete "segments_1l"
> IW 2 [http-8080-1]: commit: done
> IW 2 [http-8080-1]: at close: _19:c2765
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
