[
https://issues.apache.org/jira/browse/LUCENE-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796268#action_12796268
]
Michael McCandless commented on LUCENE-2192:
--------------------------------------------
So there seems to be two problems:
* The old _X.cfs files are not getting removed
* Each _X.cfs file is growing in size, even though you sent it exactly the
same docs
Is that right?
> Memory Leak
> ------------
>
> Key: LUCENE-2192
> URL: https://issues.apache.org/jira/browse/LUCENE-2192
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Ramazan VARLIKLI
>
> Hi All ,
> I have been working on a problem with Lucene and now gave up after trying
> many different possibilites which gives me a feeling that There is a bug on
> this .
> The scenario is we have an CMS applicaton into which we add new content every
> week , instead of updating the index which is a bit tricky, I prefer to
> delete all index documents and add them again which is straightforward . The
> problem is Lucene doesn't delete the old data somehow and increase the index
> size every time during the update . I also profile it with java tools and see
> that even if I close the IndexWriter class and sent it to Garbage Collector
> it holds all the docs in the memory .
> Here is the code I use
> Directory directory = new SimpleFSDirectory(new File(path));
> writer = new IndexWriter(directory, analyzer,
> false,IndexWriter.MaxFieldLength.LIMITED);
> writer.deleteAll();
> //after adding docs close the indexwriter
> writer.close();
> The above code invoked every time we need to update the index . I tried many
> different scenario here to overcome the problem which includes physically
> removing the index directory( see how desperate I am ) , optimizing ,
> flushing, commiting indexwriter, create=true parameter and so on .
> Here is the index file size during creation. If I shutdown the application
> and restart it , index size starts with 2,458 which is correct size.
> Any help will be appreciated
> _17.cfs 2,458 KB
> _18.cfs 3,990 KB
> _19.cfs 5,149 KB
> here is the Lucene logs during creationg of index files 3 times in a row
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@6649
> IW 0 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@3b626c
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@baa6ba
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 0 [http-8080-1]: now flush at close
> IW 0 [http-8080-1]: flush: segment=_17 docStoreSegment=_17 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 0 [http-8080-1]: index before flush
> IW 0 [http-8080-1]: DW: flush postings as segment _17 numDocs=2765
> IW 0 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _17
> numDocs=2765
> IW 0 [http-8080-1]: DW: oldRAMSize=7485440 newFlushedSize=2472818
> docs/MB=1,172.473 new/old=33.035%
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_17.fdx"
> IFD [http-8080-1]: delete "_17.tis"
> IFD [http-8080-1]: delete "_17.frq"
> IFD [http-8080-1]: delete "_17.nrm"
> IFD [http-8080-1]: delete "_17.fdt"
> IFD [http-8080-1]: delete "_17.fnm"
> IFD [http-8080-1]: delete "_17.tii"
> IFD [http-8080-1]: delete "_17.prx"
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit =
> false]
> IW 0 [http-8080-1]: LMP: findMerges: 1 segments
> IW 0 [http-8080-1]: LMP: level 6.2247195 to 6.400742: 1 segments
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS: index: _17:c2765
> IW 0 [http-8080-1]: CMS: no more merges pending; now return
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS: index: _17:c2765
> IW 0 [http-8080-1]: CMS: no more merges pending; now return
> IW 0 [http-8080-1]: now call final commit()
> IW 0 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 0 [http-8080-1]: startCommit index=_17:c2765 changeCount=5
> IW 0 [http-8080-1]: now sync _17.cfs
> IW 0 [http-8080-1]: done all syncs
> IW 0 [http-8080-1]: commit: pendingCommit != null
> IW 0 [http-8080-1]: commit: wrote segments file "segments_1k"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1j"
> IFD [http-8080-1]: delete "_16.cfs"
> IFD [http-8080-1]: delete "segments_1j"
> IW 0 [http-8080-1]: commit: done
> IW 0 [http-8080-1]: at close: _17:c2765
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@fb1ba7
> IW 1 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1d49559
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1990e2d
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 1 [http-8080-1]: now flush at close
> IW 1 [http-8080-1]: flush: segment=_18 docStoreSegment=_18 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 1 [http-8080-1]: index before flush
> IW 1 [http-8080-1]: DW: flush postings as segment _18 numDocs=2765
> IW 1 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _18
> numDocs=2765
> IW 1 [http-8080-1]: DW: oldRAMSize=9517056 newFlushedSize=4042307
> docs/MB=717.242 new/old=42.474%
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_18.nrm"
> IFD [http-8080-1]: delete "_18.frq"
> IFD [http-8080-1]: delete "_18.fdx"
> IFD [http-8080-1]: delete "_18.tii"
> IFD [http-8080-1]: delete "_18.fdt"
> IFD [http-8080-1]: delete "_18.prx"
> IFD [http-8080-1]: delete "_18.fnm"
> IFD [http-8080-1]: delete "_18.tis"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit =
> false]
> IW 1 [http-8080-1]: LMP: findMerges: 1 segments
> IW 1 [http-8080-1]: LMP: level 6.2247195 to 6.6112633: 1 segments
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS: index: _18:c2765
> IW 1 [http-8080-1]: CMS: no more merges pending; now return
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS: index: _18:c2765
> IW 1 [http-8080-1]: CMS: no more merges pending; now return
> IW 1 [http-8080-1]: now call final commit()
> IW 1 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 1 [http-8080-1]: startCommit index=_18:c2765 changeCount=5
> IW 1 [http-8080-1]: now sync _18.cfs
> IW 1 [http-8080-1]: done all syncs
> IW 1 [http-8080-1]: commit: pendingCommit != null
> IW 1 [http-8080-1]: commit: wrote segments file "segments_1l"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1k"
> IFD [http-8080-1]: delete "segments_1k"
> IFD [http-8080-1]: delete "_17.cfs"
> IW 1 [http-8080-1]: commit: done
> IW 1 [http-8080-1]: at close: _18:c2765
> IFD [http-8080-1]: setInfoStream
> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@7ceec1
> IW 2 [http-8080-1]: setInfoStream:
> dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and
> Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
> autoCommit=false
> mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1edacc
> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1ae9ba8
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
> maxFieldLength=10000 index=
> IW 2 [http-8080-1]: now flush at close
> IW 2 [http-8080-1]: flush: segment=_19 docStoreSegment=_19 docStoreOffset=0
> flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765
> numBufDelTerms=0
> IW 2 [http-8080-1]: index before flush
> IW 2 [http-8080-1]: DW: flush postings as segment _19 numDocs=2765
> IW 2 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _19
> numDocs=2765
> IW 2 [http-8080-1]: DW: oldRAMSize=11188224 newFlushedSize=5229106
> docs/MB=554.457 new/old=46.738%
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IFD [http-8080-1]: delete "_19.tis"
> IFD [http-8080-1]: delete "_19.prx"
> IFD [http-8080-1]: delete "_19.nrm"
> IFD [http-8080-1]: delete "_19.fnm"
> IFD [http-8080-1]: delete "_19.fdx"
> IFD [http-8080-1]: delete "_19.fdt"
> IFD [http-8080-1]: delete "_19.tii"
> IFD [http-8080-1]: delete "_19.frq"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit =
> false]
> IW 2 [http-8080-1]: LMP: findMerges: 1 segments
> IW 2 [http-8080-1]: LMP: level 6.2247195 to 6.722014: 1 segments
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS: index: _19:c2765
> IW 2 [http-8080-1]: CMS: no more merges pending; now return
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS: index: _19:c2765
> IW 2 [http-8080-1]: CMS: no more merges pending; now return
> IW 2 [http-8080-1]: now call final commit()
> IW 2 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 2 [http-8080-1]: startCommit index=_19:c2765 changeCount=5
> IW 2 [http-8080-1]: now sync _19.cfs
> IW 2 [http-8080-1]: done all syncs
> IW 2 [http-8080-1]: commit: pendingCommit != null
> IW 2 [http-8080-1]: commit: wrote segments file "segments_1m"
> IFD [http-8080-1]: now checkpoint "segments_1m" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1l"
> IFD [http-8080-1]: delete "_18.cfs"
> IFD [http-8080-1]: delete "segments_1l"
> IW 2 [http-8080-1]: commit: done
> IW 2 [http-8080-1]: at close: _19:c2765
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
