[ https://issues.apache.org/jira/browse/LUCENE-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramazan VARLIKLI closed LUCENE-2192. ------------------------------------ Resolution: Invalid Yes, The problem is not with Lucene but our application > Memory Leak > ------------ > > Key: LUCENE-2192 > URL: https://issues.apache.org/jira/browse/LUCENE-2192 > Project: Lucene - Java > Issue Type: Bug > Components: Index > Affects Versions: 2.9 > Reporter: Ramazan VARLIKLI > > Hi All , > I have been working on a problem with Lucene and now gave up after trying > many different possibilites which gives me a feeling that There is a bug on > this . > The scenario is we have an CMS applicaton into which we add new content every > week , instead of updating the index which is a bit tricky, I prefer to > delete all index documents and add them again which is straightforward . The > problem is Lucene doesn't delete the old data somehow and increase the index > size every time during the update . I also profile it with java tools and see > that even if I close the IndexWriter class and sent it to Garbage Collector > it holds all the docs in the memory . > Here is the code I use > Directory directory = new SimpleFSDirectory(new File(path)); > writer = new IndexWriter(directory, analyzer, > false,IndexWriter.MaxFieldLength.LIMITED); > writer.deleteAll(); > //after adding docs close the indexwriter > writer.close(); > The above code invoked every time we need to update the index . I tried many > different scenario here to overcome the problem which includes physically > removing the index directory( see how desperate I am ) , optimizing , > flushing, commiting indexwriter, create=true parameter and so on . > Here is the index file size during creation. If I shutdown the application > and restart it , index size starts with 2,458 which is correct size. > Any help will be appreciated > _17.cfs 2,458 KB > _18.cfs 3,990 KB > _19.cfs 5,149 KB > here is the Lucene logs during creationg of index files 3 times in a row > IFD [http-8080-1]: setInfoStream > deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@6649 > IW 0 [http-8080-1]: setInfoStream: > dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and > Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene > autoCommit=false > mergepolicy=org.apache.lucene.index.logbytesizemergepol...@3b626c > mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@baa6ba > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 > maxFieldLength=10000 index= > IW 0 [http-8080-1]: now flush at close > IW 0 [http-8080-1]: flush: segment=_17 docStoreSegment=_17 docStoreOffset=0 > flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765 > numBufDelTerms=0 > IW 0 [http-8080-1]: index before flush > IW 0 [http-8080-1]: DW: flush postings as segment _17 numDocs=2765 > IW 0 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _17 > numDocs=2765 > IW 0 [http-8080-1]: DW: oldRAMSize=7485440 newFlushedSize=2472818 > docs/MB=1,172.473 new/old=33.035% > IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = > false] > IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = > false] > IFD [http-8080-1]: delete "_17.fdx" > IFD [http-8080-1]: delete "_17.tis" > IFD [http-8080-1]: delete "_17.frq" > IFD [http-8080-1]: delete "_17.nrm" > IFD [http-8080-1]: delete "_17.fdt" > IFD [http-8080-1]: delete "_17.fnm" > IFD [http-8080-1]: delete "_17.tii" > IFD [http-8080-1]: delete "_17.prx" > IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = > false] > IW 0 [http-8080-1]: LMP: findMerges: 1 segments > IW 0 [http-8080-1]: LMP: level 6.2247195 to 6.400742: 1 segments > IW 0 [http-8080-1]: CMS: now merge > IW 0 [http-8080-1]: CMS: index: _17:c2765 > IW 0 [http-8080-1]: CMS: no more merges pending; now return > IW 0 [http-8080-1]: CMS: now merge > IW 0 [http-8080-1]: CMS: index: _17:c2765 > IW 0 [http-8080-1]: CMS: no more merges pending; now return > IW 0 [http-8080-1]: now call final commit() > IW 0 [http-8080-1]: startCommit(): start sizeInBytes=0 > IW 0 [http-8080-1]: startCommit index=_17:c2765 changeCount=5 > IW 0 [http-8080-1]: now sync _17.cfs > IW 0 [http-8080-1]: done all syncs > IW 0 [http-8080-1]: commit: pendingCommit != null > IW 0 [http-8080-1]: commit: wrote segments file "segments_1k" > IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = true] > IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1j" > IFD [http-8080-1]: delete "_16.cfs" > IFD [http-8080-1]: delete "segments_1j" > IW 0 [http-8080-1]: commit: done > IW 0 [http-8080-1]: at close: _17:c2765 > IFD [http-8080-1]: setInfoStream > deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@fb1ba7 > IW 1 [http-8080-1]: setInfoStream: > dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and > Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene > autoCommit=false > mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1d49559 > mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1990e2d > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 > maxFieldLength=10000 index= > IW 1 [http-8080-1]: now flush at close > IW 1 [http-8080-1]: flush: segment=_18 docStoreSegment=_18 docStoreOffset=0 > flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765 > numBufDelTerms=0 > IW 1 [http-8080-1]: index before flush > IW 1 [http-8080-1]: DW: flush postings as segment _18 numDocs=2765 > IW 1 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _18 > numDocs=2765 > IW 1 [http-8080-1]: DW: oldRAMSize=9517056 newFlushedSize=4042307 > docs/MB=717.242 new/old=42.474% > IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = > false] > IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = > false] > IFD [http-8080-1]: delete "_18.nrm" > IFD [http-8080-1]: delete "_18.frq" > IFD [http-8080-1]: delete "_18.fdx" > IFD [http-8080-1]: delete "_18.tii" > IFD [http-8080-1]: delete "_18.fdt" > IFD [http-8080-1]: delete "_18.prx" > IFD [http-8080-1]: delete "_18.fnm" > IFD [http-8080-1]: delete "_18.tis" > IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = > false] > IW 1 [http-8080-1]: LMP: findMerges: 1 segments > IW 1 [http-8080-1]: LMP: level 6.2247195 to 6.6112633: 1 segments > IW 1 [http-8080-1]: CMS: now merge > IW 1 [http-8080-1]: CMS: index: _18:c2765 > IW 1 [http-8080-1]: CMS: no more merges pending; now return > IW 1 [http-8080-1]: CMS: now merge > IW 1 [http-8080-1]: CMS: index: _18:c2765 > IW 1 [http-8080-1]: CMS: no more merges pending; now return > IW 1 [http-8080-1]: now call final commit() > IW 1 [http-8080-1]: startCommit(): start sizeInBytes=0 > IW 1 [http-8080-1]: startCommit index=_18:c2765 changeCount=5 > IW 1 [http-8080-1]: now sync _18.cfs > IW 1 [http-8080-1]: done all syncs > IW 1 [http-8080-1]: commit: pendingCommit != null > IW 1 [http-8080-1]: commit: wrote segments file "segments_1l" > IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = true] > IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1k" > IFD [http-8080-1]: delete "segments_1k" > IFD [http-8080-1]: delete "_17.cfs" > IW 1 [http-8080-1]: commit: done > IW 1 [http-8080-1]: at close: _18:c2765 > IFD [http-8080-1]: setInfoStream > deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@7ceec1 > IW 2 [http-8080-1]: setInfoStream: > dir=org.apache.lucene.store.simplefsdirect...@c:\Documents and > Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene > autoCommit=false > mergepolicy=org.apache.lucene.index.logbytesizemergepol...@1edacc > mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@1ae9ba8 > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 > maxFieldLength=10000 index= > IW 2 [http-8080-1]: now flush at close > IW 2 [http-8080-1]: flush: segment=_19 docStoreSegment=_19 docStoreOffset=0 > flushDocs=true flushDeletes=true flushDocStores=true numDocs=2765 > numBufDelTerms=0 > IW 2 [http-8080-1]: index before flush > IW 2 [http-8080-1]: DW: flush postings as segment _19 numDocs=2765 > IW 2 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _19 > numDocs=2765 > IW 2 [http-8080-1]: DW: oldRAMSize=11188224 newFlushedSize=5229106 > docs/MB=554.457 new/old=46.738% > IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = > false] > IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = > false] > IFD [http-8080-1]: delete "_19.tis" > IFD [http-8080-1]: delete "_19.prx" > IFD [http-8080-1]: delete "_19.nrm" > IFD [http-8080-1]: delete "_19.fnm" > IFD [http-8080-1]: delete "_19.fdx" > IFD [http-8080-1]: delete "_19.fdt" > IFD [http-8080-1]: delete "_19.tii" > IFD [http-8080-1]: delete "_19.frq" > IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = > false] > IW 2 [http-8080-1]: LMP: findMerges: 1 segments > IW 2 [http-8080-1]: LMP: level 6.2247195 to 6.722014: 1 segments > IW 2 [http-8080-1]: CMS: now merge > IW 2 [http-8080-1]: CMS: index: _19:c2765 > IW 2 [http-8080-1]: CMS: no more merges pending; now return > IW 2 [http-8080-1]: CMS: now merge > IW 2 [http-8080-1]: CMS: index: _19:c2765 > IW 2 [http-8080-1]: CMS: no more merges pending; now return > IW 2 [http-8080-1]: now call final commit() > IW 2 [http-8080-1]: startCommit(): start sizeInBytes=0 > IW 2 [http-8080-1]: startCommit index=_19:c2765 changeCount=5 > IW 2 [http-8080-1]: now sync _19.cfs > IW 2 [http-8080-1]: done all syncs > IW 2 [http-8080-1]: commit: pendingCommit != null > IW 2 [http-8080-1]: commit: wrote segments file "segments_1m" > IFD [http-8080-1]: now checkpoint "segments_1m" [1 segments ; isCommit = true] > IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1l" > IFD [http-8080-1]: delete "_18.cfs" > IFD [http-8080-1]: delete "segments_1l" > IW 2 [http-8080-1]: commit: done > IW 2 [http-8080-1]: at close: _19:c2765 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org