Frankly I don't know why TermInfosReader.ThreadResources is not showing up in the memory snapshot.
Yes. It's been there for a long time. But let's see what's changed : A LRU cache of termInfoCache is added. I SegmentTermEnum previously would be released, since it's relatively a simple object. But with a cache added to the same class ThreadResources, which hold many objects, with the threads still hanging around, the cache can not be released, so in turn the SegmentTermEnum can not be released, so the RAMDirectory can not be released. My test is too coupled with the software I am working on and not easy to post here. But here is a similar case from another user: ----------------------------------------------------------------------------------- i found a forum post from you here [1] where you mention that you have a memory leak using the lucene ram directory. I'd like to ask you if you already have resolved the problem and how you did it or maybe you know where i can read about the solution. We are using RAMDirectory too and figured out, that over time the memory consumption raises and raises until the system breaks down but only when we performing much index updates. if we only create the index and don't do nothing except searching it, it work fine. ----------------------------------------------------------------------------------- -- Chris Lu ------------------------- Instant Scalable Full-Text Search On Any Database/Application site: http://www.dbsight.net demo: http://search.dbsight.com Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes DBSight customer, a shopping comparison site, (anonymous per request) got 2.6 Million Euro funding! On Wed, Sep 10, 2008 at 2:45 AM, Michael McCandless < [EMAIL PROTECTED]> wrote: > > I still don't quite understand what's causing your memory growth. > > SegmentTermEnum insances have been held in a ThreadLocal cache in > TermInfosReader for a very long time (at least since Lucene 1.4). > > If indeed it's the RAMDir's contents being kept "alive" due to this, then, > you should have already been seeing this problem before rev 659602. And I > still don't get why your reference tree is missing the > TermInfosReader.ThreadResources class. > > I'd like to understand the root cause before we hash out possible > solutions. > > Can you post the sources for your load test? > > Mike > > > Chris Lu wrote: > > Actually, even I only use one IndexReader, some resources are cached via >> the ThreadLocal cache, and can not be released unless all threads do the >> close action. >> >> SegmentTermEnum itself is small, but it holds RAMDirectory along the path, >> which is big. >> >> -- >> Chris Lu >> ------------------------- >> Instant Scalable Full-Text Search On Any Database/Application >> site: http://www.dbsight.net >> demo: http://search.dbsight.com >> Lucene Database Search in 3 minutes: >> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >> DBSight customer, a shopping comparison site, (anonymous per request) got >> 2.6 Million Euro funding! >> >> On Tue, Sep 9, 2008 at 10:43 PM, robert engels <[EMAIL PROTECTED]> >> wrote: >> You do not need a pool of IndexReaders... >> >> It does not matter what class it is, what matters is the class that >> ultimately holds the reference. >> >> If the IndexReader is never closed, the SegmentReader(s) is never closed, >> so the thread local in TermInfosReader is not cleared (because the thread >> never dies). So you will get one SegmentTermEnum, per thread * per segment. >> >> The SegmentTermEnum is not a large object, so even if you had 100 threads, >> and 100 segments, for 10k instances, seems hard to believe that is the >> source of your memory issue. >> >> The SegmentTermEnum is cached by thread since it needs to enumerate the >> terms, not having a per thread cache, would lead to lots of random access >> when multiple threads read the index - very slow. >> >> You need to keep in mind, what if every thread was executing a search >> simultaneously - you would still have 100x100 SegmentTermEnum instances >> anyway ! The only way to prevent that would be to create and destroy the >> SegmentTermEnum on each call (opening and seeking to the proper spot) - >> which would be SLOW SLOW SLOW. >> >> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote: >> >> I have tried to create an IndexReader pool and dynamically create >>> searcher. But the memory leak is the same. It's not related to the Searcher >>> class specifically, but the SegmentTermEnum in TermInfosReader. >>> >>> -- >>> Chris Lu >>> ------------------------- >>> Instant Scalable Full-Text Search On Any Database/Application >>> site: http://www.dbsight.net >>> demo: http://search.dbsight.com >>> Lucene Database Search in 3 minutes: >>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>> DBSight customer, a shopping comparison site, (anonymous per request) got >>> 2.6 Million Euro funding! >>> >>> On Tue, Sep 9, 2008 at 10:14 PM, robert engels <[EMAIL PROTECTED]> >>> wrote: >>> A searcher uses an IndexReader - the IndexReader is slow to open, not a >>> Searcher. And searchers can share an IndexReader. >>> >>> You want to create a single shared (across all threads/users) IndexReader >>> (usually), and create an Searcher as needed and dispose. It is VERY CHEAP >>> to create the Searcher. >>> >>> I am fairly certain the javadoc on Searcher is incorrect. The warning >>> "For performance reasons it is recommended to open only one IndexSearcher >>> and use it for all of your searches" is not true in the case where an >>> IndexReader is passed to the ctor. >>> >>> Any caching should USUALLY be performed at the IndexReader level. >>> >>> You are most likely using the "path" ctor, and that is the source of your >>> problems, as multiple IndexReader instances are being created, and thus the >>> memory use. >>> >>> >>> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote: >>> >>> On J2EE environment, usually there is a searcher pool with several >>>> searchers open. >>>> The speed to opening a large index for every user is not acceptable. >>>> >>>> -- >>>> Chris Lu >>>> ------------------------- >>>> Instant Scalable Full-Text Search On Any Database/Application >>>> site: http://www.dbsight.net >>>> demo: http://search.dbsight.com >>>> Lucene Database Search in 3 minutes: >>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>> got 2.6 Million Euro funding! >>>> >>>> On Tue, Sep 9, 2008 at 9:03 PM, robert engels <[EMAIL PROTECTED]> >>>> wrote: >>>> You need to close the searcher within the thread that is using it, in >>>> order to have it cleaned up quickly... usually right after you display the >>>> page of results. >>>> >>>> If you are keeping multiple searcher refs across multiple threads for >>>> paging/whatever, you have not coded it correctly. >>>> >>>> Imagine 10,000 users - storing a searcher for each one is not going to >>>> work... >>>> >>>> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote: >>>> >>>> Right, in a sense I can not release it from another thread. But that's >>>>> the problem. >>>>> >>>>> It's a J2EE environment, all threads are kind of equal. It's simply not >>>>> possible to iterate through all threads to close the searcher, thus >>>>> releasing the ThreadLocal cache. >>>>> Unless Lucene is not recommended for J2EE environment, this has to be >>>>> fixed. >>>>> >>>>> -- >>>>> Chris Lu >>>>> ------------------------- >>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>> site: http://www.dbsight.net >>>>> demo: http://search.dbsight.com >>>>> Lucene Database Search in 3 minutes: >>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>> got 2.6 Million Euro funding! >>>>> >>>>> >>>>> On Tue, Sep 9, 2008 at 8:14 PM, robert engels <[EMAIL PROTECTED]> >>>>> wrote: >>>>> Your code is not correct. You cannot release it on another thread - the >>>>> first thread may creating hundreds/thousands of instances before the other >>>>> thread ever runs... >>>>> >>>>> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote: >>>>> >>>>> If I release it on the thread that's creating the searcher, by setting >>>>>> searcher=null, everything is fine, the memory is released very cleanly. >>>>>> My load test was to repeatedly create a searcher on a RAMDirectory and >>>>>> release it on another thread. The test will quickly go to OOM after >>>>>> several >>>>>> runs. I set the heap size to be 1024M, and the RAMDirectory is of size >>>>>> 250M. >>>>>> Using some profiling tool, the used size simply stepped up pretty >>>>>> obviously >>>>>> by 250M. >>>>>> >>>>>> I think we should not rely on something that's a "maybe" behavior, >>>>>> especially for a general purpose library. >>>>>> >>>>>> Since it's a multi-threaded env, the thread that's creating the >>>>>> entries in the LRU cache may not go away quickly(actually most, if not >>>>>> all, >>>>>> application servers will try to reuse threads), so the LRU cache, which >>>>>> uses >>>>>> thread as the key, can not be released, so the SegmentTermEnum which is >>>>>> in >>>>>> the same class can not be released. >>>>>> >>>>>> And yes, I close the RAMDirectory, and the fileMap is released. I >>>>>> verified that through the profiler by directly checking the values in the >>>>>> snapshot. >>>>>> >>>>>> Pretty sure the reference tree wasn't like this using code before this >>>>>> commit, because after close the searcher in another thread, the >>>>>> RAMDirectory >>>>>> totally disappeared from the memory snapshot. >>>>>> >>>>>> -- >>>>>> Chris Lu >>>>>> ------------------------- >>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>> site: http://www.dbsight.net >>>>>> demo: http://search.dbsight.com >>>>>> Lucene Database Search in 3 minutes: >>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>>> got 2.6 Million Euro funding! >>>>>> >>>>>> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless < >>>>>> [EMAIL PROTECTED]> wrote: >>>>>> >>>>>> Chris Lu wrote: >>>>>> >>>>>> The problem should be similar to what's talked about on this >>>>>> discussion. >>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal >>>>>> >>>>>> The "rough" conclusion of that thread is that, technically, this isn't >>>>>> a memory leak but rather a "delayed freeing" problem. Ie, it may take >>>>>> longer, possibly much longer, than you want for the memory to be freed. >>>>>> >>>>>> >>>>>> There is a memory leak for Lucene search from Lucene-1195.(svn >>>>>> r659602, May23,2008) >>>>>> >>>>>> This patch brings in a ThreadLocal cache to TermInfosReader. >>>>>> >>>>>> One thing that confuses me: TermInfosReader was already using a >>>>>> ThreadLocal to cache the SegmentTermEnum instance. What was added in >>>>>> this >>>>>> commit (for LUCENE-1195) was an LRU cache storing Term -> TermInfo >>>>>> instances. But it seems like it's the SegmentTermEnum instance that >>>>>> you're >>>>>> tracing below. >>>>>> >>>>>> >>>>>> It's usually recommended to keep the reader open, and reuse it when >>>>>> possible. In a common J2EE application, the http requests are usually >>>>>> handled by different threads. But since the cache is ThreadLocal, the >>>>>> cache >>>>>> are not really usable by other threads. What's worse, the cache can >>>>>> not be >>>>>> cleared by another thread! >>>>>> >>>>>> This leak is not so obvious usually. But my case is using >>>>>> RAMDirectory, >>>>>> having several hundred megabytes. So one un-released resource is >>>>>> obvious to >>>>>> me. >>>>>> >>>>>> Here is the reference tree: >>>>>> org.apache.lucene.store.RAMDirectory >>>>>> |- directory of org.apache.lucene.store.RAMFile >>>>>> |- file of org.apache.lucene.store.RAMInputStream >>>>>> |- base of >>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput >>>>>> |- input of org.apache.lucene.index.SegmentTermEnum >>>>>> |- value of java.lang.ThreadLocal$ThreadLocalMap$Entry >>>>>> >>>>>> So you have a RAMDir that has several hundred MB stored in it, that >>>>>> you're done with yet through this path Lucene is keeping it alive? >>>>>> >>>>>> Did you close the RAMDir? (which will null its fileMap and should >>>>>> also free your memory). >>>>>> >>>>>> Also, that reference tree doesn't show the ThreadResources class that >>>>>> was added in that commit -- are you sure this reference tree wasn't >>>>>> before >>>>>> the commit? >>>>>> >>>>>> Mike >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Chris Lu >>>>>> ------------------------- >>>>>> Instant Scalable Full-Text Search On Any Database/Application >>>>>> site: http://www.dbsight.net >>>>>> demo: http://search.dbsight.com >>>>>> Lucene Database Search in 3 minutes: >>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes >>>>>> DBSight customer, a shopping comparison site, (anonymous per request) >>>>>> got 2.6 Million Euro funding! >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
