Hi Jamie, YES, ist expected for the reasons described above (segments are still referenced by the open IndexReaders, but files were already deleted by IndexWriter). The approx. number of open, but already deleted files should be approx. stable.
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Jamie [mailto:ja...@stimulussoft.com] > Sent: Friday, October 01, 2010 7:41 AM > To: java-user@lucene.apache.org > Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge > > Hi Mike > > I managed to get hold of a copy of your book through Safari Books. > Quite an impressive online reading system they have there! I integrated your > SearchManager class into our code, but I am still seeing file handles marked > deleted in the index directory. I am running the following command on Linux: > > sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l" > > Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1 > 09:37:36 2010 > > 54 > > The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, etc. They > seem stable though. Is this to be expected when using NRT search? > > I am pretty certain that all Searchers are released at the end of every search. > I double checked it at least twenty times. > > Jamie > > > > On 2010/09/30 11:56 PM, Michael McCandless wrote: > > On Thu, Sep 30, 2010 at 5:59 AM, Jamie<ja...@stimulussoft.com> wrote: > >> Hi Michael / Uwe > >> > >>> It's good to cache the reader, but, finalize would worry me too > >>> since you have no control over when GC gets around to calling it... > >>> you risk tying up resources for longer than necessary. > >> I did it this way, as I didn't want to over complicate the code by > >> introducing mechanisms to track the number of search threads using a > >> shared indexreader. Admittedly, its not a very clean solution but in > >> my case it does work. Is there a particular technique for knowing > >> when to a close a reader when there are multiple search threads using > >> that reader? Should I keep some kind of counter and override the > >> close method of the reader such that the underlying reader is only closed > when everyone's done with it? > > See Uwe's response (or SearcherManager). > > > >>> IndexWriter has a reader pool, internally, where it holds open > >>> SegmentReaders for the still-live segments in the index. This is > >>> used by IndexReader.reopen to share open SegmentReaders. > >>> > >>> But the open files should correspond only to segments still "live" > >>> in the index. After segments are merged away, these readers are dropped. > >>> Is this what you are seeing? > >>> > >> I dont fully understand your explanation/question. When I run lsof, I > >> am seeing the following: > >> > >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cf > >> s > >> (deleted) > >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cf > >> s > >> (deleted) > >> > >> I assume these are left by the OS after the merge operation tried to > >> delete old segments. The OS is unable to delete the files. I think > >> its because our new code never closes the indexwriter, but rather > >> uses the > >> indexwriter.commit() method to apply the changes. Is this correct? > > Ahh I see they are deleted but held open... hmmm. > > > > Though this is also what you'd see if there were still a reader open. > > Are you certain all readers were closed (finalized) when you ran lsof? > > > > Mike > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
