Late reply, but do what Michael said. I didn't understand that you hadso many indexes, but opening/closing makes sense to me now, reusing them wouldn't be very useful. Although I *can* imagine a "keep each open for 5 minute" sort of rule being useful on the assumption that a user might search several ways in a short interval, but I sure wouldn't go there unless and until I had a performance issue to deal with....
In fact, I'd leave all the complex stuff (like, perhaps, queueing requests after there were X outstanding ones being serviced) until I could demonstrate a need. Although I would put in a bit of logging showing the max number of concurrent requests just to try to get ahead of the "perfect storm" scenario Michael mentioned. Erick On Tue, Sep 1, 2009 at 6:32 AM, Chris Bamford <chris.bamf...@scalix.com>wrote: > Hi Erick, > > >>Note that for search speed reasons, you really, really want to share your > >>readers and NOT open/close for every request. > I have often wondered about this - I hope you can help me understand it > better in the context of our app, which is an email client: > > When one of our users receives email we index and store it so he (and only > he) can search on it. This means a separate index per user. On large > customer sites this can mean hundreds/thousands of indexes. Sharing readers > seems counter-intuitive, unless I am missing something. What we do instead > is that once a user performs a search, we keep his IndexReader open in case > he searches again. At present, we have no expiry on this mechanism, so they > stay open indefinitely. I'm a bit hazy on the underlying details but we > have observed that the number of open fds jumps by around 10 each time a new > user performs a search. What would be a good strategy for managing this in > your opinon? Does it really make sense to keep the IndexReader open? Would > performance suffer that much if we did an open/close for each search? Or > would it perhaps be better to close open readers after a period of > inactivity? > > Thanks for any wisdom / thoughts/ ideas. > > - Chris > > > > ----- Original Message ----- > From: Erick Erickson <erickerick...@gmail.com> > Sent: Thu, 27/8/2009 4:49pm > To: java-user@lucene.apache.org > Subject: Re: Lucene gobbling file descriptors > > Note that for search speed reasons, you really, really want to share your > readers and NOT open/close for every request. > FWIW > Erick > > On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <chris.bamf...@scalix.com > >wrote: > > > I'm glad its not normal. That means we can fix it! I will conduct a > > review of IndexReader/Searcher open/close ops. > > > > Thanks! > > > > Chris > > > > ----- Original Message ----- > > From: Michael McCandless <luc...@mikemccandless.com> > > Sent: Wed, 26/8/2009 2:26pm > > To: java-user@lucene.apache.org > > Subject: Re: Lucene gobbling file descriptors > > > > This is not normal. As long as you are certain you close every > > IndexReader/Searcher that you opened, the number of file descriptors > > should stay "contained". > > > > Though: how many files are there in your index directory? > > > > Mike > > > > On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<chris.bamf...@scalix.com> > > wrote: > > > Hi there, > > > > > > I wonder if someone can help? We have a successful Lucene app deployed > > on Tomcat which works well. As far as we can tell, our developers have > > observed all the guidelines in the Lucene FAQ, but on some of our > > installations, Tomcat eventually runs out of file descriptors and needs a > > restart to clear it. We know Lucene is the culprit because use lsof -p > > <java PID> and the vast majority (usually tens of thousands) of files > > reported are Lucene index files. > > > > > > I am hoping to get some tips on how this can be avoided. Is it simply > > the case that as time goes by, more and more descriptors are left open > and > > no matter how high ulimit is set, you will run out? Or is there a policy > of > > recycling that we are failing to utilise properly? > > > > > > I am happy to provide more information, just don't know what at this > > point! Please ask.... > > > > > > Thanks in advance > > > > > > - Chris > > > > > > Chris Bamford > > > Senior Development Engineer > > > Scalix > > > chris.bamf...@scalix.com > > > Tel: +44 (0)1344 381814 > > > www.scalix.com > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
