Re: References to deleted file handles in long-running server application

[Paul Smith]

I have found exactly the same problem. You should consider adding an 'expired" IndexSearcher to some sort of Buffer that closes the IndexSearcher after some default timeout. I have not found in practice that allowing the GC to do the work does quite what you think it might do (particularly on Linux).

Paul Smith

On 18/11/2005, at 6:57 PM, Matt Magoffin wrote:

It could be to do with the different constructors of IndexSearcher (String vs IndexReader), but my advise is to mark the IndexSearcher as ready to be closed by placing it in a buffer, and after some sensible timeout where you would consider any existing queries that may have been running at the time of 'close' should be cancelled by force closing the IndexSearcher. If you were to do this every X minutes, then any running Searcher that was still being used after this time is probably a candidate for an interrupt. (In a web search, the user has probably navigated away and given up on the long running search anyway).

I'm updating nearly continuously (probably average about every 10

don't have any code keeping track of how many threads might be using a
given IndexSearcher.

I wonder if it's worth the effort of adding that code in, or can I be

handles will be released?

-- m@

seconds). I don't explicitly close the IndexSearcher objects I create, as I share them across threads, but do leave them to be garbage collected. I ran into index corruption issues when I explicitly closed them, since I ensured that when the IndexSearcher objects are garbage collected the file

How often are you updating your index?  Are you closing your old
IndexSearchers after switching over to the new index?  You'll need to

same issue I was experiencing:

close the searchers in order to release the file handle. This was the http://mail-archives.apache.org/mod_mbox/lucene-java-user/ 200504.mbox/%3

[EMAIL PROTECTED]

- Monsur

-----Original Message-----
From: Matt Magoffin [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 17, 2005 11:05 PM
To: java-user@lucene.apache.org
Subject: Re: References to deleted file handles in
long-running server application


I've been watching our servers today, and now there are 2500
"deleted" file handles open like this. Seems to be quite
large. Still don't know why there are so many. I'm using the
compound index format already to reduce the number of open files.

-- m@

Hello, I use Lucene in a long-running server application on a Linux
server, and the other day I got the "Too many open files"

exception.

I've increased the number of allowed file handles, but was checking
out the open file handles using "lsof", and see about 300

files listed

like the
following:

java       1458  jboss  436r   REG        8,2      3945   6258825
/var/lucene-index/_1o6hl.cfs (deleted)
java       1458  jboss  437r   REG        8,2      3467   6258829
/var/lucene-index/_1o6hp.cfs (deleted)
java       1458  jboss  438r   REG        8,2      2743   6258826
/var/lucene-index/_1o6ht.cfs (deleted)
java       1458  jboss  439r   REG        8,2      4069   6258817
/var/lucene-index/_1o6hx.cfs (deleted)
java       1458  jboss  440r   REG        8,2      4098   6258830
/var/lucene-index/_1o6i1.cfs (deleted)
java       1458  jboss  441r   REG        8,2      1817   6258821
/var/lucene-index/_1o6i3.cfs (deleted)

I haven't been monitoring this long enough to tell if the number
simply continues to grow over time or levels off at some point.

I was wondering if this behavior to be expected? When will the
application release these deleted file handles (if ever)? Perhaps
during garbage collection? Or might I be handling the index
incorrectly in some way?

Any thoughts appreciated.
-- m@

To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

-------------------------------------------------------------------- -

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

smime.p7s
Description: S/MIME cryptographic signature