Some excellent feedback guys - thanks heaps.
On my OOM issue, I think Hoss has nailed it here: > That said: if you are seeing OOM errors when you sort by a field (but > not when you use the docId ordering, or sort by score) then it sounds > like you are keeping refrences to IndexReaders arround after you've > stoped using them -- the FieldCache is kept in a WeakHashMap keyed off of > hte IndexReader, so it should get garbage collected as soon sa you let go > of it. Another possibility is that you are sorting on too many fields > for it to be able to build the FieldCache for all of them in the RAM you > have available. I'm using a piece of code written by Peter Halacsy which implements a SearcherCache class. When we do a search we request a searcher, and this class looks after giving us one. It checks whether the index has been updated since the most recent Searcher was created. If so it creates a new one. At the same time it 'retires' outdated Searchers, once they have no queries busy with them. Looking at that code, if the system gets busy indexing new stuff, and doing complex searches this is all rather open-ended as to the potential number of fresh Searchers being created, each with the overhead of building its FieldCache for the first time. No wonder I'm having problems as the archive has grown! Looking at it in this light, my OOM's all seem to come just after a bout of articles have been indexed, and querying is being done simultaneously, so it does fit. I guess a solution is probably to cap this process with a maximum number of active Searchers, meaning potentially some queries might be fobbed off with slightly out of date versions of the index, in extremis, but it would right itself once everything settles down again. Obviously the index partitioning would probably make this a non-issue, but it seems better to sort the basic problem out anyway, and make it 100% stable. Thanks Hoss! Cheers, Paul. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
