The biggest benefit I see of using the field cache to do filter
caching, is that the same cache can be used for sorting - thereby
improving the performance and memory usage.
The downside I see is that if you have a common filter that is built
from many fields, you are going to use a lot more memory, as every
field used needs to be cached. With my code you would only have a
single "bitset" for the filter.
On Dec 4, 2008, at 4:00 PM, robert engels wrote:
Lucene-831 is far more comprehensive.
I also think that by exposing access to the sub-readers it can be
far simpler (closer to what I have provided).
In the mean-time, you should be able to use the provided class with
a few modifications.
The "reload the entire cache" was a deal breaker for us, so I came
up the attached. Works very well.
On Dec 4, 2008, at 3:54 PM, Uwe Schindler wrote:
I am looking all the time to LUCENE-831, which is a new version of
FieldCache that is compatible with IndexReader.reopen() and
invalidates only
reloaded segments. In each release of Lucene I am very unhappy,
because it
is still not in. The same problem like yours is if you have a one
million
documents index that is updated by adding a few documents each
half hour. If
you use sorting by a field, whenever the index is reopened and you
really
only a very small segment is added, nevertheless the complete
FieldCache is
rebuild, very bad :(.
So I think the ultimative fix would be to hopefully apply
LUCENE-831 soon
and also use LUCENE-1461 as RangeFilter cache.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
________________________________________
From: robert engels [mailto:[EMAIL PROTECTED]
Sent: Thursday, December 04, 2008 9:39 PM
To: java-dev@lucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-855)
MemoryCachedRangeFilter to boost
performance of Range queries
I can't seem to post to Jira, so I am attaching here...
I attached QueryFilter.java.
In reading this patch, and other similar ones, the problem seems
to be that
if the index is modified, the cache is invalidated, causing a
complete
reload of the cache. Do I have this correct?
The attached patch works really well in a highly interactive
environment, as
the cache is only invalidated at the segment level.
The MyMultiReader is a subclass that allows access to the underlying
SegmentReaders.
The patch cannot be applied, but I think the implementation works
far better
in many cases - it is also far less memory intensive. Scanning the
bitset
could also be optimized very easily using internal skip values.
Maybe this is completely off-base, but the solution has worked
very well for
us. Maybe this is a completely different issue and separate
incident should
be opened ?
is there any interest in this?
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
