On Dec 4, 2008, at 4:10 PM, Paul Elschot wrote:
Op Thursday 04 December 2008 23:03:40 schreef robert engels:
The biggest benefit I see of using the field cache to do filter
caching, is that the same cache can be used for sorting - thereby
improving the performance and memory usage.
Would it be possible to build such Filter caching into
CachingWrapperFilter instead of into QueryFilter?
Both filter caching and the field value caching will need
access to the underlying (segment?) readers.
I don't see why not. The QueryFilter extends from that... We are just
on a much older code base.
Not really sure why this hierarchy exists tough, as the only
extenders are QueryFilter, and CachingWrapperFilterHelper.
I would prefer QueryFilter, and then extend that as CachingQueryFilter.
I've always been taught that is you see the words Wrapper, or Helper,
there is probably a design problem, or at least a naming problem.
The downside I see is that if you have a common filter that is built
from many fields, you are going to use a lot more memory, as every
field used needs to be cached. With my code you would only have a
single "bitset" for the filter.
But with many ranges that would mean many bitsets, and
MemoryCachedRangeFilter only needs to cache the field values once
for any number of ranges. It's a tradeoff.
That was my point. I don't see the field based caching and the filter
based caching as solving the same problem to a degree. It is going to
depend on the actual usage - that is why I would like to support both.
Regards,
Paul Elschot
On Dec 4, 2008, at 4:00 PM, robert engels wrote:
Lucene-831 is far more comprehensive.
I also think that by exposing access to the sub-readers it can be
far simpler (closer to what I have provided).
In the mean-time, you should be able to use the provided class with
a few modifications.
The "reload the entire cache" was a deal breaker for us, so I came
up the attached. Works very well.
On Dec 4, 2008, at 3:54 PM, Uwe Schindler wrote:
I am looking all the time to LUCENE-831, which is a new version of
FieldCache that is compatible with IndexReader.reopen() and
invalidates only
reloaded segments. In each release of Lucene I am very unhappy,
because it
is still not in. The same problem like yours is if you have a one
million
documents index that is updated by adding a few documents each
half hour. If
you use sorting by a field, whenever the index is reopened and you
really
only a very small segment is added, nevertheless the complete
FieldCache is
rebuild, very bad :(.
So I think the ultimative fix would be to hopefully apply
LUCENE-831 soon
and also use LUCENE-1461 as RangeFilter cache.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
________________________________________
From: robert engels [mailto:[EMAIL PROTECTED]
Sent: Thursday, December 04, 2008 9:39 PM
To: java-dev@lucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-855)
MemoryCachedRangeFilter to boost
performance of Range queries
I can't seem to post to Jira, so I am attaching here...
I attached QueryFilter.java.
In reading this patch, and other similar ones, the problem seems
to be that
if the index is modified, the cache is invalidated, causing a
complete
reload of the cache. Do I have this correct?
The attached patch works really well in a highly interactive
environment, as
the cache is only invalidated at the segment level.
The MyMultiReader is a subclass that allows access to the
underlying SegmentReaders.
The patch cannot be applied, but I think the implementation works
far better
in many cases - it is also far less memory intensive. Scanning the
bitset
could also be optimized very easily using internal skip values.
Maybe this is completely off-base, but the solution has worked
very well for
us. Maybe this is a completely different issue and separate
incident should
be opened ?
is there any interest in this?
------------------------------------------------------------------
--- To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-------------------------------------------------------------------
-- To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
