OK so in this approach, a CSF is an "on disk" format, while the
FieldCache represents loading all (or maybe eventually subsets as
controlled by a cache policy) into a memory cache. And since they
both implement FSV you can swap either in when you need it.
This sounds good!
Mike
Michael Busch wrote:
Chris Hostetter wrote:
: But then the FieldCache is just starting to feel alot like
column-stride
: fields
: (LUCENE-1231).
that's what i've been thinking ... my goal with LUCENE-831 was to
make it easier to manage FieldCache and hopefully the norms[] as
well particularly in the case of reopen ... but with column-stride
fields the need for both of those might go away completely)
(moved to java-dev, java-user cc'd)
My goal is it not to get rid of the FieldCache by adding column-
stride fields (CSF), but instead to make them the default source
for the FieldCache.
We should introduce an interface, named maybe FieldValueSource,
that the new FieldCache implements, and also the CSF API. That has
some advantages:
- Norms can be stored as CSF, and can be accessed using the
FieldValueSource API. Then we can easily add an option to
IndexReader whether to cache norms in memory (i. e. the new
FieldCache) or not. When users have huge indexes on 32bit machines,
where the norms would consume too much memory, they can disable
caching them, of course search performance will suffer (but that's
better than OutOfMemoryErrors)
- The function queries can use the FieldValueSource interface to
retrieve the values (allowing us to get rid of function/ValueSource).
- Any consumer of the FieldValueSource does not have to care about
whether or not values are cached and how. If performance is too
slow and memory permits, caching can be enabled very easily.
- We will still support loading the fieldcache from the dictionary
for backwards compatibility, but we should think about deprecating
this and eventually get rid of it. We probably shouldn't add an
implementation of FieldValueSource that reads from the dictionary,
because performance would be terrible in the non-cached mode.
-Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
