It would be cool to be able to explicitly list subreaders that were added/removed as a result of reopen(), or have some kind of notification mechanism. We have filter caches, custom field/sort caches here and they are all reader-bound. Currently warm-up delay is negated by reopening and warming up in background, before switching to the new reader/caches, but it still limits our minimum between-reopens delay.
On Fri, Dec 5, 2008 at 01:03, robert engels <[EMAIL PROTECTED]> wrote: > The biggest benefit I see of using the field cache to do filter caching, is > that the same cache can be used for sorting - thereby improving the > performance and memory usage. > > The downside I see is that if you have a common filter that is built from > many fields, you are going to use a lot more memory, as every field used > needs to be cached. With my code you would only have a single "bitset" for > the filter. > > On Dec 4, 2008, at 4:00 PM, robert engels wrote: > >> Lucene-831 is far more comprehensive. >> >> I also think that by exposing access to the sub-readers it can be far >> simpler (closer to what I have provided). >> >> In the mean-time, you should be able to use the provided class with a few >> modifications. >> >> The "reload the entire cache" was a deal breaker for us, so I came up the >> attached. Works very well. >> >> On Dec 4, 2008, at 3:54 PM, Uwe Schindler wrote: >> >>> I am looking all the time to LUCENE-831, which is a new version of >>> FieldCache that is compatible with IndexReader.reopen() and invalidates >>> only >>> reloaded segments. In each release of Lucene I am very unhappy, because >>> it >>> is still not in. The same problem like yours is if you have a one million >>> documents index that is updated by adding a few documents each half hour. >>> If >>> you use sorting by a field, whenever the index is reopened and you really >>> only a very small segment is added, nevertheless the complete FieldCache >>> is >>> rebuild, very bad :(. >>> >>> >>> So I think the ultimative fix would be to hopefully apply LUCENE-831 soon >>> and also use LUCENE-1461 as RangeFilter cache. >>> >>> ----- >>> Uwe Schindler >>> H.-H.-Meier-Allee 63, D-28213 Bremen >>> http://www.thetaphi.de >>> eMail: [EMAIL PROTECTED] >>> ________________________________________ >>> From: robert engels [mailto:[EMAIL PROTECTED] >>> Sent: Thursday, December 04, 2008 9:39 PM >>> To: java-dev@lucene.apache.org >>> Subject: Re: [jira] Commented: (LUCENE-855) MemoryCachedRangeFilter to >>> boost >>> performance of Range queries >>> >>> I can't seem to post to Jira, so I am attaching here... >>> >>> I attached QueryFilter.java. >>> >>> In reading this patch, and other similar ones, the problem seems to be >>> that >>> if the index is modified, the cache is invalidated, causing a complete >>> reload of the cache. Do I have this correct? >>> >>> The attached patch works really well in a highly interactive environment, >>> as >>> the cache is only invalidated at the segment level. >>> >>> The MyMultiReader is a subclass that allows access to the underlying >>> SegmentReaders. >>> >>> The patch cannot be applied, but I think the implementation works far >>> better >>> in many cases - it is also far less memory intensive. Scanning the bitset >>> could also be optimized very easily using internal skip values. >>> >>> Maybe this is completely off-base, but the solution has worked very well >>> for >>> us. Maybe this is a completely different issue and separate incident >>> should >>> be opened ? >>> >>> is there any interest in this? >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>> For additional commands, e-mail: [EMAIL PROTECTED] >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Kirill Zakharenko/Кирилл Захаренко ([EMAIL PROTECTED]) Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423 ICQ: 104465785
