The second approach (newSearcher listener) is a nice approach if the filter cache is too full to rely on auto-warming. Static warming queries fail on cross core joins but I believe succeed on a self core join. So you could move the inventory into the same core and use a static warming query. The downside to this is the pollution of the main index with ever changing inventory segments.
Joel Bernstein http://joelsolr.blogspot.com/ On Sun, Dec 19, 2021 at 6:10 PM David Smiley <[email protected]> wrote: > I'm not sure there is a clean/simple solution to this specific problem. > But I could imagine a more general & simple feature that could solve this > scenario, with just a bit more work by the user. > > Imagine an optional cache-key on ExtendedQuery auto-parsed, perhaps with > local-param "cacheKey". It would wrap any Query with one having a special > equals & hashcode on this key. Solr wouldn't parse the string for > this query so long as it can look it up in a special cache of these. That > special cache would be Map<String,Query> with weak values such that if it's > not used anymore (e.g. not in the filter cache), it would be GC'ed. This > would be useful for expensive queries that might resolve from some > network location (e.g. access control filters that refer to data in > who-knows-where). So that's useful on its own but doesn't solve your > conundrum. Then, imagine some new request handler that allows you to > provide this key & query and have it perform a filter cache save, > overwriting whatever entry that may have been there. You could even do > this in a newSearcher event on the inventory core, calling into the primary > product core. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Tue, Dec 14, 2021 at 4:24 PM Mikhail Khludnev <[email protected]> wrote: > >> Hello, Colleagues. >> I want to discuss one frequent usecase: inventory updates. >> Let's say we can't reindex docs when inventory numbers updated. We can >> put inventory in separate index, and apply fq={!join .. >> fromIndex=inventory}left:(0 TO *]. Once it's cached in main index filter >> cache it gets a good response time. We can even shard main collection, but >> keep inventory single shard. Ok. >> The sad moment occurs when commit goes into inventory core, after >> searcher is refreshed it's going to be cache misses on those inventory >> queries, and many of them go into new inventory searcher. That's not good. >> I can think of two workarounds: >> - relax {!join} equality regarding fromIndex timestamp, so for some time >> it will be outdated inventory, but it's ok. And then we need to somehow, >> evict, invalidate, regenerate inventory filter >> - newSearcher listener in inventory core can introspect main core cache >> entries find {!join .. fromIndex=inventory}... regenerate and insert >> results. >> I'm afraid to think about queryResult cache. >> >> Is it worth to have something like this in Solr distro? >> >> -- >> Sincerely yours >> Mikhail Khludnev >> >
