Re: cache warming questions

2014-04-18 Thread Kranti Parisa
cool, thanks.

Thanks,
Kranti K. Parisa
http://www.linkedin.com/in/krantiparisa



On Thu, Apr 17, 2014 at 11:37 PM, Erick Erickson wrote:

> No, the 5 most recently used in a query will be used to autowarm.
>
> If you have things you _know_ are going to be popular fqs, you could
> put them in newSearcher queries.
>
> Best,
> Erick
>
> On Thu, Apr 17, 2014 at 4:51 PM, Kranti Parisa 
> wrote:
> > Erik,
> >
> > I have a followup question on this topic.
> >
> > If we have used 10 unique FQs and when we configure filterCache=100 &
> > autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
> > new searcher?
> >
> > I don't think there is a way to set the preference or there is?
> >
> >
> > Thanks,
> > Kranti K. Parisa
> > http://www.linkedin.com/in/krantiparisa
> >
> >
> >
> > On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper 
> wrote:
> >
> >> Ok,  that makes sense.
> >>
> >> Thanks again,
> >> Matt
> >>
> >> Matt Kuiper - Software Engineer
> >> Intelligent Software Solutions
> >> p. 719.452.7721 | matt.kui...@issinc.com
> >> www.issinc.com | LinkedIn: intelligent-software-solutions
> >>
> >> -Original Message-
> >> From: Erick Erickson [mailto:erickerick...@gmail.com]
> >> Sent: Thursday, April 17, 2014 9:26 AM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: cache warming questions
> >>
> >> Don't go overboard warming here, you often hit diminishing returns very
> >> quickly. For instance, if the size is 512 you might set your autowarm
> count
> >> to 16 and get the most bang for your buck. Beyond some (usually small)
> >> number, the additional work you put in to warming is wasted. This is
> >> especially true if your autocommit (soft, or hard with
> >> openSearcher=true) is short.
> >>
> >> So while you're correct in your sizing bit, practically it's rarely that
> >> complicated since the autowarm count is usually so much smaller than the
> >> size that there's no danger of swapping them out. YMMV of course.
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper 
> >> wrote:
> >> > Thanks Erick, this is helpful information!
> >> >
> >> > So it sounds like, at minimum the cache size (at least for filterCache
> >> and queryResultCache) should be the sum of the autowarmCount for that
> cache
> >> and the number of queries defined for the newSearcher listener.
>  Otherwise
> >> some items in the caches will be evicted right away.
> >> >
> >> > Matt
> >> >
> >> > -Original Message-
> >> > From: Erick Erickson [mailto:erickerick...@gmail.com]
> >> > Sent: Tuesday, April 15, 2014 5:21 PM
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Re: cache warming questions
> >> >
> >> > bq: What does it mean that items will be regenerated or prepopulated
> >> from the current searcher's cache...
> >> >
> >> > You're right, the values aren't cached. They can't be since the
> internal
> >> Lucene document id is used to identify docs, and due to merging the
> >> internal ID may bear no relation to the old internal ID for a particular
> >> document.
> >> >
> >> > I find it useful to think of Solr's caches as a  map where the key is
> >> the "query" and the value is some representation of the found documents.
> >> The details of the value don't matter, so I'll skip them.
> >> >
> >> > What matters is the key. Consider the filter cache. You put something
> >> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the
> >> key to the filterCache.
> >> >
> >> > Here's the sneaky bit. When you specify an autowarm count of N for the
> >> filterCache, when a new searcher is opened the first N keys from the map
> >> are re-executed in the new searcher's context and the results put into
> the
> >> new searcher's filterCache.
> >> >
> >> > bq:  ...how does auto warming and explicit warming work together?
> >> >
> >> > They're orthogonal. IOW, the autowarming for each cache is executed as
> >> well as the newSearcher static warming queries. Use the static queries
> to
> >> do 

Re: cache warming questions

2014-04-17 Thread Erick Erickson
No, the 5 most recently used in a query will be used to autowarm.

If you have things you _know_ are going to be popular fqs, you could
put them in newSearcher queries.

Best,
Erick

On Thu, Apr 17, 2014 at 4:51 PM, Kranti Parisa  wrote:
> Erik,
>
> I have a followup question on this topic.
>
> If we have used 10 unique FQs and when we configure filterCache=100 &
> autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
> new searcher?
>
> I don't think there is a way to set the preference or there is?
>
>
> Thanks,
> Kranti K. Parisa
> http://www.linkedin.com/in/krantiparisa
>
>
>
> On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper  wrote:
>
>> Ok,  that makes sense.
>>
>> Thanks again,
>> Matt
>>
>> Matt Kuiper - Software Engineer
>> Intelligent Software Solutions
>> p. 719.452.7721 | matt.kui...@issinc.com
>> www.issinc.com | LinkedIn: intelligent-software-solutions
>>
>> -Original Message-
>> From: Erick Erickson [mailto:erickerick...@gmail.com]
>> Sent: Thursday, April 17, 2014 9:26 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: cache warming questions
>>
>> Don't go overboard warming here, you often hit diminishing returns very
>> quickly. For instance, if the size is 512 you might set your autowarm count
>> to 16 and get the most bang for your buck. Beyond some (usually small)
>> number, the additional work you put in to warming is wasted. This is
>> especially true if your autocommit (soft, or hard with
>> openSearcher=true) is short.
>>
>> So while you're correct in your sizing bit, practically it's rarely that
>> complicated since the autowarm count is usually so much smaller than the
>> size that there's no danger of swapping them out. YMMV of course.
>>
>> Best,
>> Erick
>>
>> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper 
>> wrote:
>> > Thanks Erick, this is helpful information!
>> >
>> > So it sounds like, at minimum the cache size (at least for filterCache
>> and queryResultCache) should be the sum of the autowarmCount for that cache
>> and the number of queries defined for the newSearcher listener.  Otherwise
>> some items in the caches will be evicted right away.
>> >
>> > Matt
>> >
>> > -Original Message-
>> > From: Erick Erickson [mailto:erickerick...@gmail.com]
>> > Sent: Tuesday, April 15, 2014 5:21 PM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: cache warming questions
>> >
>> > bq: What does it mean that items will be regenerated or prepopulated
>> from the current searcher's cache...
>> >
>> > You're right, the values aren't cached. They can't be since the internal
>> Lucene document id is used to identify docs, and due to merging the
>> internal ID may bear no relation to the old internal ID for a particular
>> document.
>> >
>> > I find it useful to think of Solr's caches as a  map where the key is
>> the "query" and the value is some representation of the found documents.
>> The details of the value don't matter, so I'll skip them.
>> >
>> > What matters is the key. Consider the filter cache. You put something
>> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the
>> key to the filterCache.
>> >
>> > Here's the sneaky bit. When you specify an autowarm count of N for the
>> filterCache, when a new searcher is opened the first N keys from the map
>> are re-executed in the new searcher's context and the results put into the
>> new searcher's filterCache.
>> >
>> > bq:  ...how does auto warming and explicit warming work together?
>> >
>> > They're orthogonal. IOW, the autowarming for each cache is executed as
>> well as the newSearcher static warming queries. Use the static queries to
>> do things like fill the sort caches etc.
>> >
>> > Incidentally, this bears on why there's a "firstSearcher" and
>> "newSearcher". The newSearcher queries are run in addition to the cache
>> autowarms. firstSearcher static queries are only run when a Solr server is
>> started the first time, and there are no cache entries to autowarm. So the
>> firstSearcher queries might be quite a bit more complex than newSearcher
>> queries.
>> >
>> > HTH,
>> > Erick
>> >
>> > On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper 
>> wrote:
>> >> Hello,
>> &

Re: cache warming questions

2014-04-17 Thread Kranti Parisa
Erik,

I have a followup question on this topic.

If we have used 10 unique FQs and when we configure filterCache=100 &
autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
new searcher?

I don't think there is a way to set the preference or there is?


Thanks,
Kranti K. Parisa
http://www.linkedin.com/in/krantiparisa



On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper  wrote:

> Ok,  that makes sense.
>
> Thanks again,
> Matt
>
> Matt Kuiper - Software Engineer
> Intelligent Software Solutions
> p. 719.452.7721 | matt.kui...@issinc.com
> www.issinc.com | LinkedIn: intelligent-software-solutions
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Thursday, April 17, 2014 9:26 AM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> Don't go overboard warming here, you often hit diminishing returns very
> quickly. For instance, if the size is 512 you might set your autowarm count
> to 16 and get the most bang for your buck. Beyond some (usually small)
> number, the additional work you put in to warming is wasted. This is
> especially true if your autocommit (soft, or hard with
> openSearcher=true) is short.
>
> So while you're correct in your sizing bit, practically it's rarely that
> complicated since the autowarm count is usually so much smaller than the
> size that there's no danger of swapping them out. YMMV of course.
>
> Best,
> Erick
>
> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper 
> wrote:
> > Thanks Erick, this is helpful information!
> >
> > So it sounds like, at minimum the cache size (at least for filterCache
> and queryResultCache) should be the sum of the autowarmCount for that cache
> and the number of queries defined for the newSearcher listener.  Otherwise
> some items in the caches will be evicted right away.
> >
> > Matt
> >
> > -Original Message-
> > From: Erick Erickson [mailto:erickerick...@gmail.com]
> > Sent: Tuesday, April 15, 2014 5:21 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: cache warming questions
> >
> > bq: What does it mean that items will be regenerated or prepopulated
> from the current searcher's cache...
> >
> > You're right, the values aren't cached. They can't be since the internal
> Lucene document id is used to identify docs, and due to merging the
> internal ID may bear no relation to the old internal ID for a particular
> document.
> >
> > I find it useful to think of Solr's caches as a  map where the key is
> the "query" and the value is some representation of the found documents.
> The details of the value don't matter, so I'll skip them.
> >
> > What matters is the key. Consider the filter cache. You put something
> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the
> key to the filterCache.
> >
> > Here's the sneaky bit. When you specify an autowarm count of N for the
> filterCache, when a new searcher is opened the first N keys from the map
> are re-executed in the new searcher's context and the results put into the
> new searcher's filterCache.
> >
> > bq:  ...how does auto warming and explicit warming work together?
> >
> > They're orthogonal. IOW, the autowarming for each cache is executed as
> well as the newSearcher static warming queries. Use the static queries to
> do things like fill the sort caches etc.
> >
> > Incidentally, this bears on why there's a "firstSearcher" and
> "newSearcher". The newSearcher queries are run in addition to the cache
> autowarms. firstSearcher static queries are only run when a Solr server is
> started the first time, and there are no cache entries to autowarm. So the
> firstSearcher queries might be quite a bit more complex than newSearcher
> queries.
> >
> > HTH,
> > Erick
> >
> > On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper 
> wrote:
> >> Hello,
> >>
> >> I have a few questions regarding how Solr caches are warmed.
> >>
> >> My understanding is that there are two ways to warm internal Solr
> caches (only one way for document cache and lucene FieldCache):
> >>
> >> Auto warming - occurs when there is a current searcher handling
> requests and new searcher is being prepared.  "When a new searcher is
> opened, its caches may be prepopulated or "autowarmed" with cached object
> from caches in the old searcher. autowarmCount is the number of cached
> items that will be regenerated in the new searcher."
> http://wiki.apache.org/solr

RE: cache warming questions

2014-04-17 Thread Matt Kuiper
Ok,  that makes sense.

Thanks again,
Matt

Matt Kuiper - Software Engineer
Intelligent Software Solutions
p. 719.452.7721 | matt.kui...@issinc.com 
www.issinc.com | LinkedIn: intelligent-software-solutions

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Thursday, April 17, 2014 9:26 AM
To: solr-user@lucene.apache.org
Subject: Re: cache warming questions

Don't go overboard warming here, you often hit diminishing returns very 
quickly. For instance, if the size is 512 you might set your autowarm count to 
16 and get the most bang for your buck. Beyond some (usually small) number, the 
additional work you put in to warming is wasted. This is especially true if 
your autocommit (soft, or hard with
openSearcher=true) is short.

So while you're correct in your sizing bit, practically it's rarely that 
complicated since the autowarm count is usually so much smaller than the size 
that there's no danger of swapping them out. YMMV of course.

Best,
Erick

On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper  wrote:
> Thanks Erick, this is helpful information!
>
> So it sounds like, at minimum the cache size (at least for filterCache and 
> queryResultCache) should be the sum of the autowarmCount for that cache and 
> the number of queries defined for the newSearcher listener.  Otherwise some 
> items in the caches will be evicted right away.
>
> Matt
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, April 15, 2014 5:21 PM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> bq: What does it mean that items will be regenerated or prepopulated from the 
> current searcher's cache...
>
> You're right, the values aren't cached. They can't be since the internal 
> Lucene document id is used to identify docs, and due to merging the internal 
> ID may bear no relation to the old internal ID for a particular document.
>
> I find it useful to think of Solr's caches as a  map where the key is the 
> "query" and the value is some representation of the found documents. The 
> details of the value don't matter, so I'll skip them.
>
> What matters is the key. Consider the filter cache. You put something like 
> &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the key to 
> the filterCache.
>
> Here's the sneaky bit. When you specify an autowarm count of N for the 
> filterCache, when a new searcher is opened the first N keys from the map are 
> re-executed in the new searcher's context and the results put into the new 
> searcher's filterCache.
>
> bq:  ...how does auto warming and explicit warming work together?
>
> They're orthogonal. IOW, the autowarming for each cache is executed as well 
> as the newSearcher static warming queries. Use the static queries to do 
> things like fill the sort caches etc.
>
> Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". 
> The newSearcher queries are run in addition to the cache autowarms. 
> firstSearcher static queries are only run when a Solr server is started the 
> first time, and there are no cache entries to autowarm. So the firstSearcher 
> queries might be quite a bit more complex than newSearcher queries.
>
> HTH,
> Erick
>
> On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper  wrote:
>> Hello,
>>
>> I have a few questions regarding how Solr caches are warmed.
>>
>> My understanding is that there are two ways to warm internal Solr caches 
>> (only one way for document cache and lucene FieldCache):
>>
>> Auto warming - occurs when there is a current searcher handling requests and 
>> new searcher is being prepared.  "When a new searcher is opened, its caches 
>> may be prepopulated or "autowarmed" with cached object from caches in the 
>> old searcher. autowarmCount is the number of cached items that will be 
>> regenerated in the new searcher."
>> http://wiki.apache.org/solr/SolrCaching#autowarmCount
>>
>> Explicit warming - where the static warming queries specified in 
>> Solrconfig.xml for newSearcher and firstSearcher listeners are executed when 
>> a new searcher is being prepared.
>>
>> What does it mean that items will be regenerated or prepopulated from the 
>> current searcher's cache to the new searcher's cache?  I doubt it means 
>> copy, as the index has likely changed with a commit and possibly invalidated 
>> some contents of the cache.  Are the queries, or filters, that define the 
>> contents of the current caches re-executed for the new searcher's caches?
>>
>> For the case where auto warming is configured, a current searcher is active, 
>> and static warming queries are defined how does auto warming and explicit 
>> warming work together? Or do they?  Is only one type of warming activated to 
>> fill the caches?
>>
>> Thanks,
>> Matt


Re: cache warming questions

2014-04-17 Thread Erick Erickson
Don't go overboard warming here, you often hit diminishing returns
very quickly. For instance, if the size is 512 you might set your
autowarm count to 16 and get the most bang for your buck. Beyond some
(usually small) number, the additional work you put in to warming is
wasted. This is especially true if your autocommit (soft, or hard with
openSearcher=true) is short.

So while you're correct in your sizing bit, practically it's rarely
that complicated since the autowarm count is usually so much smaller
than the size that there's no danger of swapping them out. YMMV of
course.

Best,
Erick

On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper  wrote:
> Thanks Erick, this is helpful information!
>
> So it sounds like, at minimum the cache size (at least for filterCache and 
> queryResultCache) should be the sum of the autowarmCount for that cache and 
> the number of queries defined for the newSearcher listener.  Otherwise some 
> items in the caches will be evicted right away.
>
> Matt
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, April 15, 2014 5:21 PM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> bq: What does it mean that items will be regenerated or prepopulated from the 
> current searcher's cache...
>
> You're right, the values aren't cached. They can't be since the internal 
> Lucene document id is used to identify docs, and due to merging the internal 
> ID may bear no relation to the old internal ID for a particular document.
>
> I find it useful to think of Solr's caches as a  map where the key is the 
> "query" and the value is some representation of the found documents. The 
> details of the value don't matter, so I'll skip them.
>
> What matters is the key. Consider the filter cache. You put something like 
> &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the key to 
> the filterCache.
>
> Here's the sneaky bit. When you specify an autowarm count of N for the 
> filterCache, when a new searcher is opened the first N keys from the map are 
> re-executed in the new searcher's context and the results put into the new 
> searcher's filterCache.
>
> bq:  ...how does auto warming and explicit warming work together?
>
> They're orthogonal. IOW, the autowarming for each cache is executed as well 
> as the newSearcher static warming queries. Use the static queries to do 
> things like fill the sort caches etc.
>
> Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". 
> The newSearcher queries are run in addition to the cache autowarms. 
> firstSearcher static queries are only run when a Solr server is started the 
> first time, and there are no cache entries to autowarm. So the firstSearcher 
> queries might be quite a bit more complex than newSearcher queries.
>
> HTH,
> Erick
>
> On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper  wrote:
>> Hello,
>>
>> I have a few questions regarding how Solr caches are warmed.
>>
>> My understanding is that there are two ways to warm internal Solr caches 
>> (only one way for document cache and lucene FieldCache):
>>
>> Auto warming - occurs when there is a current searcher handling requests and 
>> new searcher is being prepared.  "When a new searcher is opened, its caches 
>> may be prepopulated or "autowarmed" with cached object from caches in the 
>> old searcher. autowarmCount is the number of cached items that will be 
>> regenerated in the new searcher."
>> http://wiki.apache.org/solr/SolrCaching#autowarmCount
>>
>> Explicit warming - where the static warming queries specified in 
>> Solrconfig.xml for newSearcher and firstSearcher listeners are executed when 
>> a new searcher is being prepared.
>>
>> What does it mean that items will be regenerated or prepopulated from the 
>> current searcher's cache to the new searcher's cache?  I doubt it means 
>> copy, as the index has likely changed with a commit and possibly invalidated 
>> some contents of the cache.  Are the queries, or filters, that define the 
>> contents of the current caches re-executed for the new searcher's caches?
>>
>> For the case where auto warming is configured, a current searcher is active, 
>> and static warming queries are defined how does auto warming and explicit 
>> warming work together? Or do they?  Is only one type of warming activated to 
>> fill the caches?
>>
>> Thanks,
>> Matt


RE: cache warming questions

2014-04-16 Thread Matt Kuiper
Thanks Erick, this is helpful information!

So it sounds like, at minimum the cache size (at least for filterCache and 
queryResultCache) should be the sum of the autowarmCount for that cache and the 
number of queries defined for the newSearcher listener.  Otherwise some items 
in the caches will be evicted right away.

Matt 

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, April 15, 2014 5:21 PM
To: solr-user@lucene.apache.org
Subject: Re: cache warming questions

bq: What does it mean that items will be regenerated or prepopulated from the 
current searcher's cache...

You're right, the values aren't cached. They can't be since the internal Lucene 
document id is used to identify docs, and due to merging the internal ID may 
bear no relation to the old internal ID for a particular document.

I find it useful to think of Solr's caches as a  map where the key is the 
"query" and the value is some representation of the found documents. The 
details of the value don't matter, so I'll skip them.

What matters is the key. Consider the filter cache. You put something like 
&fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as the key to the 
filterCache.

Here's the sneaky bit. When you specify an autowarm count of N for the 
filterCache, when a new searcher is opened the first N keys from the map are 
re-executed in the new searcher's context and the results put into the new 
searcher's filterCache.

bq:  ...how does auto warming and explicit warming work together?

They're orthogonal. IOW, the autowarming for each cache is executed as well as 
the newSearcher static warming queries. Use the static queries to do things 
like fill the sort caches etc.

Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". 
The newSearcher queries are run in addition to the cache autowarms. 
firstSearcher static queries are only run when a Solr server is started the 
first time, and there are no cache entries to autowarm. So the firstSearcher 
queries might be quite a bit more complex than newSearcher queries.

HTH,
Erick

On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper  wrote:
> Hello,
>
> I have a few questions regarding how Solr caches are warmed.
>
> My understanding is that there are two ways to warm internal Solr caches 
> (only one way for document cache and lucene FieldCache):
>
> Auto warming - occurs when there is a current searcher handling requests and 
> new searcher is being prepared.  "When a new searcher is opened, its caches 
> may be prepopulated or "autowarmed" with cached object from caches in the old 
> searcher. autowarmCount is the number of cached items that will be 
> regenerated in the new searcher."
> http://wiki.apache.org/solr/SolrCaching#autowarmCount
>
> Explicit warming - where the static warming queries specified in 
> Solrconfig.xml for newSearcher and firstSearcher listeners are executed when 
> a new searcher is being prepared.
>
> What does it mean that items will be regenerated or prepopulated from the 
> current searcher's cache to the new searcher's cache?  I doubt it means copy, 
> as the index has likely changed with a commit and possibly invalidated some 
> contents of the cache.  Are the queries, or filters, that define the contents 
> of the current caches re-executed for the new searcher's caches?
>
> For the case where auto warming is configured, a current searcher is active, 
> and static warming queries are defined how does auto warming and explicit 
> warming work together? Or do they?  Is only one type of warming activated to 
> fill the caches?
>
> Thanks,
> Matt


Re: cache warming questions

2014-04-15 Thread Erick Erickson
bq: What does it mean that items will be regenerated or prepopulated
from the current searcher's cache...

You're right, the values aren't cached. They can't be since the
internal Lucene document id is used to identify docs, and due to
merging the internal ID may bear no relation to the old internal ID
for a particular document.

I find it useful to think of Solr's caches as a  map where the key is
the "query" and the value is some representation of the found
documents. The details of the value don't matter, so I'll skip them.

What matters is the key. Consider the filter cache. You put something
like &fq=price:[0 TO 100] on a URL. Solr then uses the fq  clause as
the key to the filterCache.

Here's the sneaky bit. When you specify an autowarm count of N for the
filterCache, when a new searcher is opened the first N keys from the
map are re-executed in the new searcher's context and the results put
into the new searcher's filterCache.

bq:  ...how does auto warming and explicit warming work together?

They're orthogonal. IOW, the autowarming for each cache is executed as
well as the newSearcher static warming queries. Use the static queries
to do things like fill the sort caches etc.

Incidentally, this bears on why there's a "firstSearcher" and
"newSearcher". The newSearcher queries are run in addition to the
cache autowarms. firstSearcher static queries are only run when a Solr
server is started the first time, and there are no cache entries to
autowarm. So the firstSearcher queries might be quite a bit more
complex than newSearcher queries.

HTH,
Erick

On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper  wrote:
> Hello,
>
> I have a few questions regarding how Solr caches are warmed.
>
> My understanding is that there are two ways to warm internal Solr caches 
> (only one way for document cache and lucene FieldCache):
>
> Auto warming - occurs when there is a current searcher handling requests and 
> new searcher is being prepared.  "When a new searcher is opened, its caches 
> may be prepopulated or "autowarmed" with cached object from caches in the old 
> searcher. autowarmCount is the number of cached items that will be 
> regenerated in the new searcher."
> http://wiki.apache.org/solr/SolrCaching#autowarmCount
>
> Explicit warming - where the static warming queries specified in 
> Solrconfig.xml for newSearcher and firstSearcher listeners are executed when 
> a new searcher is being prepared.
>
> What does it mean that items will be regenerated or prepopulated from the 
> current searcher's cache to the new searcher's cache?  I doubt it means copy, 
> as the index has likely changed with a commit and possibly invalidated some 
> contents of the cache.  Are the queries, or filters, that define the contents 
> of the current caches re-executed for the new searcher's caches?
>
> For the case where auto warming is configured, a current searcher is active, 
> and static warming queries are defined how does auto warming and explicit 
> warming work together? Or do they?  Is only one type of warming activated to 
> fill the caches?
>
> Thanks,
> Matt


cache warming questions

2014-04-15 Thread Matt Kuiper
Hello,

I have a few questions regarding how Solr caches are warmed.

My understanding is that there are two ways to warm internal Solr caches (only 
one way for document cache and lucene FieldCache):

Auto warming - occurs when there is a current searcher handling requests and 
new searcher is being prepared.  "When a new searcher is opened, its caches may 
be prepopulated or "autowarmed" with cached object from caches in the old 
searcher. autowarmCount is the number of cached items that will be regenerated 
in the new searcher."http://wiki.apache.org/solr/SolrCaching#autowarmCount

Explicit warming - where the static warming queries specified in Solrconfig.xml 
for newSearcher and firstSearcher listeners are executed when a new searcher is 
being prepared.

What does it mean that items will be regenerated or prepopulated from the 
current searcher's cache to the new searcher's cache?  I doubt it means copy, 
as the index has likely changed with a commit and possibly invalidated some 
contents of the cache.  Are the queries, or filters, that define the contents 
of the current caches re-executed for the new searcher's caches?

For the case where auto warming is configured, a current searcher is active, 
and static warming queries are defined how does auto warming and explicit 
warming work together? Or do they?  Is only one type of warming activated to 
fill the caches?

Thanks,
Matt