On Mon, Jun 2, 2008 at 1:49 PM, Bram de Jong <[EMAIL PROTECTED]> wrote:
> On Mon, Jun 2, 2008 at 10:13 AM, Erick Erickson <[EMAIL PROTECTED]> wrote:
>> But are you sure you're not just masking the problem? That is, your limit
>> may now be 90,000 queries...
>>
>> I always assume this kind of thing is a memory leak somewhere, have you
>> any tools to monitor your memory consumption and see if that's ever-rising?
>
> 1. is it possible to tell Solr to use more cache mem than is allocated
> to the VM? Will this crash Solr? What are the defaults that java runs
> with? This is my config script:
> http://iua-share.upf.edu/svn/nightingale/trunk/sandbox/solr/config/solrconfig.xml
Use smaller caches unless you need bigger ones for some reason.
<filterCache
class="solr.search.LRUCache"
size="65536"
initialSize="16384"
autowarmCount="16384"/>
If you aren't faceting, I'd make this smaller.
<!-- queryResultCache caches results of searches - ordered lists of
document ids (DocList) based on a query, a sort, and the range
of documents requested. -->
<queryResultCache
class="solr.LRUCache"
size="32768"
initialSize="32768"
autowarmCount="16384"/>
This is definitely too big. There's almost never a reason to autowarm
that many queries too. I could see setting the autowarm count
anywhere from 10 to 100. The query cache itself probably doesn't need
to be larger than a few hundred.
<!-- documentCache caches Lucene Document objects (the stored fields
for each document).
Since Lucene internal document ids are transient, this cache
will not be autowarmed. -->
<documentCache
class="solr.LRUCache"
size="32768"
initialSize="32768"
autowarmCount="0"/>
This is probably too big also, and probably doesn't buy you much.
I'd set it anywhere from 64 to 1024
-Yonik
> 2. I noticed that if I let the script keep running (after slowdown) it
> would crash eventually, giving an *if I recall correctly* Heap
> Allocation Exception or something like it. In the traceback I saw
> DisMax somewhere mentioned as well. If you want (if it is useful) I
> can run it again until it crashes and give the full stack trace.
>
> 3. after changing the VM mem, I ran it again and it went to well over
> 100K queries without any problems (I think I also noticed less disk
> access - but I can be wrong)
>
>
> I can monitor mem usage through top or through anything you would like
> me to run, ... If there are any particular tests to run let me know!
>
>
> - Bram
>
> PS:
> http://freesound.iua.upf.edu/blog/ has now more about my adventures with Solr
>
