On Mon, Jun 2, 2008 at 1:49 PM, Bram de Jong <[EMAIL PROTECTED]> wrote: > On Mon, Jun 2, 2008 at 10:13 AM, Erick Erickson <[EMAIL PROTECTED]> wrote: >> But are you sure you're not just masking the problem? That is, your limit >> may now be 90,000 queries... >> >> I always assume this kind of thing is a memory leak somewhere, have you >> any tools to monitor your memory consumption and see if that's ever-rising? > > 1. is it possible to tell Solr to use more cache mem than is allocated > to the VM? Will this crash Solr? What are the defaults that java runs > with? This is my config script: > http://iua-share.upf.edu/svn/nightingale/trunk/sandbox/solr/config/solrconfig.xml
Use smaller caches unless you need bigger ones for some reason. <filterCache class="solr.search.LRUCache" size="65536" initialSize="16384" autowarmCount="16384"/> If you aren't faceting, I'd make this smaller. <!-- queryResultCache caches results of searches - ordered lists of document ids (DocList) based on a query, a sort, and the range of documents requested. --> <queryResultCache class="solr.LRUCache" size="32768" initialSize="32768" autowarmCount="16384"/> This is definitely too big. There's almost never a reason to autowarm that many queries too. I could see setting the autowarm count anywhere from 10 to 100. The query cache itself probably doesn't need to be larger than a few hundred. <!-- documentCache caches Lucene Document objects (the stored fields for each document). Since Lucene internal document ids are transient, this cache will not be autowarmed. --> <documentCache class="solr.LRUCache" size="32768" initialSize="32768" autowarmCount="0"/> This is probably too big also, and probably doesn't buy you much. I'd set it anywhere from 64 to 1024 -Yonik > 2. I noticed that if I let the script keep running (after slowdown) it > would crash eventually, giving an *if I recall correctly* Heap > Allocation Exception or something like it. In the traceback I saw > DisMax somewhere mentioned as well. If you want (if it is useful) I > can run it again until it crashes and give the full stack trace. > > 3. after changing the VM mem, I ran it again and it went to well over > 100K queries without any problems (I think I also noticed less disk > access - but I can be wrong) > > > I can monitor mem usage through top or through anything you would like > me to run, ... If there are any particular tests to run let me know! > > > - Bram > > PS: > http://freesound.iua.upf.edu/blog/ has now more about my adventures with Solr >