On 12/14/2010 8:31 AM, Mark wrote:
Can anyone offer some advice on what some good settings would be for
an index or around 6 million documents totaling around 20-25gb? It
seems like when our index gets to this size our CPU load spikes
tremendously.
If you are adding, deleting, or updating documents on a regular basis, I
would bet that it's your autoWarmCount. You've told it that whenever
you do a commit, it needs to make up to 32768 queries against the new
index. That's very intense and time-consuming. If you are also
optimizing the index, the problem gets even worse. On the
documentCache, autowarm doesn't happen, so the 16384 specified there
isn't actually doing anything.
Below are my settings. I originally had much larger caches with equally
large autoWarmCounts ... reducing them to this level was the only way I
could get my autowarm time below 30 seconds on each index. If you go to
the admin page for your index and click on Statistics, then search for
"warmupTime" you'll see how long it took to do the queries. Later on
the page you'll also see this broken down on each cache.
Since I made the changes, performance is actually better now, not
worse. I have been experimenting with FastLRUCache versus LRUCache,
because I read that below a certain hitratio, the latter is better.
I've got 8 million documents in each shard, taking up about 15GB.
My mergeFactor is 16 and my ramBufferSize is 256MB. These really only
come into play when I do a full re-index, which is rare.
<filterCache
class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="32"
cleanupThread="true"
/>
<queryResultCache
class="solr.FastLRUCache"
size="1024"
initialSize="1024"
autowarmCount="64"
cleanupThread="true"
/>
<documentCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
cleanupThread="true"
/>
