On Sep 10, 2010, at 7:01 PM, Burton-West, Tom wrote: > We have noticed that when the first query hits Solr after starting it up, > memory use increases significantly, from about 1GB to about 16GB, and then as > queries are received it goes up to about 19GB at which point there is a Full > Garbage Collection which takes about 30 seconds and then memory use drops > back down to 16GB. Under a relatively heavy load, the full GC happens about > every 10-20 minutes. > > We are running 3 Solr shards under one Tomcat with 20GB allocated to the jvm. > Each shard has a total index size of about 400GB on and a tii size of about > 600MB and indexes about 650,000 full-text books. (The server has a total of > 72GB of memory, so we are leaving quite a bit of memory for the OS disk > cache). > > Is there some argument we could give the jvm so that it would collect garbage > more frequently? Or some other JVM tuning action that might reduce the amount > of time where Solr is waiting on GC? > > If we could get the time for each GC to take under a second, with the > trade-off being that GC would occur much more frequently, that would help us > avoid the occasional query taking more than 30 seconds at the cost of a > larger number of queries taking at least a second. >
What are your current GC settings? Also, I guess I'd look at ways you can reduce the heap size needed. Caching, field type choices, faceting choices. Also could try playing with the termIndexInterval which will load fewer terms into memory at the cost of longer seeks. At some point, though, you just may need more shards and the resulting smaller indexes. How many CPU cores do you have on each machine?
