On 11/22/2013 8:13 AM, Dave Seltzer wrote:
Regarding memory: Including duplicate data in shard replicas the entire index is 350GB. Each server hosts a total of 44GB of data. Each server has 28GB of memory. I haven't been setting -Xmx or -Xms, in the hopes that Java would take the memory it needs and leave the rest to the OS for cache.
That's not how Java works. Java has a min heap and max heap setting. If you (or the auto-detected settings) tell it that the max heap is 4GB, it will only ever use slightly more than 4GB of RAM. If the app needs more than that, this will lead to terrible performance and/or out of memory errors.
You can see how much the max heap is in the Solr admin UI dashboard - it'll be the right-most number on the JVM-Memory graph. On my 64-bit linux development machine with 16GB of RAM, it looks like Java defaults to a 4GB max heap. I have the heap size manually set to 7GB for Solr on that machine. The 6GB heap you have mentioned might not be enough, or it might be more than you need. It all depends on the kind of queries you are doing and exactly how Solr is configured.
If it were me, I'd want a memory size between 48 and 64GB for a total index size of 44GB. Whether you really need that much is very dependent on your exact requirements, index makeup, and queries. To support the high query load you're sending, it probably is a requirement. More memory is likely to help performance, but I can't guarantee it without looking a lot deeper into your setup, and that's difficult to do via email.
One thing I can tell you about checking performance - see how much of your 70% CPU usage is going to I/O wait. If it's more than a few percent, more memory might help. First try increasing the max heap by 1 or 2GB.
Given that I'll never need to serve 200 concurrent connections in production, do you think my servers need more memory? Should I be tinkering with -Xmx and -Xms?
If you'll never need to serve that many, test with a lower number. Make it higher than you'll need, but not a lot higher. The test with 200 connections isn't a bad idea -- you do want to stress test things way beyond your actual requirements, but you'll also want to see how it does with a more realistic load.
Those are the min/max heap settings I just mentioned. IMHO you should set at least the max heap. If you want to handle a high load, it's a good idea to set the min heap to the same value as the max heap, so that it doesn't need to worry about hitting limits in order to allocate additional memory. It'll eventually allocate the max heap anyway.
Regarding commits: My end-users want new data to be made available quickly. Thankfully I'm only inserting between 1 and 3 documents per second so the change-rate isn't crazy. Should I just slow down my commit frequency, and depend on soft-commits? If I do this, will the commits take even longer? Given 1000 documents, is it generally faster to do 10 commits of 100, or 1 commit of 1000?
Fewer commits is always better. The amount of time they take isn't strongly affected by the number of new documents, unless there are a LOT of them. Figure out the timeframe that's the maximum amount of time (in milliseconds) that you think people are willing to wait for new data to become visible. Use that as your autoSoftCommit interval, or as the commitWithin parameter on your indexing requests. Set your autoCommit interval to around five minutes, as described on the wiki page I linked. If you are using auto settings and/or commitWithin, then you will never need to send an explicit commit command. Reducing commit frequency is one of the first things you'll want to try. Frequent commits use a *lot* of I/O and CPU resources.
Although there are exceptions, most installs rarely NEED commits to happen more often than about once a minute, and longer intervals are often perfectly acceptable. Even in situations where a higher frequency is required, 10-15 seconds is often good enough. Getting sub-second commit times is *possible*, but usually requires significant hardware investment or changing the config in a way that is detrimental to query performance.
Thanks, Shawn
