On 5/21/2018 8:25 AM, Kelly, Frank wrote:
We have an indexing heavy workload (we do more indexing than searching) and for those searches we do perform we have very few cache hits (25% of our index is in memory and the hit rate is < 0.1%)
Which cache are you looking at for that hitrate? How are you looking at it? What precisely do you see?
We are currently using r3.xlarge (memory optimized instances as we originally thought we’d have a higher cache hit rate) with EBS optimization to IOPs configurable EBS drives. Our EBS traffic bandwidth seems to work great so searches on disk are pretty fast. Now though we seem CPU bound and if/ when Solr CPU gets pegged for too long replication falls behind and then starts to recover which causes more usage and then eventually shards go “Down”.
A system with enough memory to fully cache the index would almost never need to actually read the disk. If there is a lot of disk activity, the machine may need more memory for the OS disk cache. You've said that 25 percent of your index is in memory. I've seen good performance with 10 percent, and terrible performance with 50 percent. A lot of factors will affect what percentage is needed.
What precisely are you looking at to determine that the machine is CPU-bound? Some of the things that people assume are evidence of CPU problems are actually evidence of I/O problems caused by not having enough memory.
Our key question: Scale up (fewer instances to manage) or Scale out (more instances to manage) and do we switch to compute optimized instances (the answer given our usage I assume is probably)
Generally if you want to handle a higher request rate, you need more machines -- scale out -- and a way to load balance requests. If each request takes too long when the request rate is low, that's probably an indication that you need to increase the resources available on each machine - scale up.
Memory is the most valuable resource in a Solr install. CPU is important, but adding CPU can't solve issues that require more memory.
Thanks, Shawn
