On 3/16/2016 8:59 AM, Patrick Plaatje wrote: > From the sar output you supplied, it looks like you might have a memory issue > on your hosts. The memory usage just before your crash seems to be *very* > close to 100%. Even the slightest increase (Solr itself, or possibly by a > system service) could caused the system crash. What are the specifications of > your hosts and how much memory are you allocating?
It's completely normal for a machine, especially a machine running Solr with a very large index, to run at nearly 100% memory usage. The "Average" line from the sar output indicates 97.45 percent usage, but it also shows 81GB of memory in the "kbcached" column -- this is memory that can be instantly claimed by any program that asks for it. If we discount this 81GB, since it is instantly available, the "true" memory usage is closer to 70 percent than 100. https://en.wikipedia.org/wiki/Page_cache If YouPeng can run top and sort it by memory usage (press shift-M), then grab a screenshot, that will be helpful for more insight. Here's an example of this from one of my servers, shared on dropbox: https://www.dropbox.com/s/qfuxhw20q0y1ckx/linux-8gb-heap.png?dl=0 This is a server with 64GB of RAM and 110GB of index data. About 48GB of my memory is used by the disk cache. I've got slightly less than half my index data in the cache. Thanks, Shawn
