On 9/7/2018 8:39 AM, Pavel Micka wrote:
I found on wiki (https://wiki.apache.org/solr/SolrPerformanceProblems#RAM) that optimal amount of RAM for SOLR is equal to index size. This is lets say the ideal case to have everything in memory.
I wrote that page.
We plan to have small installation with 2 nodes and 8shards. We'll have inside the cluster 100M of documents. We expect that each document will take 5kB to index. With in-memory index this would mean that those two nodes would require ~500GB RAM. This would mean 2x 256GB to have everything in memory. And those are really big machines... Is this calculation even correct in new Solr versions? And we do have a bit restricted problem: Our data are time based logs and we generally have a restricted search for last 3 months. Which will match let's say 10M of documents. How will this affect SOLR memory requirements? Will we still need to have the whole inverted indexes in memory? Or is there some internal optimization, which will ensure that only some part will need to be in memory? The questions: 1) Is the 500GB of memory reqs correct assumption?
There are two things that Solr needs memory for. One is Solr's heap, which is memory directly used by Solr itself. The other is unused memory, which the operating system will use to cache data on disk. Solr performance is helped dramatically by the latter kind of memory.
For *OPTIMAL* performance with a 500GB index, you need 500GB of memory for the OS to cache the data. This is memory that is not used by programs, including Solr's heap.
For *good* performance, it's rare that you will need enough memory to cache the entire index. But I cannot tell you with any reliability how much of the index you must be able to cache. Some people are doing fine with only a few percent of their index cached. Others see terrible performance unless they can get 75 percent of the index cached.
2) Will the fact that we have time-based logs with majority of accesses to recent data only help?
Yes, it most likely will help, and reduce your memory requirements.
3) Is there some best practice how to reduce required RAM in Solr?
The biggest thing you can do is to reduce the size of the index, so there is less data that must be accessed for a query. The page you referenced lists some things you might be able to do to reduce Solr's heap requirements. If you reduce the heap requirements, then more of the server's memory available for caching.
Thanks, Shawn
