I have faced similar issues and the culprit was filterCache when using maxRAMMB. More specifically on a sharded Solr cluster with lots of faceting during search (which makes use of the filterCache in a distributed setting) I noticed that maxRAMMB value was not respected. I had a value of 300MB set but I witnessed an instance sized a couple of GBs in a heap dump at some point. The thing that I found was that because the keys of the Map (BooleanQuery or something if I recall correctly) was not implementing the Accountable interface it was NOT taken into account when calculating the cache's size. But all that was on a 7.5 cluster using FastLRUCache.
There's also https://issues.apache.org/jira/browse/SOLR-12743 on caches memory leak which does not seem to have been fixed yet although the trigger points of this memory leak are not clear. I've witnessed this as well on a 7.5 cluster with multiple (>10) filter cache objects for a single core each holding from a few MBs to GBs. Try to get a heap dump from your cluster, the truth is almost always hidden there. One workaround which seems to alleviate the problem is to check you running Solr cluster and see in reality how many cache entries actually give you a good hit ratio and get rid of the maxRAMMB attribute. Play only with the size. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html