Hi Adrien, Thanks for your response. On Sunday, March 29, 2015 at 10:34:07 PM UTC+2, Adrien Grand wrote: > > I think it depends what takes memory in your heap. For instance, fielddata > mainly uses very large primitive arrays, so the impact of compressed oops > is almost null. The same would be true for aggregations and p/c queries. On > the other hand the filter cache happily uses objects in order to model the > keys (filter/segment pairs) and values (sets of doc ids). >
I can absolutely agree that the number of references in JVM depend on the query characteristics and agree that in my case aggregations are widely used which confirms the number of big objects in the heap. So if you have a large filter cache that contains lots of small entries, > compressed oops could help a bit. This might be true for users who have > large collections of percolator queries too. That's right but without analysis and counting number of instances for both large filter cache and large number of percolator queries (depending what big number means here) it's hard to say what does it meant that compressed oops "could help a *bit**"* I agree this is very important and this is a reason why eg. elasticsearch > is moving to doc values by default on not_analyzed fields in 2.0: > https://github.com/elastic/elasticsearch/pull/10209 (which effectively > moves fielddata memory usage from the heap to the OS cache). I think doc values also have some overhead. When I was analyzing the heap dump I could meet a lot of WeakReferences which come from doc values. From one side they are only *weak* references. From other side I don't know if ES/Lucene remove them from heap by some mechanism or more and more of them are added to the heap and they are removed only when OOM happens and FullGC is triggered by JVM. > The main issue about large heaps is when they make major garbage > collections long enough so that they look like the node left from a cluster > perspective. I don't think there is a magical line since it also depends on > the hardware and the complexity of the graph of objects but it is probably > something to keep an eye on. > Again agree but this "long GC" is probably connected with Full GC which can STW long enough but as you also mentioned "don't think there is a magical line". -- Paweł Róg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c17cba43-b706-4a8b-802e-408f6e860fa5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
