Problem solved. "indices.fielddata.cache.size:2GB" doesn't mean use 2GB for caching it means use up to 134MB (2GB / 16) for each of the 16 segments within the cache. Due to the particular combination of queries, shard / node and the eviction pattern, I can explain all of the perf results I have documented.
As a short term solution I made the "size" 4.5GB, but that will mean about 300mb per segment. The right solution is to fix GUAVA to allow segments to be overcommitted and then do an async eviction across segments to ensure the overall size remains close to the target. Craig. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d1281de-55a6-48c1-a2a9-567ab8605373%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.