Re: Filter aggregation and nested documents

2014-07-14 Thread x0ne
When fielddata is loaded, is it only that of which the aggregation job needs (items.question_label.raw in this case) or does it load the full _source of every match and extract the field? On Monday, April 28, 2014 9:04:09 PM UTC-4, Olivier B wrote: > > Thanks you. > OK, that's what I was fearin

Re: Filter aggregation and nested documents

2014-04-28 Thread Olivier B
Thanks you. OK, that's what I was fearing: the cache is loaded regardless of the filter condition. Which is a shame, even if we filter a lot, targeting only one document, we still need to fill up the cache! I will try to have a lot of RAM and see if I'm reaching a stable memory occupation and l

Re: Filter aggregation and nested documents

2014-04-28 Thread Binh Ly
You are correct. Unfortunately the fielddata is loaded for all docs regardless of filter condition. You can: 1) Add more RAM 2) Add more nodes (and shard your index out so that RAM usage will distributed across multiple nodes) 3) Use disk-based fielddata (fielddata will not be loaded into memo

Filter aggregation and nested documents

2014-04-28 Thread Olivier B
Hi all, I'm working with nested documents (like millions of documents) and I do aggregation on nested documents. And of course, I need to use filter aggregation (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html), however