Aggregation on parent/child documents
Hi, I wanted to ask whether is possible to perform aggregations combining parent/child documents, something similar with the nested aggregation and the reverse nested aggregation. It would be very helpful to have the ability to create for instance buckets based on parent document fields and get back aggregations that contain fields of both parent and children documents combined. Any thoughts, future features to be added in the near releases, related to the above? Thank you Thomas -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91d60d52-c538-45b5-8cf0-91cb1e9d9a9a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Aggregation on parent/child documents
Hi Thomas, None of the aggregations that we have today can leverage parent/child relations. However, there is a `children` aggregation in the pipeline: https://github.com/elasticsearch/elasticsearch/pull/6936 On Fri, Jul 25, 2014 at 1:54 PM, Thomas thomas.bo...@gmail.com wrote: Hi, I wanted to ask whether is possible to perform aggregations combining parent/child documents, something similar with the nested aggregation and the reverse nested aggregation. It would be very helpful to have the ability to create for instance buckets based on parent document fields and get back aggregations that contain fields of both parent and children documents combined. Any thoughts, future features to be added in the near releases, related to the above? Thank you Thomas -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91d60d52-c538-45b5-8cf0-91cb1e9d9a9a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/91d60d52-c538-45b5-8cf0-91cb1e9d9a9a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6s3BM-fS-LdZU0hfdBBwAYBaVGpi3j95xzhBGsckrpgg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Aggregation on parent/child documents
Hi Adrien and thank you for the reply, This is exactly what i had in mind alongside with the reversed search equivalent with the reverse_nested, this is planed for version 1.4.0 onwards as i see, will keep track of any updates on this, thanks Thomas On Friday, 25 July 2014 14:54:50 UTC+3, Thomas wrote: Hi, I wanted to ask whether is possible to perform aggregations combining parent/child documents, something similar with the nested aggregation and the reverse nested aggregation. It would be very helpful to have the ability to create for instance buckets based on parent document fields and get back aggregations that contain fields of both parent and children documents combined. Any thoughts, future features to be added in the near releases, related to the above? Thank you Thomas -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6c7dfa1-d8b1-4ce5-8046-73892f74b33e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Aggregation on parent/child documents
We run 4 instances of ES 1.0.0 using 30G for JVM. We run 64-bit OpenJDK 1.7.0_25 on ubuntu servers. $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515139 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 64000 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 515139 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited And I also disabled swap on linux. You can use this gist to simulate the issue we have: https://gist.github.com/chaos-generator/9143655 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6db68fc-a7c8-43af-bbc4-59a0866aba36%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Aggregation on parent/child documents
Hello ElasticSearchers! I've been working on a BI system with ES 0.90 and we needed count users which have certain attributes, for instance let's say gender and star sign. A user is a parent-level document and the attributes are child documents. Fro the sample above, we were doing so by creating a query for each combination of male / female and the star signs and querying individually, as one can imagine, this was slow, but the results are exactly what we want. We could run this in roughly 2 minutes. We considered using the msearch query to get these results in a single query and we ended up with something similar to this: https://gist.github.com/chaos-generator/9133118 The sample above runs in 40 seconds give or take. And along came elastic search 1.0.0 and now we have aggregations, so we simplified our query to this: https://gist.github.com/chaos-generator/9133139 This runs lightning fast and we get the results in 200ms on average, which is ideal for us, BUT we get the total number of documents with the attributes, rather than the count on the parent documents. Our problem, as you can see in the msearch gist, is that we have a parent level document and child documents, which would only be updated if another document with the exact same attributes came in, this means that a parent level user document can have three child documents that will have gender and star sign, but I only want to count the parent document, rather than each individual child document. As we don't know in advance the attributes our users will be searching, we cannot use a script in index time to help us do this aggregation. We tried to use a script in search time like this: https://gist.github.com/chaos-generator/9133321 , but it didn't work as we wanted too: Any suggestions would be greatly appreciated. Best regards, Augusto -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3d9a0683-efa3-4398-b546-087005ab2b67%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Aggregation on parent/child documents
I'm wondering if the filter aggregation will work for you: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html However, it does not support parent child, but if you have the children embedded directly inside the parent document, I think it should be similar in functionality to your _msearch solution. BTW, if you are only doing aggregations or counts and don't really need search hits returned, you can further optimize by using the count search type: _search?search_type=count -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/262d7ef2-e794-4927-b6d9-cb021fee3b00%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Aggregation on parent/child documents
Thank you for your reply Binh, I've tried the bucket filter, but had problems with parent/child relationships. I've modified the multi-search query to use type = count, but the performance didn't change much, it took about 40 seconds to return the results. It was almost 20% faster indeed, but it is not the performance we want yet. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/569b670e-d1d7-4edb-81bf-199f24cce552%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
