Firstly, I think the reason you are only getting results from one index when you are asking for a size of 1 in your terms aggregation is because you are asking for the top 1 bucket from each shard on each index. This will then be merged together and only the top bucket will be kept. If the top bucket is not the same on all indexes then you will not get results from all indices. Setting the shard_size parameter to something like 10 can help with this (see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate for more information on this)
Second, I wonder if the reason you are getting the error from your script is that you don't have a 'datetime' value for all of your documents in some of your indices? Regards, Colin On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote: > > Hi Colin, > > now it gets really strange. First my alias > curl 'http://localhost:9200/_alias?pretty' > { > "live-2014-07-27" : { > > "aliases" : { > > "aggtest" : { } > > } > > }, > > "live-2014-07-26" : { > > "aliases" : { > > "aggtest" : { } > > } > > } > > } > > > I tried two different queries: > curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d > '{ > "size": 0, > "aggs": { > "winners": { > "terms": { > "field": "tit", > "order": { > "diff": "desc" > }, > "size": 1 > }, > "aggs": { > "articles_over_time": { > "date_histogram": { > "field": "datetime", > "interval": "1d" > } > }, > "diff": { > "sum": { > "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1", > "lang": "groovy" > } > } > } > } > } > }' > > and > > curl -XPOST ' > http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true' > > ..... > > both do give me a result (but a wrong one) when I do query using > elasticsearch-head but result in an error if I use the commandline > > { > > "error" : "SearchPhaseExecutionException[Failed to execute phase > [query], all shards failed; shardFailures > {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]: > QueryPhaseExecutionException[[live-2014-07-26][1]: > query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed > [Failed to execute main query]]; nested: > GroovyScriptExecutionException[MissingPropertyException[No such property: > datetime for class: Script126]]; > }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]: > RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]]; > > nested: QueryPhaseExecutionException[[live-2014-07-27][0]: > query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed > [Failed to execute main query]]; nested: > GroovyScriptExecutionException[MissingPropertyException[No such property: > datetime for class: Script119]]; > }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]: > QueryPhaseExecutionException[[live-2014-07-27][1]: > query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed > [Failed to execute main query]]; nested: > GroovyScriptExecutionException[MissingPropertyException[No such property: > datetime for class: Script126]]; > }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]: > RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]]; > > nested: QueryPhaseExecutionException[[live-2014-07-26][0]: > query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed > [Failed to execute main query]]; nested: > GroovyScriptExecutionException[MissingPropertyException[No such property: > datetime for class: Script119]]; }]", > > "status" : 500 > > } > > > But I noticed something strange. This works: > curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d > '{ > "size": 0, > "aggs": { > "winners": { > "terms": { > "field": "tit" > }, > "aggs": { > "articles_over_time": { > "date_histogram": { > "field": "datetime", > "interval": "1d" > } > } > } > } > } > }' > result: > > { > > "took" : 26, > > "timed_out" : false, > > "_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 89419, > > "max_score" : 0.0, > > "hits" : [ ] > > }, > > "aggregations" : { > > "winners" : { > > "buckets" : [ { > > "key" : "videotitle", > > "doc_count" : 3539, > > "articles_over_time" : { > > "buckets" : [ { > > "key_as_string" : "2014-07-26T00:00:00.000Z", > > "key" : 1406332800000, > > "doc_count" : 2820 > > }, { > > "key_as_string" : "2014-07-27T00:00:00.000Z", > > "key" : 1406419200000, > > "doc_count" : 719 > > } ] > > } > > }, { > > But this does not: (notice the size-limit to 1) > curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d > '{ > "size": 0, > "aggs": { > "winners": { > "terms": { > "field": "tit", > "size": 1 > }, > "aggs": { > "articles_over_time": { > "date_histogram": { > "field": "datetime", > "interval": "1d" > } > } > } > } > } > }' > result: > > { > > "took" : 17, > > "timed_out" : false, > > "_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 89419, > > "max_score" : 0.0, > > "hits" : [ ] > > }, > > "aggregations" : { > > "winners" : { > > "buckets" : [ { > > "key" : "videotitle", > > "doc_count" : 2820, > > "articles_over_time" : { > > "buckets" : [ { > > "key_as_string" : "2014-07-26T00:00:00.000Z", > > "key" : 1406332800000, > > "doc_count" : 2820 > > } ] > > } > > } ] > > } > > } > > } > > Which seems to be a related problem to my original query, because it > always seem to query one index but not the other > > my original query I used in elasticsearch-head: > /aggtest/video/ > > { > "size": 0, > "aggs": { > "winners": { > "terms": { > "field": "tit", > "order": { > "diff": "desc" > } > }, > "aggs": { > "articles_over_time": { > "date_histogram": { > "field": "datetime", > "interval": "1d" > } > }, > "diff": { > "sum": { > "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1", > "lang": "groovy" > } > } > } > } > } > } > and the result: > > - { > - key: videotitle > - doc_count: 719 > - articles_over_time: { > - buckets: [ > - { > - key_as_string: 2014-07-27T00:00:00.000Z > - key: 1406419200000 > - doc_count: 719 > } > ] > } > - diff: { > - value: 719 > } > } > > > > Thanks, > Valentin > > > > > On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe wrote: >> >> How are you searching over the multiple indexes? are you using aliases? >> It would be helpful if you could post your alias configuration (see [1]) >> and an cURL example of a search request that fails >> >> [1] >> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving >> >> Thanks >> >> Colin >> >> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote: >>> >>> Hi Colin, >>> >>> thanks for checking. I could successfully reproduce your example and I >>> even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0). But as >>> soon as I try it with my data it doesnt work. I ran some additional tests >>> and it works if I only use the current index (day) and split it in half. >>> But as soon as I try to compare yesterday and the day before it only seems >>> to get the data from one day but not the other. >>> >>> Cheers, >>> Valentin >>> >>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe wrote: >>>> >>>> Hi, >>>> >>>> I ran the commands in the following gist, on master, without error. >>>> Would you be able to post the error you get and a similar reproducible >>>> example to help diagnose the issue you are running into? Also, which >>>> version of Elasticsearch are you running? >>>> >>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae >>>> >>>> Thanks >>>> >>>> Colin >>>> >>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am trying to use this aggregation which does not work: >>>>> "aggs": { >>>>> "winners": { >>>>> "terms": { >>>>> "field": "urls", >>>>> "order": { >>>>> "diff": "desc" >>>>> } >>>>> }, >>>>> "aggs": { >>>>> "diff": { >>>>> "sum": { >>>>> "script": "(doc['datetime'].date.getMillis() < >>>>> 1406332800000) ? -1 : 1", >>>>> "lang": "groovy" >>>>> } >>>>> } >>>>> } >>>>> } >>>>> } >>>>> >>>>> Can anyone help? >>>>> >>>>> Cheers, >>>>> Valentin >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/50971f05-ed13-48f1-b228-3cbc54997048%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
