Hi Colin, I could figure out the shard_size problem thanks to your help.
For the 'datetime' error: I checked and it exists in all the indices. It has the correct mappings and the therefor probably could not have wrong values I guess. And using the elasticsearch-head plugin I dont get the error but a wrong result which really seems strange. Thanks Valentin On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe wrote: > > Firstly, I think the reason you are only getting results from one index > when you are asking for a size of 1 in your terms aggregation is because > you are asking for the top 1 bucket from each shard on each index. This > will then be merged together and only the top bucket will be kept. If the > top bucket is not the same on all indexes then you will not get results > from all indices. Setting the shard_size parameter to something like 10 > can help with this (see > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate > > for more information on this) > > Second, I wonder if the reason you are getting the error from your script > is that you don't have a 'datetime' value for all of your documents in some > of your indices? > > Regards, > > Colin > > On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote: >> >> Hi Colin, >> >> now it gets really strange. First my alias >> curl 'http://localhost:9200/_alias?pretty' >> { >> "live-2014-07-27" : { >> >> "aliases" : { >> >> "aggtest" : { } >> >> } >> >> }, >> >> "live-2014-07-26" : { >> >> "aliases" : { >> >> "aggtest" : { } >> >> } >> >> } >> >> } >> >> >> I tried two different queries: >> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d >> '{ >> "size": 0, >> "aggs": { >> "winners": { >> "terms": { >> "field": "tit", >> "order": { >> "diff": "desc" >> }, >> "size": 1 >> }, >> "aggs": { >> "articles_over_time": { >> "date_histogram": { >> "field": "datetime", >> "interval": "1d" >> } >> }, >> "diff": { >> "sum": { >> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1", >> "lang": "groovy" >> } >> } >> } >> } >> } >> }' >> >> and >> >> curl -XPOST ' >> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true' >> >> ..... >> >> both do give me a result (but a wrong one) when I do query using >> elasticsearch-head but result in an error if I use the commandline >> >> { >> >> "error" : "SearchPhaseExecutionException[Failed to execute phase >> [query], all shards failed; shardFailures >> {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]: >> QueryPhaseExecutionException[[live-2014-07-26][1]: >> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed >> [Failed to execute main query]]; nested: >> GroovyScriptExecutionException[MissingPropertyException[No such property: >> datetime for class: Script126]]; >> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]: >> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]]; >> >> nested: QueryPhaseExecutionException[[live-2014-07-27][0]: >> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed >> [Failed to execute main query]]; nested: >> GroovyScriptExecutionException[MissingPropertyException[No such property: >> datetime for class: Script119]]; >> }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]: >> QueryPhaseExecutionException[[live-2014-07-27][1]: >> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed >> [Failed to execute main query]]; nested: >> GroovyScriptExecutionException[MissingPropertyException[No such property: >> datetime for class: Script126]]; >> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]: >> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]]; >> >> nested: QueryPhaseExecutionException[[live-2014-07-26][0]: >> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed >> [Failed to execute main query]]; nested: >> GroovyScriptExecutionException[MissingPropertyException[No such property: >> datetime for class: Script119]]; }]", >> >> "status" : 500 >> >> } >> >> >> But I noticed something strange. This works: >> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d >> '{ >> "size": 0, >> "aggs": { >> "winners": { >> "terms": { >> "field": "tit" >> }, >> "aggs": { >> "articles_over_time": { >> "date_histogram": { >> "field": "datetime", >> "interval": "1d" >> } >> } >> } >> } >> } >> }' >> result: >> >> { >> >> "took" : 26, >> >> "timed_out" : false, >> >> "_shards" : { >> >> "total" : 4, >> >> "successful" : 4, >> >> "failed" : 0 >> >> }, >> >> "hits" : { >> >> "total" : 89419, >> >> "max_score" : 0.0, >> >> "hits" : [ ] >> >> }, >> >> "aggregations" : { >> >> "winners" : { >> >> "buckets" : [ { >> >> "key" : "videotitle", >> >> "doc_count" : 3539, >> >> "articles_over_time" : { >> >> "buckets" : [ { >> >> "key_as_string" : "2014-07-26T00:00:00.000Z", >> >> "key" : 1406332800000, >> >> "doc_count" : 2820 >> >> }, { >> >> "key_as_string" : "2014-07-27T00:00:00.000Z", >> >> "key" : 1406419200000, >> >> "doc_count" : 719 >> >> } ] >> >> } >> >> }, { >> >> But this does not: (notice the size-limit to 1) >> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' -d >> '{ >> "size": 0, >> "aggs": { >> "winners": { >> "terms": { >> "field": "tit", >> "size": 1 >> }, >> "aggs": { >> "articles_over_time": { >> "date_histogram": { >> "field": "datetime", >> "interval": "1d" >> } >> } >> } >> } >> } >> }' >> result: >> >> { >> >> "took" : 17, >> >> "timed_out" : false, >> >> "_shards" : { >> >> "total" : 4, >> >> "successful" : 4, >> >> "failed" : 0 >> >> }, >> >> "hits" : { >> >> "total" : 89419, >> >> "max_score" : 0.0, >> >> "hits" : [ ] >> >> }, >> >> "aggregations" : { >> >> "winners" : { >> >> "buckets" : [ { >> >> "key" : "videotitle", >> >> "doc_count" : 2820, >> >> "articles_over_time" : { >> >> "buckets" : [ { >> >> "key_as_string" : "2014-07-26T00:00:00.000Z", >> >> "key" : 1406332800000, >> >> "doc_count" : 2820 >> >> } ] >> >> } >> >> } ] >> >> } >> >> } >> >> } >> >> Which seems to be a related problem to my original query, because it >> always seem to query one index but not the other >> >> my original query I used in elasticsearch-head: >> /aggtest/video/ >> >> { >> "size": 0, >> "aggs": { >> "winners": { >> "terms": { >> "field": "tit", >> "order": { >> "diff": "desc" >> } >> }, >> "aggs": { >> "articles_over_time": { >> "date_histogram": { >> "field": "datetime", >> "interval": "1d" >> } >> }, >> "diff": { >> "sum": { >> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1", >> "lang": "groovy" >> } >> } >> } >> } >> } >> } >> and the result: >> >> - { >> - key: videotitle >> - doc_count: 719 >> - articles_over_time: { >> - buckets: [ >> - { >> - key_as_string: 2014-07-27T00:00:00.000Z >> - key: 1406419200000 >> - doc_count: 719 >> } >> ] >> } >> - diff: { >> - value: 719 >> } >> } >> >> >> >> Thanks, >> Valentin >> >> >> >> >> On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe wrote: >>> >>> How are you searching over the multiple indexes? are you using aliases? >>> It would be helpful if you could post your alias configuration (see [1]) >>> and an cURL example of a search request that fails >>> >>> [1] >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving >>> >>> Thanks >>> >>> Colin >>> >>> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote: >>>> >>>> Hi Colin, >>>> >>>> thanks for checking. I could successfully reproduce your example and I >>>> even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0). But >>>> as >>>> soon as I try it with my data it doesnt work. I ran some additional tests >>>> and it works if I only use the current index (day) and split it in half. >>>> But as soon as I try to compare yesterday and the day before it only seems >>>> to get the data from one day but not the other. >>>> >>>> Cheers, >>>> Valentin >>>> >>>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I ran the commands in the following gist, on master, without error. >>>>> Would you be able to post the error you get and a similar reproducible >>>>> example to help diagnose the issue you are running into? Also, which >>>>> version of Elasticsearch are you running? >>>>> >>>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae >>>>> >>>>> Thanks >>>>> >>>>> Colin >>>>> >>>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to use this aggregation which does not work: >>>>>> "aggs": { >>>>>> "winners": { >>>>>> "terms": { >>>>>> "field": "urls", >>>>>> "order": { >>>>>> "diff": "desc" >>>>>> } >>>>>> }, >>>>>> "aggs": { >>>>>> "diff": { >>>>>> "sum": { >>>>>> "script": "(doc['datetime'].date.getMillis() < >>>>>> 1406332800000) ? -1 : 1", >>>>>> "lang": "groovy" >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>>> Can anyone help? >>>>>> >>>>>> Cheers, >>>>>> Valentin >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6d1dab85-c04d-4896-a792-cdec654d62b4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
