Also, your shard_size parameter should always be greater than the size
parameter. So if you are asking for size of 10 then I would try setting
shard_size to 20 or 30.
On Wednesday, 30 July 2014 09:22:16 UTC+1, Colin Goodheart-Smithe wrote:
>
> Would you be able to re-run your query and post the stack trace from the
> Elasticsearch server logs. This might help to work out whats going on.
>
> Thanks
>
> Colin
>
> On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>>
>> Ok. I think I found the problem. As soon as I try to sort on the script
>> value it ceases to work
>>
>> works, but unsorted
>> {
>> "size": 0,
>> "aggs": {
>> "winners": {
>> "terms": {
>> "field": "tit",
>> "size": 10,
>> "shard_size": 4
>> },
>> "aggs": {
>> "articles_over_time": {
>> "date_histogram": {
>> "field": "datetime",
>> "interval": "1d"
>> }
>> },
>> "diff": {
>> "sum": {
>> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
>> "lang": "groovy"
>> }
>> }
>> }
>> }
>> }
>> }
>>
>> does not work:
>> {
>> "size": 0,
>> "aggs": {
>> "winners": {
>> "terms": {
>> "field": "tit",
>> "size": 10,
>> "order": {
>> "diff": "desc"
>> },
>> "shard_size": 4
>> },
>> "aggs": {
>> "articles_over_time": {
>> "date_histogram": {
>> "field": "datetime",
>> "interval": "1d"
>> }
>> },
>> "diff": {
>> "sum": {
>> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
>> "lang": "groovy"
>> }
>> }
>> }
>> }
>> }
>> }
>>
>>
>>
>>
>> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>>>
>>> Hi Colin,
>>>
>>> I could figure out the shard_size problem thanks to your help.
>>>
>>> For the 'datetime' error: I checked and it exists in all the indices. It
>>> has the correct mappings and the therefor probably could not have wrong
>>> values I guess. And using the elasticsearch-head plugin I dont get the
>>> error but a wrong result which really seems strange.
>>>
>>> Thanks
>>> Valentin
>>>
>>> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe
>>> wrote:
>>>>
>>>> Firstly, I think the reason you are only getting results from one index
>>>> when you are asking for a size of 1 in your terms aggregation is because
>>>> you are asking for the top 1 bucket from each shard on each index. This
>>>> will then be merged together and only the top bucket will be kept. If the
>>>> top bucket is not the same on all indexes then you will not get results
>>>> from all indices. Setting the shard_size parameter to something like 10
>>>> can help with this (see
>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>>>>
>>>> for more information on this)
>>>>
>>>> Second, I wonder if the reason you are getting the error from your
>>>> script is that you don't have a 'datetime' value for all of your documents
>>>> in some of your indices?
>>>>
>>>> Regards,
>>>>
>>>> Colin
>>>>
>>>> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>>>>>
>>>>> Hi Colin,
>>>>>
>>>>> now it gets really strange. First my alias
>>>>> curl 'http://localhost:9200/_alias?pretty'
>>>>> {
>>>>> "live-2014-07-27" : {
>>>>>
>>>>> "aliases" : {
>>>>>
>>>>> "aggtest" : { }
>>>>>
>>>>> }
>>>>>
>>>>> },
>>>>>
>>>>> "live-2014-07-26" : {
>>>>>
>>>>> "aliases" : {
>>>>>
>>>>> "aggtest" : { }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> I tried two different queries:
>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>> -d '{
>>>>> "size": 0,
>>>>> "aggs": {
>>>>> "winners": {
>>>>> "terms": {
>>>>> "field": "tit",
>>>>> "order": {
>>>>> "diff": "desc"
>>>>> },
>>>>> "size": 1
>>>>> },
>>>>> "aggs": {
>>>>> "articles_over_time": {
>>>>> "date_histogram": {
>>>>> "field": "datetime",
>>>>> "interval": "1d"
>>>>> }
>>>>> },
>>>>> "diff": {
>>>>> "sum": {
>>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>>> 1",
>>>>> "lang": "groovy"
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }'
>>>>>
>>>>> and
>>>>>
>>>>> curl -XPOST '
>>>>> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
>>>>>
>>>>> .....
>>>>>
>>>>> both do give me a result (but a wrong one) when I do query using
>>>>> elasticsearch-head but result in an error if I use the commandline
>>>>>
>>>>> {
>>>>>
>>>>> "error" : "SearchPhaseExecutionException[Failed to execute phase
>>>>> [query], all shards failed; shardFailures
>>>>> {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]:
>>>>> QueryPhaseExecutionException[[live-2014-07-26][1]:
>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>> [Failed to execute main query]]; nested:
>>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>>> datetime for class: Script126]];
>>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]:
>>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>>
>>>>> nested: QueryPhaseExecutionException[[live-2014-07-27][0]:
>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>> [Failed to execute main query]]; nested:
>>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>>> datetime for class: Script119]];
>>>>> }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]:
>>>>> QueryPhaseExecutionException[[live-2014-07-27][1]:
>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>> [Failed to execute main query]]; nested:
>>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>>> datetime for class: Script126]];
>>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]:
>>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>>
>>>>> nested: QueryPhaseExecutionException[[live-2014-07-26][0]:
>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>> [Failed to execute main query]]; nested:
>>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>>> datetime for class: Script119]]; }]",
>>>>>
>>>>> "status" : 500
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> But I noticed something strange. This works:
>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>> -d '{
>>>>> "size": 0,
>>>>> "aggs": {
>>>>> "winners": {
>>>>> "terms": {
>>>>> "field": "tit"
>>>>> },
>>>>> "aggs": {
>>>>> "articles_over_time": {
>>>>> "date_histogram": {
>>>>> "field": "datetime",
>>>>> "interval": "1d"
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }'
>>>>> result:
>>>>>
>>>>> {
>>>>>
>>>>> "took" : 26,
>>>>>
>>>>> "timed_out" : false,
>>>>>
>>>>> "_shards" : {
>>>>>
>>>>> "total" : 4,
>>>>>
>>>>> "successful" : 4,
>>>>>
>>>>> "failed" : 0
>>>>>
>>>>> },
>>>>>
>>>>> "hits" : {
>>>>>
>>>>> "total" : 89419,
>>>>>
>>>>> "max_score" : 0.0,
>>>>>
>>>>> "hits" : [ ]
>>>>>
>>>>> },
>>>>>
>>>>> "aggregations" : {
>>>>>
>>>>> "winners" : {
>>>>>
>>>>> "buckets" : [ {
>>>>>
>>>>> "key" : "videotitle",
>>>>>
>>>>> "doc_count" : 3539,
>>>>>
>>>>> "articles_over_time" : {
>>>>>
>>>>> "buckets" : [ {
>>>>>
>>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>>
>>>>> "key" : 1406332800000,
>>>>>
>>>>> "doc_count" : 2820
>>>>>
>>>>> }, {
>>>>>
>>>>> "key_as_string" : "2014-07-27T00:00:00.000Z",
>>>>>
>>>>> "key" : 1406419200000,
>>>>>
>>>>> "doc_count" : 719
>>>>>
>>>>> } ]
>>>>>
>>>>> }
>>>>>
>>>>> }, {
>>>>>
>>>>> But this does not: (notice the size-limit to 1)
>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>> -d '{
>>>>> "size": 0,
>>>>> "aggs": {
>>>>> "winners": {
>>>>> "terms": {
>>>>> "field": "tit",
>>>>> "size": 1
>>>>> },
>>>>> "aggs": {
>>>>> "articles_over_time": {
>>>>> "date_histogram": {
>>>>> "field": "datetime",
>>>>> "interval": "1d"
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }'
>>>>> result:
>>>>>
>>>>> {
>>>>>
>>>>> "took" : 17,
>>>>>
>>>>> "timed_out" : false,
>>>>>
>>>>> "_shards" : {
>>>>>
>>>>> "total" : 4,
>>>>>
>>>>> "successful" : 4,
>>>>>
>>>>> "failed" : 0
>>>>>
>>>>> },
>>>>>
>>>>> "hits" : {
>>>>>
>>>>> "total" : 89419,
>>>>>
>>>>> "max_score" : 0.0,
>>>>>
>>>>> "hits" : [ ]
>>>>>
>>>>> },
>>>>>
>>>>> "aggregations" : {
>>>>>
>>>>> "winners" : {
>>>>>
>>>>> "buckets" : [ {
>>>>>
>>>>> "key" : "videotitle",
>>>>>
>>>>> "doc_count" : 2820,
>>>>>
>>>>> "articles_over_time" : {
>>>>>
>>>>> "buckets" : [ {
>>>>>
>>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>>
>>>>> "key" : 1406332800000,
>>>>>
>>>>> "doc_count" : 2820
>>>>>
>>>>> } ]
>>>>>
>>>>> }
>>>>>
>>>>> } ]
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> Which seems to be a related problem to my original query, because it
>>>>> always seem to query one index but not the other
>>>>>
>>>>> my original query I used in elasticsearch-head:
>>>>> /aggtest/video/
>>>>>
>>>>> {
>>>>> "size": 0,
>>>>> "aggs": {
>>>>> "winners": {
>>>>> "terms": {
>>>>> "field": "tit",
>>>>> "order": {
>>>>> "diff": "desc"
>>>>> }
>>>>> },
>>>>> "aggs": {
>>>>> "articles_over_time": {
>>>>> "date_histogram": {
>>>>> "field": "datetime",
>>>>> "interval": "1d"
>>>>> }
>>>>> },
>>>>> "diff": {
>>>>> "sum": {
>>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>>> 1",
>>>>> "lang": "groovy"
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> and the result:
>>>>>
>>>>> - {
>>>>> - key: videotitle
>>>>> - doc_count: 719
>>>>> - articles_over_time: {
>>>>> - buckets: [
>>>>> - {
>>>>> - key_as_string: 2014-07-27T00:00:00.000Z
>>>>> - key: 1406419200000
>>>>> - doc_count: 719
>>>>> }
>>>>> ]
>>>>> }
>>>>> - diff: {
>>>>> - value: 719
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Valentin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe
>>>>> wrote:
>>>>>>
>>>>>> How are you searching over the multiple indexes? are you using
>>>>>> aliases? It would be helpful if you could post your alias configuration
>>>>>> (see [1]) and an cURL example of a search request that fails
>>>>>>
>>>>>> [1]
>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote:
>>>>>>>
>>>>>>> Hi Colin,
>>>>>>>
>>>>>>> thanks for checking. I could successfully reproduce your example and
>>>>>>> I even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0).
>>>>>>> But
>>>>>>> as soon as I try it with my data it doesnt work. I ran some additional
>>>>>>> tests and it works if I only use the current index (day) and split it
>>>>>>> in
>>>>>>> half. But as soon as I try to compare yesterday and the day before it
>>>>>>> only
>>>>>>> seems to get the data from one day but not the other.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Valentin
>>>>>>>
>>>>>>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I ran the commands in the following gist, on master, without error.
>>>>>>>> Would you be able to post the error you get and a similar
>>>>>>>> reproducible
>>>>>>>> example to help diagnose the issue you are running into? Also, which
>>>>>>>> version of Elasticsearch are you running?
>>>>>>>>
>>>>>>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Colin
>>>>>>>>
>>>>>>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to use this aggregation which does not work:
>>>>>>>>> "aggs": {
>>>>>>>>> "winners": {
>>>>>>>>> "terms": {
>>>>>>>>> "field": "urls",
>>>>>>>>> "order": {
>>>>>>>>> "diff": "desc"
>>>>>>>>> }
>>>>>>>>> },
>>>>>>>>> "aggs": {
>>>>>>>>> "diff": {
>>>>>>>>> "sum": {
>>>>>>>>> "script": "(doc['datetime'].date.getMillis() <
>>>>>>>>> 1406332800000) ? -1 : 1",
>>>>>>>>> "lang": "groovy"
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Can anyone help?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Valentin
>>>>>>>>>
>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/99841285-725a-41d0-b337-65a2eca0b4af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
