Rename aggregation metric in Kibana 4.1

[Kevin S]

In my visualization I have several sum aggregations.  The title for the 
metric is always "Sum of ..."  for example "Sum of profit".  Is it possible 
to rename the aggregation metric to just "Profit"?  There are several 
columns and the "Sum of" clutters the UI, prevents us from fitting more 
columns, and is really not useful in any way.

e.g.

"Sum of profit" -> "Profit"

"Count of event" -> "Events"

I see a JSON input but I can not find any references to renaming the metric.

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6b5b2dce-e915-443e-a61c-75185545459e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sum aggregation with results from other aggregations?

[Josh Harrison]

Is it possible to create an aggregation where I can do a sum on the results 
of a sub bucket?

I'm working on twitter data. In this data I have a bunch of retweets of 
different users.
Say that user A has 10 tweets that are retweeted a hundred times in my 
dataset. I want to find the maximum retweet_count for each individual 
tweet, and then I want to find the sum of all of those maximums from an 
individual user.
This is the base query structure I'm working with: 

{
  "aggs": {
"user_id": {
  "terms": {
"field": "retweet_user_id"
  },
  "aggs": {
"tweet_ids": {
  "terms": {
"field": "retweet_id",
"order": "max_tweet.value"
  },
  "aggs": {
"max_tweet": {
  "max": {
"field": "retweet_count"
  }
}
  }
}
  }
}
  }
}



Importantly here, I don't want to just take a sum of "retweet_count" for a 
given retweet_user_id - this doesn't give the max value per tweet.


Essentially, is it possible for me to take a sum of the agg results at 
user_id.tweet_ids.max_tweet.value, and use that as an "order" term in the 
user_id terms agg?



-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/100b463d-0f95-4801-aec7-e32544624518%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation profiling?

[James Macdonald]

I don't have an answer, but I really like this question. I too would love
to see more query and aggregation profiling tools for performance
optimization purposes.

Also, I assume you have already looked at this, but have you made sure you
are not evicting anything from your in memory field data?

James

On Mon, May 25, 2015 at 4:08 PM, Mike Sukmanowsky <
mike.sukmanow...@gmail.com> wrote:

> I don't believe there are any current endpoints in the API that support
> this, but are there plans to add better profiling information to ES
> aggregation queries? We'll see some agg queries return in 11s, then <5s
> then >11s again. Sometimes we can see associated filter cache expirations,
> but it's really hard to line these up to one specific query in our
> production environment since multiple users are executing queries
> simultaneously.
>
> It'd be really helpful to optionally see where aggregation queries are
> spending the bulk of their time to help us understand what to improve in
> the future.
>
> Anything we can do here right now?
>
> --
> Mike Sukmanowsky
> Aspiring Digital Carpenter
>
> *e*: mike.sukmanow...@gmail.com
>
> facebook <http://facebook.com/mike.sukmanowsky> | twitter
> <http://twitter.com/msukmanowsky> | LinkedIn
> <http://www.linkedin.com/profile/view?id=10897143> | github
> <https://github.com/msukmanowsky>
>
>   --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOH6cu5WSGqQ%2BZ0_qrofXEvwo8JuSH9xoSbZgSwiT90MJ_wxdA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAOH6cu5WSGqQ%2BZ0_qrofXEvwo8JuSH9xoSbZgSwiT90MJ_wxdA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTZOmx-fk%2BG9dR6-XYB_1j7mGRNRwTqvQRwKx0YAcopFWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How can distinct document result without using Aggregation

[ali balci]

I explain in simple way : Select distinct(*) from Product ı try to do in 
elasticsearch but ı find aggregation cardinality but this can work for me ı 
need distinct all document , how can ı do that , 
I think ı can do with script but find any clue. Is there any solution ??

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1b839f26-abfd-48a1-9d52-23e4db9d48db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation profiling?

[Mike Sukmanowsky]

I don't believe there are any current endpoints in the API that support
this, but are there plans to add better profiling information to ES
aggregation queries? We'll see some agg queries return in 11s, then <5s
then >11s again. Sometimes we can see associated filter cache expirations,
but it's really hard to line these up to one specific query in our
production environment since multiple users are executing queries
simultaneously.

It'd be really helpful to optionally see where aggregation queries are
spending the bulk of their time to help us understand what to improve in
the future.

Anything we can do here right now?

-- 
Mike Sukmanowsky
Aspiring Digital Carpenter

*e*: mike.sukmanow...@gmail.com

facebook <http://facebook.com/mike.sukmanowsky> | twitter
<http://twitter.com/msukmanowsky> | LinkedIn
<http://www.linkedin.com/profile/view?id=10897143> | github
<https://github.com/msukmanowsky>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOH6cu5WSGqQ%2BZ0_qrofXEvwo8JuSH9xoSbZgSwiT90MJ_wxdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Cardinality Aggregation - Different Unique Counts!

[Herick Paiva]

Hey guys, 


I am still trying to figure this out, 2 ElasticSearch queries, unique count 
on one is 


112019


And the other :


164322 


Here are the Requests and responses http://pastebin.com/4kHJiL9t and 
http://pastebin.com/KL9XyQ6w


The actual count match for both but not the unique.


FYI, the ip_address field type is string, Im storing IPs like I get them 
from client (xxx.xxx.xxx.xxx)


I just can't understand why they are coming up different, and honestly 
would like to know which is probably right.


Can anyone help?



-- 
Please update your bookmarks! We moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5cfcb07d-6ed8-45da-8637-49fac231edaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

aggregation for similar strings

[Lior Goldemberg]

hi guys,

i have similar string that i want to aggregate,
but they seem to be different within few characters at the end.

for example:
doc1: {"message":"hello world and good morning1 300"}
doc2: {"message":"hello world and good morning1 200"}
doc3: {"message":"hello world and good morning1 100"}

i would like to have this result in the aggregation:

"hello world and good morning1" - count: 3

the field currently defined with default analyzer

btw, is it possible to identify also more complected string such as:
doc3: {"message":"500 hello world and good morning1 100"}

thanks a lot for advanced!

Lior

-- 
Please update your bookmarks! We moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7c402360-6940-400e-8a2f-933107f410a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search Term Aggregation on numeric / long field

[Deepak Chauhan]

Okk ,Congratulation buddy

On Mon, Apr 27, 2015 at 6:15 PM,  wrote:

> problem solved, when switching to 1.5.0+
>
>
>
> On Monday, April 27, 2015 at 12:48:19 PM UTC+2, r...@hubrick.com wrote:
>>
>> Hey,
>>
>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>
>> My schema looks like that:
>>
>> { ...
>>starRating:
>>{
>>
>>-   type: "long"
>>
>>}
>> ...
>> }
>>
>> I pass documents there, works fine.
>> I query them, using a Range Query like, works also fine:
>>
>> {
>>   "range": {
>> "starRating": {
>>   "from": 0,
>>   "to": 5,
>> }
>>   }
>> }
>>
>> Whenever I ask for Term Aggregations like:
>>
>> {
>>   "aggs": {
>> stars" : {
>>"terms" : { "field" : "starRating" }
>> }
>> }
>>
>> OR Facets with
>>
>>   "facets" : {
>> "tag" : {
>>   "terms" : {
>> "field" : "starRating"
>>   }
>> }
>>   }
>>
>> Strange values are returned:
>>
>> "facets": {
>> "tag": {
>>   "_type": "terms",
>>   "missing": 0,
>>   "total": 3814,
>>   "other": 0,
>>   "terms": [
>> {
>>   "term": 0.0,
>>   "count": 2397
>> },
>> {
>>   "term": 1.5E-323,
>>   "count": 610
>> },
>> {
>>   "term": 2.0E-323,
>>   "count": 378
>> },
>> {
>>   "term": 1.0E-323,
>>   "count": 269
>> },
>> {
>>   "term": 4.9E-324,
>>   "count": 93
>> },
>> {
>>   "term": 2.5E-323,
>>   "count": 67
>> }
>>   ]
>> }
>>   },
>>   "aggregations": {
>> "stars": {
>>   "doc_count_error_upper_bound": 0,
>>   "sum_other_doc_count": 0,
>>   "buckets": [
>> {
>>   "key": 0.0,
>>   "doc_count": 2397
>> },
>> {
>>   "key": 1.5E-323,
>>   "doc_count": 610
>> },
>> {
>>   "key": 2.0E-323,
>>   "doc_count": 378
>> },
>> {
>>   "key": 1.0E-323,
>>   "doc_count": 269
>> },
>> {
>>   "key": 4.9E-324,
>>   "doc_count": 93
>> },
>> {
>>   "key": 2.5E-323,
>>   "doc_count": 67
>> }
>>   ]
>> }
>>   }
>>
>>
>> Even in the Java Client, I get DoubleTerms instead of LongTerms.
>>
>> Does anybody have an Idea, what wents wrong there. I would expect
>> Long-Keys and not Double keys (which are basically close to 0)
>> The Histogram (with interval of 1) also does not work, everything is
>> close to the zero valued key.
>>
>>
>> Cheers, Ralf
>>
>>
>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9fb08d23-49ee-4baa-8df8-247797e0dee9%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHzCMpoqkTnDab513ZPY-B7VqkHjqVsQC0ACrE%3DOEe6rD9Rtaw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search Term Aggregation on numeric / long field

[rh]

problem solved, when switching to 1.5.0+



On Monday, April 27, 2015 at 12:48:19 PM UTC+2, r...@hubrick.com wrote:
>
> Hey, 
>
> I'm a little bit stucked using Terms aggregations on a numeric field.
>
> My schema looks like that:
>
> { ...
>starRating: 
>{
>
>-   type: "long"
>
>}
> ...
> }
>
> I pass documents there, works fine.
> I query them, using a Range Query like, works also fine: 
>
> {
>   "range": {
> "starRating": {
>   "from": 0,
>   "to": 5,
> }
>   }
> }
>
> Whenever I ask for Term Aggregations like:
>
> {
>   "aggs": {
> stars" : {
>"terms" : { "field" : "starRating" }
> }
> }
>
> OR Facets with
>
>   "facets" : {
> "tag" : {
>   "terms" : {
> "field" : "starRating"
>   }
> }
>   }
>
> Strange values are returned:
>
> "facets": {
> "tag": {
>   "_type": "terms",
>   "missing": 0,
>   "total": 3814,
>   "other": 0,
>   "terms": [
> {
>   "term": 0.0,
>   "count": 2397
> },
> {
>   "term": 1.5E-323,
>   "count": 610
> },
> {
>   "term": 2.0E-323,
>   "count": 378
> },
> {
>   "term": 1.0E-323,
>   "count": 269
> },
> {
>   "term": 4.9E-324,
>   "count": 93
> },
> {
>   "term": 2.5E-323,
>   "count": 67
> }
>   ]
> }
>   },
>   "aggregations": {
> "stars": {
>   "doc_count_error_upper_bound": 0,
>   "sum_other_doc_count": 0,
>   "buckets": [
> {
>   "key": 0.0,
>   "doc_count": 2397
> },
> {
>   "key": 1.5E-323,
>   "doc_count": 610
> },
> {
>   "key": 2.0E-323,
>   "doc_count": 378
> },
> {
>   "key": 1.0E-323,
>   "doc_count": 269
> },
> {
>   "key": 4.9E-324,
>   "doc_count": 93
> },
> {
>   "key": 2.5E-323,
>   "doc_count": 67
> }
>   ]
> }
>   }
>
>
> Even in the Java Client, I get DoubleTerms instead of LongTerms.
>
> Does anybody have an Idea, what wents wrong there. I would expect 
> Long-Keys and not Double keys (which are basically close to 0)
> The Histogram (with interval of 1) also does not work, everything is close 
> to the zero valued key.
>
>
> Cheers, Ralf
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9fb08d23-49ee-4baa-8df8-247797e0dee9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search Term Aggregation on numeric / long field

[Deepak Chauhan]

i have also tried on 1.5.1 , it is woring properly


On Mon, Apr 27, 2015 at 5:59 PM, Deepak Chauhan <
deepak.chau...@daffodilsw.com> wrote:

> yeah may be due to that
> ..
> i am using 1.4.2
>
> On Mon, Apr 27, 2015 at 5:47 PM,  wrote:
>
>> which version do you use - i'm using 1.4.4. ?
>>
>> On Monday, April 27, 2015 at 1:27:12 PM UTC+2, deepak.chauhan wrote:
>>>
>>> may be .. i have tried this again in my index ..
>>> it is working properly
>>> Please check your mapping
>>>  or try on another field which is long
>>>
>>> On Mon, Apr 27, 2015 at 4:31 PM,  wrote:
>>>
>>>> unfortunately the same issue, ...
>>>> Whats strange, filtering / querying these values works as expected. I
>>>> guess, that there is an internal mapping working unexpected.
>>>>
>>>> "starRating": {
>>>>   "buckets": [
>>>> {
>>>>   "key": "0.0-1.0",
>>>>   "from": 0.0,
>>>>   "from_as_string": "0.0",
>>>>   "to": 1.0,
>>>>   "to_as_string": "1.0",
>>>>   "doc_count": 5119
>>>> },
>>>> {
>>>>   "key": "1.0-2.0",
>>>>   "from": 1.0,
>>>>   "from_as_string": "1.0",
>>>>   "to": 2.0,
>>>>   "to_as_string": "2.0",
>>>>   "doc_count": 0
>>>> },
>>>> {
>>>>   "key": "2.0-3.0",
>>>>   "from": 2.0,
>>>>   "from_as_string": "2.0",
>>>>   "to": 3.0,
>>>>   "to_as_string": "3.0",
>>>>   "doc_count": 0
>>>> },
>>>> {
>>>>   "key": "3.0-4.0",
>>>>   "from": 3.0,
>>>>   "from_as_string": "3.0",
>>>>   "to": 4.0,
>>>>   "to_as_string": "4.0",
>>>>   "doc_count": 0
>>>> },
>>>> {
>>>>   "key": "4.0-5.0",
>>>>   "from": 4.0,
>>>>   "from_as_string": "4.0",
>>>>   "to": 5.0,
>>>>   "to_as_string": "5.0",
>>>>   "doc_count": 0
>>>> },
>>>> {
>>>>   "key": "5.0-6.0",
>>>>   "from": 5.0,
>>>>   "from_as_string": "5.0",
>>>>   "to": 6.0,
>>>>   "to_as_string": "6.0",
>>>>   "doc_count": 0
>>>> }
>>>>   ]
>>>> }
>>>>
>>>>
>>>> On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>>>>>
>>>>> Besides the Terms filter in aggregation , you should use range
>>>>> aggregation .
>>>>> It would work like..
>>>>>
>>>>> {
>>>>>   "aggs": {
>>>>> stars" : {
>>>>>"range" : {
>>>>>   "field" : "starRating" ,
>>>>>  "ranges": [{
>>>>> "to": 5
>>>>> }, {
>>>>> "from": 5,
>>>>> "to": 10
>>>>> }, {
>>>>> "from": 10,
>>>>> "to": 15
>>>>> }, {
>>>>> "from": 15
>>>>> }]
>>>>>
>>>>>}
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>&

Re: Elastic Search Term Aggregation on numeric / long field

[Deepak Chauhan]

yeah may be due to that
..
i am using 1.4.2

On Mon, Apr 27, 2015 at 5:47 PM,  wrote:

> which version do you use - i'm using 1.4.4. ?
>
> On Monday, April 27, 2015 at 1:27:12 PM UTC+2, deepak.chauhan wrote:
>>
>> may be .. i have tried this again in my index ..
>> it is working properly
>> Please check your mapping
>>  or try on another field which is long
>>
>> On Mon, Apr 27, 2015 at 4:31 PM,  wrote:
>>
>>> unfortunately the same issue, ...
>>> Whats strange, filtering / querying these values works as expected. I
>>> guess, that there is an internal mapping working unexpected.
>>>
>>> "starRating": {
>>>   "buckets": [
>>> {
>>>   "key": "0.0-1.0",
>>>   "from": 0.0,
>>>   "from_as_string": "0.0",
>>>   "to": 1.0,
>>>   "to_as_string": "1.0",
>>>   "doc_count": 5119
>>> },
>>> {
>>>   "key": "1.0-2.0",
>>>   "from": 1.0,
>>>   "from_as_string": "1.0",
>>>   "to": 2.0,
>>>   "to_as_string": "2.0",
>>>   "doc_count": 0
>>> },
>>> {
>>>   "key": "2.0-3.0",
>>>   "from": 2.0,
>>>   "from_as_string": "2.0",
>>>   "to": 3.0,
>>>   "to_as_string": "3.0",
>>>   "doc_count": 0
>>> },
>>> {
>>>   "key": "3.0-4.0",
>>>   "from": 3.0,
>>>   "from_as_string": "3.0",
>>>   "to": 4.0,
>>>   "to_as_string": "4.0",
>>>   "doc_count": 0
>>> },
>>> {
>>>   "key": "4.0-5.0",
>>>   "from": 4.0,
>>>   "from_as_string": "4.0",
>>>   "to": 5.0,
>>>   "to_as_string": "5.0",
>>>   "doc_count": 0
>>> },
>>> {
>>>   "key": "5.0-6.0",
>>>   "from": 5.0,
>>>   "from_as_string": "5.0",
>>>   "to": 6.0,
>>>   "to_as_string": "6.0",
>>>   "doc_count": 0
>>> }
>>>   ]
>>> }
>>>
>>>
>>> On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>>>>
>>>> Besides the Terms filter in aggregation , you should use range
>>>> aggregation .
>>>> It would work like..
>>>>
>>>> {
>>>>   "aggs": {
>>>> stars" : {
>>>>"range" : {
>>>>   "field" : "starRating" ,
>>>>  "ranges": [{
>>>> "to": 5
>>>> }, {
>>>> "from": 5,
>>>> "to": 10
>>>> }, {
>>>> "from": 10,
>>>> "to": 15
>>>> }, {
>>>> "from": 15
>>>> }]
>>>>
>>>>}
>>>> }
>>>> }
>>>>
>>>>
>>>>
>>>> On Mon, Apr 27, 2015 at 4:18 PM,  wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>>>>
>>>>> My schema looks like that:
>>>>>
>>>>> { ...
>>>>>starRating:
>>>>>{
>>>>>
>>>>>-   type: "long"
>>>>>
>>>>>}
>>>>> ...

Re: Elastic Search Term Aggregation on numeric / long field

[rh]

which version do you use - i'm using 1.4.4. ?

On Monday, April 27, 2015 at 1:27:12 PM UTC+2, deepak.chauhan wrote:
>
> may be .. i have tried this again in my index ..
> it is working properly 
> Please check your mapping
>  or try on another field which is long 
>
> On Mon, Apr 27, 2015 at 4:31 PM, > wrote:
>
>> unfortunately the same issue, ... 
>> Whats strange, filtering / querying these values works as expected. I 
>> guess, that there is an internal mapping working unexpected.
>>
>> "starRating": {
>>   "buckets": [
>> {
>>   "key": "0.0-1.0",
>>   "from": 0.0,
>>   "from_as_string": "0.0",
>>   "to": 1.0,
>>   "to_as_string": "1.0",
>>   "doc_count": 5119
>> },
>> {
>>   "key": "1.0-2.0",
>>   "from": 1.0,
>>   "from_as_string": "1.0",
>>   "to": 2.0,
>>   "to_as_string": "2.0",
>>   "doc_count": 0
>> },
>> {
>>   "key": "2.0-3.0",
>>   "from": 2.0,
>>   "from_as_string": "2.0",
>>   "to": 3.0,
>>   "to_as_string": "3.0",
>>   "doc_count": 0
>> },
>> {
>>   "key": "3.0-4.0",
>>   "from": 3.0,
>>   "from_as_string": "3.0",
>>   "to": 4.0,
>>   "to_as_string": "4.0",
>>   "doc_count": 0
>> },
>> {
>>   "key": "4.0-5.0",
>>   "from": 4.0,
>>   "from_as_string": "4.0",
>>   "to": 5.0,
>>   "to_as_string": "5.0",
>>   "doc_count": 0
>> },
>> {
>>   "key": "5.0-6.0",
>>   "from": 5.0,
>>   "from_as_string": "5.0",
>>   "to": 6.0,
>>   "to_as_string": "6.0",
>>   "doc_count": 0
>> }
>>   ]
>> }
>>
>>
>> On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>>>
>>> Besides the Terms filter in aggregation , you should use range 
>>> aggregation .
>>> It would work like..
>>>
>>> {
>>>   "aggs": {
>>> stars" : {
>>>"range" : { 
>>>   "field" : "starRating" ,
>>>  "ranges": [{
>>> "to": 5
>>> }, {
>>> "from": 5,
>>> "to": 10
>>> }, {
>>> "from": 10,
>>> "to": 15
>>> }, {
>>> "from": 15
>>> }]
>>>  
>>>}
>>> }
>>> }
>>>
>>>
>>>
>>> On Mon, Apr 27, 2015 at 4:18 PM,  wrote:
>>>
>>>> Hey, 
>>>>
>>>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>>>
>>>> My schema looks like that:
>>>>
>>>> { ...
>>>>starRating: 
>>>>{
>>>>
>>>>-   type: "long"
>>>>
>>>>}
>>>> ...
>>>> }
>>>>
>>>> I pass documents there, works fine.
>>>> I query them, using a Range Query like, works also fine: 
>>>>
>>>> {
>>>>   "range": {
>>>> "starRating": {
>>>>   "from": 0,
>>>>   "to": 5,
>>>> }
>>>>   }
>>>> }
>>>>
>>>> Whenever I ask for Term Aggregations like:
>>>>
>>

Re: Elastic Search Term Aggregation on numeric / long field

[Deepak Chauhan]

may be .. i have tried this again in my index ..
it is working properly
Please check your mapping
 or try on another field which is long

On Mon, Apr 27, 2015 at 4:31 PM,  wrote:

> unfortunately the same issue, ...
> Whats strange, filtering / querying these values works as expected. I
> guess, that there is an internal mapping working unexpected.
>
> "starRating": {
>   "buckets": [
> {
>   "key": "0.0-1.0",
>   "from": 0.0,
>   "from_as_string": "0.0",
>   "to": 1.0,
>   "to_as_string": "1.0",
>   "doc_count": 5119
> },
> {
>   "key": "1.0-2.0",
>   "from": 1.0,
>   "from_as_string": "1.0",
>   "to": 2.0,
>   "to_as_string": "2.0",
>   "doc_count": 0
> },
> {
>   "key": "2.0-3.0",
>   "from": 2.0,
>   "from_as_string": "2.0",
>   "to": 3.0,
>   "to_as_string": "3.0",
>   "doc_count": 0
> },
> {
>   "key": "3.0-4.0",
>   "from": 3.0,
>   "from_as_string": "3.0",
>   "to": 4.0,
>   "to_as_string": "4.0",
>   "doc_count": 0
> },
> {
>   "key": "4.0-5.0",
>   "from": 4.0,
>   "from_as_string": "4.0",
>   "to": 5.0,
>   "to_as_string": "5.0",
>   "doc_count": 0
> },
> {
>   "key": "5.0-6.0",
>   "from": 5.0,
>   "from_as_string": "5.0",
>   "to": 6.0,
>   "to_as_string": "6.0",
>   "doc_count": 0
> }
>   ]
> }
>
>
> On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>>
>> Besides the Terms filter in aggregation , you should use range
>> aggregation .
>> It would work like..
>>
>> {
>>   "aggs": {
>> stars" : {
>>"range" : {
>>   "field" : "starRating" ,
>>  "ranges": [{
>> "to": 5
>> }, {
>> "from": 5,
>> "to": 10
>> }, {
>> "from": 10,
>> "to": 15
>> }, {
>> "from": 15
>> }]
>>
>>}
>> }
>> }
>>
>>
>>
>> On Mon, Apr 27, 2015 at 4:18 PM,  wrote:
>>
>>> Hey,
>>>
>>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>>
>>> My schema looks like that:
>>>
>>> { ...
>>>starRating:
>>>{
>>>
>>>-   type: "long"
>>>
>>>}
>>> ...
>>> }
>>>
>>> I pass documents there, works fine.
>>> I query them, using a Range Query like, works also fine:
>>>
>>> {
>>>   "range": {
>>> "starRating": {
>>>   "from": 0,
>>>   "to": 5,
>>> }
>>>   }
>>> }
>>>
>>> Whenever I ask for Term Aggregations like:
>>>
>>> {
>>>   "aggs": {
>>> stars" : {
>>>"terms" : { "field" : "starRating" }
>>> }
>>> }
>>>
>>> OR Facets with
>>>
>>>   "facets" : {
>>> "tag" : {
>>>   "terms" : {
>>> "field" : "starRating"
>>>   }
>>> }
>>>   }
>>>
>>> Strange values are returned:
>>>
>>> "facets": {
>>> "tag": {
>>>   "_type": "terms

Re: Elastic Search Term Aggregation on numeric / long field

[rh]

unfortunately the same issue, ... 
Whats strange, filtering / querying these values works as expected. I 
guess, that there is an internal mapping working unexpected.

"starRating": {
  "buckets": [
{
  "key": "0.0-1.0",
  "from": 0.0,
  "from_as_string": "0.0",
  "to": 1.0,
  "to_as_string": "1.0",
  "doc_count": 5119
},
{
  "key": "1.0-2.0",
  "from": 1.0,
  "from_as_string": "1.0",
  "to": 2.0,
  "to_as_string": "2.0",
  "doc_count": 0
},
{
  "key": "2.0-3.0",
  "from": 2.0,
  "from_as_string": "2.0",
  "to": 3.0,
  "to_as_string": "3.0",
  "doc_count": 0
},
{
  "key": "3.0-4.0",
  "from": 3.0,
  "from_as_string": "3.0",
  "to": 4.0,
  "to_as_string": "4.0",
  "doc_count": 0
},
{
  "key": "4.0-5.0",
  "from": 4.0,
  "from_as_string": "4.0",
  "to": 5.0,
  "to_as_string": "5.0",
  "doc_count": 0
},
{
  "key": "5.0-6.0",
  "from": 5.0,
  "from_as_string": "5.0",
  "to": 6.0,
  "to_as_string": "6.0",
  "doc_count": 0
}
  ]
}


On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>
> Besides the Terms filter in aggregation , you should use range aggregation 
> .
> It would work like..
>
> {
>   "aggs": {
> stars" : {
>"range" : { 
>   "field" : "starRating" ,
>  "ranges": [{
> "to": 5
> }, {
> "from": 5,
> "to": 10
> }, {
> "from": 10,
> "to": 15
> }, {
> "from": 15
> }]
>  
>}
> }
> }
>
>
>
> On Mon, Apr 27, 2015 at 4:18 PM, > wrote:
>
>> Hey, 
>>
>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>
>> My schema looks like that:
>>
>> { ...
>>starRating: 
>>{
>>
>>-   type: "long"
>>
>>}
>> ...
>> }
>>
>> I pass documents there, works fine.
>> I query them, using a Range Query like, works also fine: 
>>
>> {
>>   "range": {
>> "starRating": {
>>   "from": 0,
>>   "to": 5,
>> }
>>   }
>> }
>>
>> Whenever I ask for Term Aggregations like:
>>
>> {
>>   "aggs": {
>> stars" : {
>>"terms" : { "field" : "starRating" }
>> }
>> }
>>
>> OR Facets with
>>
>>   "facets" : {
>> "tag" : {
>>   "terms" : {
>> "field" : "starRating"
>>   }
>> }
>>   }
>>
>> Strange values are returned:
>>
>> "facets": {
>> "tag": {
>>   "_type": "terms",
>>   "missing": 0,
>>   "total": 3814,
>>   "other": 0,
>>   "terms": [
>> {
>>   "term": 0.0,
>>   "count": 2397
>> },
>> {
>>   "term": 1.5E-323,
>>   "count": 610
>> },
>> {
>>   "term": 2.0E-323,
>>   "count": 378
>> },
>> {
>>   "term": 1.0E-323,
>>   "count": 269
>> },
>> {
>>   "term": 4.9E-324,
>>   "count": 93
>>   

Re: Elastic Search Term Aggregation on numeric / long field

[rh]

unfortunately the same "issue".
The Ranged based Aggregation returns me:

"starRating": {
  "buckets": [
{
  "key": "0.0-0.0",
  "from": 0.0,
  "from_as_string": "0.0",
  "to": 0.0,
  "to_as_string": "0.0",
  "doc_count": 0
},
{
  "key": "1.0-1.0",
  "from": 1.0,
  "from_as_string": "1.0",
  "to": 1.0,
  "to_as_string": "1.0",
  "doc_count": 0
},
{
  "key": "2.0-2.0",
  "from": 2.0,
  "from_as_string": "2.0",
  "to": 2.0,
  "to_as_string": "2.0",
  "doc_count": 0
},
{
  "key": "3.0-3.0",
  "from": 3.0,
  "from_as_string": "3.0",
  "to": 3.0,
  "to_as_string": "3.0",
  "doc_count": 0
},
{
  "key": "4.0-4.0",
  "from": 4.0,
  "from_as_string": "4.0",
  "to": 4.0,
  "to_as_string": "4.0",
  "doc_count": 0
},
{
  "key": "5.0-5.0",
  "from": 5.0,
  "from_as_string": "5.0",
  "to": 5.0,
  "to_as_string": "5.0",
  "doc_count": 0
}
  ]
}

It looks like, that the values are read / indexed "wrong". Whats strange, 
the filtering by range works.



On Monday, April 27, 2015 at 12:52:35 PM UTC+2, deepak.chauhan wrote:
>
> Besides the Terms filter in aggregation , you should use range aggregation 
> .
> It would work like..
>
> {
>   "aggs": {
> stars" : {
>"range" : { 
>   "field" : "starRating" ,
>  "ranges": [{
> "to": 5
> }, {
> "from": 5,
> "to": 10
> }, {
> "from": 10,
> "to": 15
> }, {
> "from": 15
> }]
>  
>}
> }
> }
>
 

>
>
>
> On Mon, Apr 27, 2015 at 4:18 PM, > wrote:
>
>> Hey, 
>>
>> I'm a little bit stucked using Terms aggregations on a numeric field.
>>
>> My schema looks like that:
>>
>> { ...
>>starRating: 
>>{
>>
>>-   type: "long"
>>
>>}
>> ...
>> }
>>
>> I pass documents there, works fine.
>> I query them, using a Range Query like, works also fine: 
>>
>> {
>>   "range": {
>> "starRating": {
>>   "from": 0,
>>   "to": 5,
>> }
>>   }
>> }
>>
>> Whenever I ask for Term Aggregations like:
>>
>> {
>>   "aggs": {
>> stars" : {
>>"terms" : { "field" : "starRating" }
>> }
>> }
>>
>> OR Facets with
>>
>>   "facets" : {
>> "tag" : {
>>   "terms" : {
>> "field" : "starRating"
>>   }
>> }
>>   }
>>
>> Strange values are returned:
>>
>> "facets": {
>> "tag": {
>>   "_type": "terms",
>>   "missing": 0,
>>   "total": 3814,
>>   "other": 0,
>>   "terms": [
>> {
>>   "term": 0.0,
>>   "count": 2397
>> },
>> {
>>   "term": 1.5E-323,
>>   "count": 610
>> },
>> {
>>   "term": 2.0E-323,
>>   "count": 378
>> },
>> {
>>   "term": 1.0E-323,
>>   "count": 269
>> },
>> {
>>   "term": 4.9E-324,
>>  

Re: Elastic Search Term Aggregation on numeric / long field

[Deepak Chauhan]

Besides the Terms filter in aggregation , you should use range aggregation .
It would work like..

{
  "aggs": {
stars" : {
   "range" : {
  "field" : "starRating" ,
 "ranges": [{
"to": 5
}, {
"from": 5,
"to": 10
}, {
"from": 10,
"to": 15
}, {
"from": 15
}]

   }
}
}



On Mon, Apr 27, 2015 at 4:18 PM,  wrote:

> Hey,
>
> I'm a little bit stucked using Terms aggregations on a numeric field.
>
> My schema looks like that:
>
> { ...
>starRating:
>{
>
>-   type: "long"
>
>}
> ...
> }
>
> I pass documents there, works fine.
> I query them, using a Range Query like, works also fine:
>
> {
>   "range": {
> "starRating": {
>   "from": 0,
>   "to": 5,
> }
>   }
> }
>
> Whenever I ask for Term Aggregations like:
>
> {
>   "aggs": {
> stars" : {
>"terms" : { "field" : "starRating" }
> }
> }
>
> OR Facets with
>
>   "facets" : {
> "tag" : {
>   "terms" : {
> "field" : "starRating"
>   }
> }
>   }
>
> Strange values are returned:
>
> "facets": {
> "tag": {
>   "_type": "terms",
>   "missing": 0,
>   "total": 3814,
>   "other": 0,
>   "terms": [
> {
>   "term": 0.0,
>   "count": 2397
> },
> {
>   "term": 1.5E-323,
>   "count": 610
> },
> {
>   "term": 2.0E-323,
>   "count": 378
> },
> {
>   "term": 1.0E-323,
>   "count": 269
> },
> {
>   "term": 4.9E-324,
>   "count": 93
> },
> {
>   "term": 2.5E-323,
>   "count": 67
> }
>   ]
> }
>   },
>   "aggregations": {
> "stars": {
>   "doc_count_error_upper_bound": 0,
>   "sum_other_doc_count": 0,
>   "buckets": [
> {
>   "key": 0.0,
>   "doc_count": 2397
> },
> {
>   "key": 1.5E-323,
>   "doc_count": 610
> },
> {
>   "key": 2.0E-323,
>   "doc_count": 378
> },
> {
>   "key": 1.0E-323,
>   "doc_count": 269
> },
> {
>   "key": 4.9E-324,
>   "doc_count": 93
> },
> {
>   "key": 2.5E-323,
>   "doc_count": 67
> }
>   ]
> }
>   }
>
>
> Even in the Java Client, I get DoubleTerms instead of LongTerms.
>
> Does anybody have an Idea, what wents wrong there. I would expect
> Long-Keys and not Double keys (which are basically close to 0)
> The Histogram (with interval of 1) also does not work, everything is close
> to the zero valued key.
>
>
> Cheers, Ralf
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7c604a6a-aaf8-4645-903b-d9472868b1bd%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/7c604a6a-aaf8-4645-903b-d9472868b1bd%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHzCMprJiRGLs50K9o1OfdWYA65-RNFOmF-gWCOQTtSo2UB8Wg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search Term Aggregation on numeric / long field

[rh]

Hey, 

I'm a little bit stucked using Terms aggregations on a numeric field.

My schema looks like that:

{ ...
   starRating: 
   {
   
   -   type: "long"
   
   }
...
}

I pass documents there, works fine.
I query them, using a Range Query like, works also fine: 

{
  "range": {
"starRating": {
  "from": 0,
  "to": 5,
}
  }
}

Whenever I ask for Term Aggregations like:

{
  "aggs": {
stars" : {
   "terms" : { "field" : "starRating" }
}
}

OR Facets with

  "facets" : {
"tag" : {
  "terms" : {
"field" : "starRating"
  }
}
  }

Strange values are returned:

"facets": {
"tag": {
  "_type": "terms",
  "missing": 0,
  "total": 3814,
  "other": 0,
  "terms": [
{
  "term": 0.0,
  "count": 2397
},
{
  "term": 1.5E-323,
  "count": 610
},
{
  "term": 2.0E-323,
  "count": 378
},
{
  "term": 1.0E-323,
  "count": 269
},
{
  "term": 4.9E-324,
  "count": 93
},
{
  "term": 2.5E-323,
  "count": 67
}
  ]
}
  },
  "aggregations": {
"stars": {
  "doc_count_error_upper_bound": 0,
  "sum_other_doc_count": 0,
  "buckets": [
{
  "key": 0.0,
  "doc_count": 2397
},
{
  "key": 1.5E-323,
  "doc_count": 610
},
{
  "key": 2.0E-323,
  "doc_count": 378
},
{
  "key": 1.0E-323,
  "doc_count": 269
},
{
  "key": 4.9E-324,
  "doc_count": 93
},
{
  "key": 2.5E-323,
  "doc_count": 67
}
  ]
}
  }


Even in the Java Client, I get DoubleTerms instead of LongTerms.

Does anybody have an Idea, what wents wrong there. I would expect Long-Keys 
and not Double keys (which are basically close to 0)
The Histogram (with interval of 1) also does not work, everything is close 
to the zero valued key.


Cheers, Ralf




-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7c604a6a-aaf8-4645-903b-d9472868b1bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Count, Aggregation, Frequency: Bit Stumped

[rolandino]

Hi,
Wonder if anyone can help? 

I have the following query:

{
  "from": 0,
  "sort": [
"_score"
  ],
  "fields": [
"id",
"title",
"text"
  ],
  "query": {
"query_string": {
  "fields": [
"title",
"text"
  ],
  "query": "(\"green socks\" OR \"red socks\") AND NOT (\"yellow\" OR 
\"blue\")"
}
  },
  "size": 100
}

This works fine and is great. What I would like to do now though, is take 
the documents identified by this query, and retrieve the number of 
documents that contained "green socks", and the number that contained "red 
socks".

So, essentially apply an aggregation (I think) to the articles identified 
by the "query": "(\"green socks\" OR \"red socks\") AND NOT (\"yellow\" OR 
\"blue\")" query.

If this is an aggregation, what kind of aggregation is it?
If it isn't, what kind of query should I run?

To be clear, what i'm looking for, is at the end of whatever I query I run 
I end up with:

"green socks": 10,
"red socks": 13,

where 10 and 13 are the number of documents in the "(\"green socks\" OR 
\"red socks\") AND NOT (\"yellow\" OR \"blue\")"  result set that contain 
"green socks" at least once, and "red socks" at least once (respectively).


Any help v gratefully received,

Thanks,
R

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f2731846-7536-44ab-9df6-808a5b0f1c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nested Filter & Nested Aggregation don't work together

[BradVido]

Am I misunderstanding the Nested Filter?  I expected it to exclude any 
nested objects that didn't match it (and subsequently not match them in any 
nested aggregations for the same path).

Example:
I have a field "foo" with a mapping type of "nested".

I execute a mach-all query with a Nested Terms Filter where the field 
"foo.bar" is "value1".
I also add a Nested terms aggregation on for the same field.

The problem is that the aggregation returns things other than "value1"! 
It will return "value2" if other nested objects in the same document had 
foo.bar=value1.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b26d172f-38ab-4b6f-880c-053c12a2700f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Script to return array for scripted metric aggregation from combine

[vineeth mohan]

Hello Colin ,

You are the man :).
Seems i have a lot to learn in groovy.

Thanks a ton man , it really helped me.

Thanks
  Vineeth

On Tue, Apr 21, 2015 at 9:39 PM, Colin Goodheart-Smithe <
coling...@elastic.co> wrote:

> Vineeth,
>
> You can return any standard groovy object (by this i mean primitives,
> strings, arrays or maps) from the combine script and it will be passed to
> the reduce script. Below is a sense recreation script for a more complex
> example which counts the number of occurances of each word in a field
> (basically a crude version of the terms aggregation). Please note that
> these scripts are for test purposes and should not be used in aa production
> environment, not least because they are written in Groovy and require
> dynamic scripting to be enabled.
>
> DELETE test
>
>
> POST /test/doc/1
> {
>   "l": 10,
>   "s": "ten"
> }
> POST /test/doc/2
> {
>   "l": 4,
>   "s": "four"
> }
> POST /test/doc/3
> {
>   "l": 10,
>   "s": "ten"
> }
> POST /test/doc/4
> {
>   "l": 7,
>   "s": "seven"
> }
> POST /test/doc/5
> {
>   "l": 10,
>   "s": "ten"
> }
> POST /test/doc/6
> {
>   "l": 4,
>   "s": "four"
> }
> POST /test/doc/7
> {
>   "l": 6,
>   "s": "six"
> }
> POST /test/doc/8
> {
>   "l": 6,
>   "s": "six"
> }
>
>
> # Output of combine script on each shard is a map with a key for every
> word and values for the number of occurances of that word
> GET /test/_search?search_type=count
> {
>   "aggs": {
> "scripted_terms": {
>   "scripted_metric": {
> "init_script": "_agg['words'] = []",
> "map_script": "word = doc['s']; _agg.words.add(word.value)",
> "combine_script": "combined = [:]; for (word in _agg.words) { if
> (combined[word]) { combined[word] += 1 } else { combined[word] = 1 } };
> return combined"
>   }
> }
>   }
> }
>
>
> # Reduce script uses the map from each shard and adds together the values
> for common keys to produce a final map as output
> GET /test/_search?search_type=count
> {
>   "aggs": {
> "scripted_terms": {
>   "scripted_metric": {
> "init_script": "_agg['words'] = []",
> "map_script": "word = doc['s']; _agg.words.add(word.value)",
> "combine_script": "combined = [:]; for (word in _agg.words) { if
> (combined[word]) { combined[word] += 1 } else { combined[word] = 1 } };
> return combined",
> "reduce_script": "reduced = [:]; for (a in _aggs) { for (entry in
> a) { word = entry.key; if (reduced[word]) { reduced[word] += entry.value }
> else { reduced[word] = entry.value } } }; return reduced"
>   }
> }
>   }
> }
>
> Hope this helps,
>
> Colin
>
> On Tuesday, April 21, 2015 at 4:31:21 PM UTC+1, vineeth mohan wrote:
>>
>> Hi ,
>>
>> For scripted metric aggregation
>> <http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html>
>>  ,
>> in the example shown in the documentation , the combine script returns a
>> single number.
>>
>> Instead here , can i pass an array or hash ? I tried doing it , though it
>> did not return any error , i am not able to access those values from reduce
>> script. In reduce script per shard i am getting an instance when converted
>> to string read as 'Script2$_run_closure1@52ef3bd9'
>>
>> Kindly let me know , if this can be accomplished in any way.
>> Thanks
>>Vineeth
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/57d34e94-203f-4a4f-83ef-2e89f6ab6328%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/57d34e94-203f-4a4f-83ef-2e89f6ab6328%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5ndSHyhXX0rMnjMs9K3nQWMJ6cZH3_mSx-gsKX0q15D1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Script to return array for scripted metric aggregation from combine

[Colin Goodheart-Smithe]

Vineeth,

You can return any standard groovy object (by this i mean primitives, 
strings, arrays or maps) from the combine script and it will be passed to 
the reduce script. Below is a sense recreation script for a more complex 
example which counts the number of occurances of each word in a field 
(basically a crude version of the terms aggregation). Please note that 
these scripts are for test purposes and should not be used in aa production 
environment, not least because they are written in Groovy and require 
dynamic scripting to be enabled.

DELETE test


POST /test/doc/1
{
  "l": 10,
  "s": "ten"
}
POST /test/doc/2
{
  "l": 4,
  "s": "four"
}
POST /test/doc/3
{
  "l": 10,
  "s": "ten"
}
POST /test/doc/4
{
  "l": 7,
  "s": "seven"
}
POST /test/doc/5
{
  "l": 10,
  "s": "ten"
}
POST /test/doc/6
{
  "l": 4,
  "s": "four"
}
POST /test/doc/7
{
  "l": 6,
  "s": "six"
}
POST /test/doc/8
{
  "l": 6,
  "s": "six"
}


# Output of combine script on each shard is a map with a key for every word 
and values for the number of occurances of that word
GET /test/_search?search_type=count
{
  "aggs": {
"scripted_terms": {
  "scripted_metric": {
"init_script": "_agg['words'] = []",
"map_script": "word = doc['s']; _agg.words.add(word.value)",
"combine_script": "combined = [:]; for (word in _agg.words) { if 
(combined[word]) { combined[word] += 1 } else { combined[word] = 1 } }; 
return combined"
  }
}
  }
}


# Reduce script uses the map from each shard and adds together the values 
for common keys to produce a final map as output
GET /test/_search?search_type=count
{
  "aggs": {
"scripted_terms": {
  "scripted_metric": {
"init_script": "_agg['words'] = []",
"map_script": "word = doc['s']; _agg.words.add(word.value)",
"combine_script": "combined = [:]; for (word in _agg.words) { if 
(combined[word]) { combined[word] += 1 } else { combined[word] = 1 } }; 
return combined",
"reduce_script": "reduced = [:]; for (a in _aggs) { for (entry in 
a) { word = entry.key; if (reduced[word]) { reduced[word] += entry.value } 
else { reduced[word] = entry.value } } }; return reduced"
  }
}
  }
}

Hope this helps,

Colin

On Tuesday, April 21, 2015 at 4:31:21 PM UTC+1, vineeth mohan wrote:
>
> Hi ,
>
> For scripted metric aggregation 
> <http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html>
>  , 
> in the example shown in the documentation , the combine script returns a 
> single number.
>
> Instead here , can i pass an array or hash ? I tried doing it , though it 
> did not return any error , i am not able to access those values from reduce 
> script. In reduce script per shard i am getting an instance when converted 
> to string read as 'Script2$_run_closure1@52ef3bd9'
>
> Kindly let me know , if this can be accomplished in any way.
> Thanks
>Vineeth
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/57d34e94-203f-4a4f-83ef-2e89f6ab6328%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Script to return array for scripted metric aggregation from combine

[vineeth mohan]

Hi ,

For scripted metric aggregation
<http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html>
,
in the example shown in the documentation , the combine script returns a
single number.

Instead here , can i pass an array or hash ? I tried doing it , though it
did not return any error , i am not able to access those values from reduce
script. In reduce script per shard i am getting an instance when converted
to string read as 'Script2$_run_closure1@52ef3bd9'

Kindly let me know , if this can be accomplished in any way.
Thanks
   Vineeth

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5krMeqn%2Bmh-r2kp0kYBrF9%2Ba%3Dc09CEtSBdVzT8rSvq1AQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana - Error including scripted fields in aggregation

[ashish kudva]

Thanks for letting me know. Appreciate it.

For my future reference to ask such questions, can you us know which irc 
channel did you post that question? 


On Sunday, April 19, 2015 at 5:25:03 PM UTC-5, ke...@reversity.org wrote:
>
> I've asked on IRC and got this:
> "Unfortunatly that feature was removed due to a security issue with 
> Groovy. For now you can only use Lucene Expressions for scripted fields. We 
> are working to try and get the date functionality back."
>
>
>
> On Thursday, April 9, 2015 at 2:21:10 AM UTC+2, ashish kudva wrote:
>>
>> Hello,
>>
>> I am trying to add a manually created scripted field using the dropdown 
>> box withing Aggregation:
>>
>> Any pointers on how to resolve this error would be very helpful.
>>
>> Scriped field:
>> Name: @hourOfDay
>> Script: Integer.parseInt(new Date(doc["@timestamp"}.value).format("H"))
>>
>> Below is the error:
>>
>> Request to Elasticsearch failed: 
>> {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all 
>> shards failed; shardFailures 
>> {[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][0]: 
>> SearchParseException[[logstash-2015.04.08][0]: 
>> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
>> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
>> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>>  
>> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
>> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; 
>> nested: ParseException[ invalid sequence of tokens near 'Date' at position 
>> (21).]; nested: NoViableAltException; 
>> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][1]: 
>> SearchParseException[[logstash-2015.04.08][1]: 
>> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
>> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
>> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>>  
>> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
>> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; 
>> nested: ParseException[ invalid sequence of tokens near 'Date' at position 
>> (21).]; nested: NoViableAltException; 
>> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][2]: 
>> SearchParseException[[logstash-2015.04.08][2]: 
>> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
>> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
>> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>>  
>> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
>> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\&quo

Re: Kibana - Error including scripted fields in aggregation

[kexik]

I've asked on IRC and got this:
"Unfortunatly that feature was removed due to a security issue with Groovy. 
For now you can only use Lucene Expressions for scripted fields. We are 
working to try and get the date functionality back."



On Thursday, April 9, 2015 at 2:21:10 AM UTC+2, ashish kudva wrote:
>
> Hello,
>
> I am trying to add a manually created scripted field using the dropdown 
> box withing Aggregation:
>
> Any pointers on how to resolve this error would be very helpful.
>
> Scriped field:
> Name: @hourOfDay
> Script: Integer.parseInt(new Date(doc["@timestamp"}.value).format("H"))
>
> Below is the error:
>
> Request to Elasticsearch failed: 
> {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all 
> shards failed; shardFailures 
> {[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][0]: 
> SearchParseException[[logstash-2015.04.08][0]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][1]: 
> SearchParseException[[logstash-2015.04.08][1]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][2]: 
> SearchParseException[[logstash-2015.04.08][2]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][3]: 
> SearchParseException[[logstash-2015.04.08][3]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\&quo

Re: Kibana - Error including scripted fields in aggregation

[kexik]

The same problem here. elasticsearch 1.5.1-1 on Arch Linux community.
Were you able to make it work yet?



On Thursday, April 9, 2015 at 2:21:10 AM UTC+2, ashish kudva wrote:
>
> Hello,
>
> I am trying to add a manually created scripted field using the dropdown 
> box withing Aggregation:
>
> Any pointers on how to resolve this error would be very helpful.
>
> Scriped field:
> Name: @hourOfDay
> Script: Integer.parseInt(new Date(doc["@timestamp"}.value).format("H"))
>
> Below is the error:
>
> Request to Elasticsearch failed: 
> {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all 
> shards failed; shardFailures 
> {[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][0]: 
> SearchParseException[[logstash-2015.04.08][0]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][1]: 
> SearchParseException[[logstash-2015.04.08][1]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][2]: 
> SearchParseException[[logstash-2015.04.08][2]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
>  
> Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
>  nested: ExpressionScriptCompilationException[Failed to parse expression: 
> Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
> ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
> nested: NoViableAltException; 
> }{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][3]: 
> SearchParseException[[logstash-2015.04.08][3]: 
> query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
> 142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
> [{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},

Re: Aggregation not limited to filter?

[Ivan Brusic]

Which version are you using! The old post filter methods simply named
filter, should have been removed, or at least deprecated.

Cheers,

Ivan
On Apr 13, 2015 1:33 PM, "James Green"  wrote:

> Indeed. I had used postFilter to add my filters. The documentation for
> filters doesn't show how to use a query with a matchAll and a bunch of
> filters so I blindly followed IDE auto-complete.
>
> Lesson learned.
>
> On 10 April 2015 at 21:17, James Macdonald 
> wrote:
>
>> I had a similar problem recently and solved it by moving my filter into a
>> filtered query (leaving the query as a match_all), see documentation here
>> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
>> .
>>
>> I am not certain why filters do not restrict the scope of the aggregates,
>> but queries do, but I suspect it interprets the filter (not wrapped in a
>> filtered_query) as a post_filter (
>> http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
>> Maybe someone else actually knows why.
>>
>>
>> Hope that helps,
>> James
>>
>> On Fri, Apr 10, 2015 at 11:39 AM, James Green 
>> wrote:
>>
>>> I must be doing something stupid!
>>>
>>> Using the Java client I can perform a search with a filter and iterate
>>> over the hits. I see exactly the right source documents.
>>>
>>> If I add an aggregation, I see the expected keyAsText string but the
>>> docCount reflects the volume if the filter had not been applied.
>>>
>>> I expected the aggregation to be restricted to the results within that
>>> filter?
>>>
>>> Thanks,
>>>
>>> James
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxDfHvCicw5rewNOAun5Vy2qZe8X_awGD3wR8B-vVZY-A%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxDfHvCicw5rewNOAun5Vy2qZe8X_awGD3wR8B-vVZY-A%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCy8fZvnKZpuGFJMWXvt9MFQdUQzFO8au77mZj7r3VW0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Green]

Indeed. I had used postFilter to add my filters. The documentation for
filters doesn't show how to use a query with a matchAll and a bunch of
filters so I blindly followed IDE auto-complete.

Lesson learned.

On 10 April 2015 at 21:17, James Macdonald 
wrote:

> I had a similar problem recently and solved it by moving my filter into a
> filtered query (leaving the query as a match_all), see documentation here
> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
> .
>
> I am not certain why filters do not restrict the scope of the aggregates,
> but queries do, but I suspect it interprets the filter (not wrapped in a
> filtered_query) as a post_filter (
> http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
> Maybe someone else actually knows why.
>
>
> Hope that helps,
> James
>
> On Fri, Apr 10, 2015 at 11:39 AM, James Green 
> wrote:
>
>> I must be doing something stupid!
>>
>> Using the Java client I can perform a search with a filter and iterate
>> over the hits. I see exactly the right source documents.
>>
>> If I add an aggregation, I see the expected keyAsText string but the
>> docCount reflects the volume if the filter had not been applied.
>>
>> I expected the aggregation to be restricted to the results within that
>> filter?
>>
>> Thanks,
>>
>> James
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxhjdOQrfqy9Upsvnh%2B%2BgHmFDwqw%3DQqjHW2Z7DUm7BvJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Green]

Indeed. I had used postFilter to add my filters. The documentation for
filters doesn't show how to use a query with a matchAll and a bunch of
filters so I blindly followed IDE auto-complete.

Lesson learned.

On 10 April 2015 at 21:17, James Macdonald 
wrote:

> I had a similar problem recently and solved it by moving my filter into a
> filtered query (leaving the query as a match_all), see documentation here
> http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
> .
>
> I am not certain why filters do not restrict the scope of the aggregates,
> but queries do, but I suspect it interprets the filter (not wrapped in a
> filtered_query) as a post_filter (
> http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
> Maybe someone else actually knows why.
>
>
> Hope that helps,
> James
>
> On Fri, Apr 10, 2015 at 11:39 AM, James Green 
> wrote:
>
>> I must be doing something stupid!
>>
>> Using the Java client I can perform a search with a filter and iterate
>> over the hits. I see exactly the right source documents.
>>
>> If I add an aggregation, I see the expected keyAsText string but the
>> docCount reflects the volume if the filter had not been applied.
>>
>> I expected the aggregation to be restricted to the results within that
>> filter?
>>
>> Thanks,
>>
>> James
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxDfHvCicw5rewNOAun5Vy2qZe8X_awGD3wR8B-vVZY-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation not limited to filter?

[James Macdonald]

I had a similar problem recently and solved it by moving my filter into a
filtered query (leaving the query as a match_all), see documentation here
http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-filtered-query.html
.

I am not certain why filters do not restrict the scope of the aggregates,
but queries do, but I suspect it interprets the filter (not wrapped in a
filtered_query) as a post_filter (
http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-request-post-filter.html).
Maybe someone else actually knows why.


Hope that helps,
James

On Fri, Apr 10, 2015 at 11:39 AM, James Green 
wrote:

> I must be doing something stupid!
>
> Using the Java client I can perform a search with a filter and iterate
> over the hits. I see exactly the right source documents.
>
> If I add an aggregation, I see the expected keyAsText string but the
> docCount reflects the volume if the filter had not been applied.
>
> I expected the aggregation to be restricted to the results within that
> filter?
>
> Thanks,
>
> James
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxkmZVfDhkJW-bWPrRs5BMzTem-2zCQRWeF%2BLQCR2L5sA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAABsnTbD0JgcpMCMWuzjVC1W3C-pt6pC6PJG0xT31O44MZQs%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Kibana - Error including scripted fields in aggregation

[ashish kudva]

Hello,

I am trying to add a manually created scripted field using the dropdown box 
withing Aggregation:

Any pointers on how to resolve this error would be very helpful.

Scriped field:
Name: @hourOfDay
Script: Integer.parseInt(new Date(doc["@timestamp"}.value).format("H"))

Below is the error:

Request to Elasticsearch failed: {"error":"SearchPhaseExecutionException[Failed 
to execute phase [query], all shards failed; shardFailures 
{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][0]: 
SearchParseException[[logstash-2015.04.08][0]: 
query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
[{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
 
Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
 nested: ExpressionScriptCompilationException[Failed to parse expression: 
Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
nested: NoViableAltException; 
}{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][1]: 
SearchParseException[[logstash-2015.04.08][1]: 
query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
[{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
 
Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
 nested: ExpressionScriptCompilationException[Failed to parse expression: 
Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
nested: NoViableAltException; 
}{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][2]: 
SearchParseException[[logstash-2015.04.08][2]: 
query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
[{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
 
Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"interval\":1]]];
 nested: ExpressionScriptCompilationException[Failed to parse expression: 
Integer.parseInt(new Date(doc[\"@timestamp\"}.value).format(\"H\"))]; nested: 
ParseException[ invalid sequence of tokens near 'Date' at position (21).]; 
nested: NoViableAltException; 
}{[qsuoq24rQCmt-BkEuwe1tg][logstash-2015.04.08][3]: 
SearchParseException[[logstash-2015.04.08][3]: 
query[ConstantScore(BooleanFilter(+cache(@timestamp:[142846560 TO 
142855199])))],from[-1],size[0]: Parse Failure [Failed to parse source 
[{\"size\":0,\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":142846560,\"lte\":142855199}}}],\"must_not\":[],\"aggs\":{\"2\":{\"histogram\":{\"script\":\"Integer.parseInt(new
 
Date(doc[\\\"@timestamp\\\"}.value).format(\\\"H\\\"))\",\"lang\":\"expression\",\"i

Scripted metric aggregation over nested docs

[marko]

I'm trying to do scripted metric aggregation 
<http://www.elastic.co/guide/en/elasticsearch/reference/1.x/search-aggregations-metrics-scripted-metric-aggregation.html>
 
using Groovy with Elasticsearch 1.4.4
I've a document model where each parent document can have 0-n nested 
documents and I need to list distinct values and count their occurrences.

How can I access nested document field values in a script?
With this map_script fragment:
for (r in doc['_source.nested']) { _agg.agg.add('abc') }

I get an error saying:
"No field found for [_source.nested] in mapping with types [foo]"

I've also tried omitting "_source" from the field name and also doing a 
doc.containsKey check for the field but neither seem to work.

marko

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dc2ea339-f088-49b1-9dd9-1a428f28155b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Filters on Aggregation not working

[rakesh rakshit]

Thanks a lot Colin. Changing to lower case helped me.

Regards,
Rakesh

On Tuesday, April 7, 2015 at 7:46:43 PM UTC+5:30, Colin Goodheart-Smithe 
wrote:
>
> HI Rakesh,
>
> Term filters do not get passed through the analyzer and so need to specify 
> the term as it appears in the index. Since you are using the standard 
> analyzer your `orderPlaced` and `splashView` will have `Y` indexed as `y` 
> because the standard analyzer lowercases all terms. This is most likely to 
> be your issue. You can do one of the following to solve this:
>
> 1) Change your query to reflect the terms as they appear in the index:
>
> {
>   "aggs" : {
> "splashcount" : {
>   "filters" : {
> "filters" : {
>   "orderPlaced1" :   { "term" : { "orderPlaced" : "y"   }},
>   "splashView1" : { "term" : { "splashView" : "y" }}
> }
>   }
> }
>   }
> }
>
>
> 2) Change the mapping on those fields to specify `"index": "not_analyzed"` 
> (more info here "
> http://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#_index_2";)
>  
> and keep your query the same. Note that this will require that you re-index 
> your data as this setting on the mapping cannot be updated on an existing 
> index
>
> 3) Use the Boolean field type (
> http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#boolean).
>  
> Note that this will also require re-indexing your data
>
> Hope this helps,
>
> Colin
>
> On Tuesday, April 7, 2015 at 1:59:57 PM UTC+1, rakesh rakshit wrote:
>>
>> Hi all,
>>
>> I am using ES 1.2.1.
>>
>> I have the following type of data:
>>
>>
>> {"splashView":"Y","orderPlaced":"Y","timestamp":"1428402321850","parsed":"true","type":"cpeevent"}
>>
>> {"splashView":"Y","orderPlaced":"N","timestamp":"1428402322100","parsed":"true","type":"cpeevent"}
>>
>> The mapping for these fields are:
>>
>> "splashView":
>> {
>> "type": "string",
>> "store": true,
>> "analyzer": "standard"
>> },
>>
>>  "orderPlaced":
>> {
>> "type": "string",
>> "store": true,
>> "analyzer": "standard"
>> },
>>
>>
>> I am performing the following query:
>>
>> {
>>   "aggs" : {
>> "splashcount" : {
>>   "filters" : {
>> "filters" : {
>>   "orderPlaced1" :   { "term" : { "orderPlaced" : "Y"   }},
>>   "splashView1" : { "term" : { "splashView" : "Y" }}
>> }
>>   }
>> }
>>   }
>> }
>>
>> But not getting any response:
>>
>> {
>> "took": 3,
>> "timed_out": false,
>> "_shards":
>> {
>> "total": 5,
>> "successful": 5,
>> "failed": 0
>> },
>> "hits":
>> {
>> "total": 11,
>> "max_score": 1,
>>
>> "aggregations":
>> {
>> "splashcount":
>> {
>> "buckets":
>> {
>> "orderPlaced1":
>> {
>> "doc_count": 0
>> },
>> "splashView1":
>> {
>> "doc_count": 0
>> }
>> }
>> }
>> }
>> }
>>
>>
>> Can anyone tell me why I am not getting doc count when the data matching 
>> the filter is there?
>>
>> Regards,
>> Rakesh Kumar Rakshit
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/76ef7773-ff03-4150-886d-c9db00943baa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Filters on Aggregation not working

[Colin Goodheart-Smithe]

HI Rakesh,

Term filters do not get passed through the analyzer and so need to specify 
the term as it appears in the index. Since you are using the standard 
analyzer your `orderPlaced` and `splashView` will have `Y` indexed as `y` 
because the standard analyzer lowercases all terms. This is most likely to 
be your issue. You can do one of the following to solve this:

1) Change your query to reflect the terms as they appear in the index:

{
  "aggs" : {
"splashcount" : {
  "filters" : {
"filters" : {
  "orderPlaced1" :   { "term" : { "orderPlaced" : "y"   }},
  "splashView1" : { "term" : { "splashView" : "y" }}
}
  }
}
  }
}


2) Change the mapping on those fields to specify `"index": "not_analyzed"` 
(more info here 
"http://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#_index_2";)
 
and keep your query the same. Note that this will require that you re-index 
your data as this setting on the mapping cannot be updated on an existing 
index

3) Use the Boolean field type 
(http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#boolean).
 
Note that this will also require re-indexing your data

Hope this helps,

Colin

On Tuesday, April 7, 2015 at 1:59:57 PM UTC+1, rakesh rakshit wrote:
>
> Hi all,
>
> I am using ES 1.2.1.
>
> I have the following type of data:
>
>
> {"splashView":"Y","orderPlaced":"Y","timestamp":"1428402321850","parsed":"true","type":"cpeevent"}
>
> {"splashView":"Y","orderPlaced":"N","timestamp":"1428402322100","parsed":"true","type":"cpeevent"}
>
> The mapping for these fields are:
>
> "splashView":
> {
> "type": "string",
> "store": true,
> "analyzer": "standard"
> },
>
>  "orderPlaced":
> {
> "type": "string",
> "store": true,
> "analyzer": "standard"
> },
>
>
> I am performing the following query:
>
> {
>   "aggs" : {
> "splashcount" : {
>   "filters" : {
> "filters" : {
>   "orderPlaced1" :   { "term" : { "orderPlaced" : "Y"   }},
>   "splashView1" : { "term" : { "splashView" : "Y" }}
> }
>   }
> }
>   }
> }
>
> But not getting any response:
>
> {
> "took": 3,
> "timed_out": false,
> "_shards":
> {
> "total": 5,
> "successful": 5,
> "failed": 0
> },
> "hits":
> {
> "total": 11,
> "max_score": 1,
>
> "aggregations":
> {
> "splashcount":
> {
> "buckets":
> {
> "orderPlaced1":
> {
> "doc_count": 0
> },
> "splashView1":
> {
> "doc_count": 0
> }
> }
> }
> }
> }
>
>
> Can anyone tell me why I am not getting doc count when the data matching 
> the filter is there?
>
> Regards,
> Rakesh Kumar Rakshit
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2737b591-ab70-413b-a368-9af9f2066e16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Filters on Aggregation not working

[rakesh rakshit]

Hi all,

I am using ES 1.2.1.

I have the following type of data:

{"splashView":"Y","orderPlaced":"Y","timestamp":"1428402321850","parsed":"true","type":"cpeevent"}
{"splashView":"Y","orderPlaced":"N","timestamp":"1428402322100","parsed":"true","type":"cpeevent"}

The mapping for these fields are:

"splashView":
{
"type": "string",
"store": true,
"analyzer": "standard"
},

 "orderPlaced":
{
"type": "string",
"store": true,
"analyzer": "standard"
},


I am performing the following query:

{
  "aggs" : {
"splashcount" : {
  "filters" : {
"filters" : {
  "orderPlaced1" :   { "term" : { "orderPlaced" : "Y"   }},
  "splashView1" : { "term" : { "splashView" : "Y" }}
}
  }
}
  }
}

But not getting any response:

{
"took": 3,
"timed_out": false,
"_shards":
{
"total": 5,
"successful": 5,
"failed": 0
},
"hits":
{
"total": 11,
"max_score": 1,

"aggregations":
{
"splashcount":
{
"buckets":
{
"orderPlaced1":
{
"doc_count": 0
},
"splashView1":
{
"doc_count": 0
}
}
}
}
}


Can anyone tell me why I am not getting doc count when the data matching 
the filter is there?

Regards,
Rakesh Kumar Rakshit


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1a70b439-ed94-4e8c-a0e9-25d146ea896f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation: trouble bucketing over a text field value

[marko]

Hi,

I'm trying to bucket docs stored in Elasticsearch 1.4 based on nested 
document field values.
To use an illustrative example, I've an index with documents representing 
companies and their agents like this:

{
  "companyName": "Acme ltd.",
  "country": "us",
  "agents": [ { "name": "McKenzie, Brackman, Chaney and Kuzak" } ]
}


I'd like to count the number of companies that each agent represents and 
order agents by this count.
In SQL I would do something like this:

SELECT COUNT(*) cnt FROM agents WHERE country = 'us' AND GROUP BY name 
ORDER BY cnt DESC;

The following ES aggregation query comes close to solving this:

{
  "query": {
"term": { "country": "us" }
  },
  "aggs": {
"agents": {
  "nested": {
"path": "agents"
  },
  "aggs": {
"agent_stats": {
  "terms": {
"field": "agents.name",
"size": 9
  }
}
  }
}
  }
}

However, there's a problem that in the above case e.g. separate buckets get 
created for McKenzie and Brackman etc. instead of just one bucket. This is 
probably caused by agents.name field being currently mapped as analyzed.

One solution I've thought of is to copy agents.name field and index it also 
as not_analyzed.

Is there another way to get this aggregation query to work without index 
mapping changes?
I'd prefer to not use ES-side scripting for security reasons.

marko

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c113a1fd-1797-4ef9-b281-3f71c9d0245c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

_parent in top hits aggregation

[Maxim Valyanskiy]

Hello!

We are using script field 'doc["_parent"].value' to read parent id in tog 
hits aggregation. Is there better (more efficient) way to read it?

Maxim

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/917dd78e-4608-4b17-a11c-ea030e1b0065%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multiple sub aggregation in java api

[Grigory Rubstein]

[solved]

here is the solution:
AggregationBuilders.terms("by_gender").field("gende").size(10).

subAggregation(AggregationBuilders.terms("by_city").field("city")).

subAggregation(AggregationBuilders.terms("by_language").field("language"));



On Thursday, April 2, 2015 at 11:29:07 AM UTC+3, Grigory Rubstein wrote:
>
> Hi
>
> I wondering if the java client support multiple sub aggregation on same 
> level :
>
> for example how to translate this to java api :
> by_city and by_language is count aggregation on same level.
>
> {
>   "aggs": {
> "by_gender": {
>   "terms": {
> "field": "gender",
> "size": 10
>   },
>   "aggs": {
> "by_city": {
>   "terms": {
> "field": "city",
> "size": 3
>   }
> },
> "by_language": {
>   "terms": {
> "field": "gender",
> "size": 1
>   }
> }
>   }
> }
>   }
> }
>
> This message may contain confidential and/or privileged information. 
> If you are not the addressee or authorized to receive this on behalf of 
> the addressee you must not use, copy, disclose or take action based on this 
> message or any information herein. 
> If you have received this message in error, please advise the sender 
> immediately by reply email and delete this message. Thank you.
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d30428a3-72da-4460-85dc-a1bb205b0d9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multiple sub aggregation in java api

[Grigory Rubstein]

Hi

I wondering if the java client support multiple sub aggregation on same 
level :

for example how to translate this to java api :
by_city and by_language is count aggregation on same level.

{
  "aggs": {
"by_gender": {
  "terms": {
"field": "gender",
"size": 10
  },
  "aggs": {
"by_city": {
  "terms": {
"field": "city",
"size": 3
  }
},
"by_language": {
  "terms": {
"field": "gender",
"size": 1
  }
}
  }
}
  }
}

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d74224cb-5bb9-4fa7-bc79-ada08dbd8cc8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Nested list aggregation

[Masaru Hasegawa]

Hi,

If you define "stages" as nested type, query like this should work:

{
  "query": {
"match_all": {}
  },
  "aggs": {
"0": {
  "nested": {
"path": "msg.stat.stages"
  },
  "aggs": {
"1": {
  "terms": {
"field": "stage"
  },
  "aggs": {
"2": {
  "sum": {
"field": "duration"
  }
}
  }
}
  }
}
  }
}


Masaru

On March 24, 2015 at 21:26:33, Vasily Kirichenko (vas...@gmail.com) wrote:

I have documents like this in my index:

{
   "time": "2015-03-24T08:24:55.9056988",
   "msg": {
        "corrupted": false,
        "stat": {
            "fileSize": 10186,
            "stages": [
                {
                   "stage": "queued",
                   "duration": 420
                },
                {
                   "stage": "validate",
                   "duration": 27
                },
                {
                   "stage": "cacheFile",
                   "duration": 87
                },
                {
                   "stage": "sendResult",
                   "duration": 1332
                }
           ]
       }
   }
}

I'd like to calculate sum(msg.stat.stages.duration) grouped by 
msg.stat.stages.stage.
I tried the following:

{
  "size": 0,
  "aggs": 
  {
    "1": 
    {
      "terms": { "field": "msg.stat.stages.stage" },
      "aggs":
      {
        "2":
        {
          "nested": { "path": "stat.stages" },
          "aggs": 
          {
            "3": {
              "sum": {
                "field": "stat.stages.duration"
              }
            }
          }
        }
      }
    }
  },
  "query": {
    "match_all": {}
  }
}

and got:

{
   "took": 6,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "1": {
         "doc_count_error_upper_bound": 0,
         "sum_other_doc_count": 0,
         "buckets": [
            {
               "2": {
                  "doc_count": 0
               },
               "key": "cachefile",
               "doc_count": 1
            },
                      {
               "2": {
                  "doc_count": 0
               },
               "key": "queued",
               "doc_count": 1
            },
                 {
               "2": {
                  "doc_count": 0
               },
               "key": "sendresult",
               "doc_count": 1
            },
            {
               "2": {
                  "doc_count": 0
               },
               "key": "validate",
               "doc_count": 1
            }
         ]
      }
   }
}

which is not what I expected. Any ideas?

Thanks!
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/22173f85-6d94-4b29-a9f5-b13c46a4850d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.55126f6c.238e1f29.166%40citra-2.local.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch aggregation

[Raghav salotra]

Hi All,
I have and index which contains details of job opening with some fields 
like 
[create_date,Open_positions,skils_required,project_start_date,project_end_date]

Now my concern is to find the top 10 skills in a year based on quarter. A 
skill will be considered among the top if it has more number of openings in 
partcular quarter and number of openings are not decreasing in consecutive 
quartar.
This is really urgent any help will be highly appreciated.

Regards,
Raghav Salotra

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4de2a55b-dfbb-435e-8233-9acf7d67cae6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nested list aggregation

[Vasily Kirichenko]

I have documents like this in my index:

{
   "time": "2015-03-24T08:24:55.9056988",
   "msg": {
"corrupted": false,
"stat": {
"fileSize": 10186,
"stages": [
{
   "stage": "queued",
   "duration": 420
},
{
   "stage": "validate",
   "duration": 27
},
{
   "stage": "cacheFile",
   "duration": 87
},
{
   "stage": "sendResult",
   "duration": 1332
}
   ]
   }
   }
}

I'd like to calculate sum(msg.stat.stages.duration) grouped by 
msg.stat.stages.stage.
I tried the following:

{
  "size": 0,
  "aggs": 
  {
"1": 
{
  "terms": { "field": "msg.stat.stages.stage" },
  "aggs":
  {
"2":
{
  "nested": { "path": "stat.stages" },
  "aggs": 
  {
"3": {
  "sum": {
"field": "stat.stages.duration"
  }
}
  }
}
  }
}
  },
  "query": {
"match_all": {}
  }
}

and got:

{
   "took": 6,
   "timed_out": false,
   "_shards": {
  "total": 5,
  "successful": 5,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 0,
  "hits": []
   },
   "aggregations": {
  "1": {
 "doc_count_error_upper_bound": 0,
 "sum_other_doc_count": 0,
 "buckets": [
{
   "2": {
  "doc_count": 0
   },
   "key": "cachefile",
   "doc_count": 1
},
  {
   "2": {
  "doc_count": 0
   },
   "key": "queued",
   "doc_count": 1
},
 {
   "2": {
  "doc_count": 0
   },
   "key": "sendresult",
   "doc_count": 1
},
{
   "2": {
  "doc_count": 0
   },
   "key": "validate",
   "doc_count": 1
}
 ]
  }
   }
}

which is not what I expected. Any ideas?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/22173f85-6d94-4b29-a9f5-b13c46a4850d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kibana: how to show values directly without aggregation?

[Jason]

ELK experts, I'm desperately need your help. I've spend hours and days on 
this simple thing but still can't figure it out.

My settings are Elasticsearch 1.4.4 and Kibana 4.0.1. My program feeds data 
into ElasticSearch. The data contain a timestamp and a value. I want to 
create a bar chart, with the timestamp as X-axis and the value as Y-axis, 
directly. No aggregation. How can I make it? Appreciate any hints. Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7dedd653-c168-4f4f-ac06-30bd3a9b155c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reverse Nested Aggregation Sort

[JZ]

Hi all,

I am wondering whether it is possible to sort a reverse nested aggregation
by the document count of the joined root / document the nested object
belongs to. The counts of the nested objects and the root document counts
can be off.

I was hoping it is possible to do this in ES and not by double traversing
the array to get the sort order by document count right.

Thanks in advance!

/JZ

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAA%2BD3eV2qDcXNd_tNCE%2B6XtP5Kt_uuAd0yhC7OjW0xxcWZAVBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Terms aggregation and regex filter

[JZ]

Hi all,

With the terms aggregation, you can use include/exclude to filter the
buckets with regular expressions as described here:
http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
.

I am wondering whether it is possible to do the regex filtering before the
buckets are returned? I have used the include/exclude option now, but I am
getting incorrect total document counts.Could someone share an example
query on how to do this?

Thanks in advance!

/JZ

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAA%2BD3eXKangrYSiT29ACka%3DJ3PZborkPJvHjr4Ms-Tw9Me7Qww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Saved scripted metric aggregation in Elasticsearch and Kibana 4

[Krzysztof Zarzycki]

Hi, I'm also very interested in answer of Anna's question. 
I'll be grateful if anyone can help! 


W dniu środa, 26 listopada 2014 17:33:07 UTC+1 użytkownik Anna napisał:
>
>
> Hi Hendrik,
>
> thanks for your interest.
>
> I would like to approach the following use case:
> In Kibana 4, I would like to create data table containing a column with a 
> metric. Instead of a predefined metric (e.g., min, max, or average), I 
> would like to use a custom metric (i.e, a scripted metric) for computing a 
> value over sets of documents (i.e., buckets).
>
> From my point of view, there are two options to do this:
> (1) In Kibana's Data Table Visualization Menu, an option for copying the 
> code of the scripted metric could be included. This option might raise 
> security issues. 
> (2) The custom metric could be saved in Elasticsearch and in Kibana's Data 
> Table Visualization Menu an option for selecting this metric is included.
>
> Currently, these options seem not to be supported. Maybe I have missed an 
> option.
>
> Anna
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/98ef56b5-6524-4f9e-b028-917f789cccb4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation post processing

[Daniel Chang]

Hi all,

Are there any ways to perform post-processing on aggregation results with 
elasticsearch version 1.4.4?

My scenario is as follows: for every document of type X, we want to find 
out how many of its nested documents matches specific criteria, which are 
divided into buckets with the filters aggregation.
For example: see https://gist.github.com/mwdchang/067c64097589d6983ed3

However we want to score/sort the documents based on how many nested 
documents are matched in each bucket, with different weightings. 

For example: score = buckets[0].doc_count * 0.2 + buckets[1].doc_count * 
0.7 + buckets[2].doc_count * 0.1

I don't think this can be done inside a scripted field, as our filters 
criteria may reference other documents per 
(http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html#_terms_lookup_mechanism)
We also have close to 13 million documents so we would like to avoid 
bringing the post processing to client side.

Any help or advice would be appreciated.

Thanks,
~Daniel

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/68d23f54-9f72-4337-bbaa-589f2c07f316%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

i want to know sum aggregation result accuracy.

[hongsgo]

hello, 

dear community members.

i want to know sum aggregation result accuracy.
is it result  100% confidence possible?

http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html

it's have accuracy different for both term aggregation and sum aggregation?

http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts

please let me know.

thank you.




--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/i-want-to-know-sum-aggregation-result-accuracy-tp4072175.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1426649552579-4072175.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Multiply aggregation value by a number within the query.

[Tobi Wo]

{"took":106,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":339795,"max_score":0.0,"hits":[]},"aggregations":{"date_histogram:doc.timeframe":{"buckets":[{"key_as_string":"2015-02-27T07:00:00.000Z","key":142502040,"doc_count":864,"cardinality:doc.key":{"value":216}},{"key_as_string":"2015-02-27T08:00:00.000Z","key":142502400,"doc_count":1550,"cardinality:doc.key":{"value":322}}]}
}}


Hello, 


my result looks like the one above.

The query aggregates all keys by the hour. The value is the count of equal keys 
by the hour.


Since the keys only reflect 50% of the observations, I want to multiply each 
value by the factor 2.


So 

key = 142502040 with value = 216 

would be

key = 142502040 with value = 432


How can I achieve this?

I haven't found anything. I think scripting could be an option.

But how to reuse the aggregation in the same query?


Thanks for an answer.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d6b602f2-c3ea-40eb-9968-1991bc888618%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation / Sort and CircuitBreakingException

[joergpra...@gmail.com]

You should sort over doc values (recommended, it will be the default in
next ES version). Sorting over not_analyzed / keyword analyzed fields is
old school.

Doc values for analyzed strings make not much sense in my opinion and lead
to unwanted results. If you use multifield, then you do not have to worry
because you can set up both doc values and analyzed field.

Example:

https://gist.github.com/jprante/da2980446108b5c112a8

> The kind of filtering I'm looking for would be something like, "only
consider terms in field1 from documents where field2=valueA".

This always needs a complete load of all values of a field into the field
cache, with an inverted index. There is no lunch for free. And that's why
doc values (columnar style) were invented, to avoid this field cache
loading, for example for high cardinality fields.

Jörg

On Mon, Mar 16, 2015 at 3:17 AM, Lindsey Poole  wrote:

> Also, if I understand correctly, there are negative implications when
> sorting over a column that has been analyzed - in our case, to remove
> stop-words.
>
> Since the total cardinality of our sort field exceeds the heap available,
> we can't sort a single users documents when using stop word analysis since
> doc_values do not support analyzed fields.
>
> It seems like we'll have to preprocess the field to remove stop-words?
>
> On Sunday, March 15, 2015 at 7:01:21 PM UTC-7, Lindsey Poole wrote:
>>
>> Well, we have a field that is supporting a backward compatibility use
>> case. Clients are executing a partial match query on this field, so we used
>> the keyword tokenizer instead of not_analyzed. Since this is supporting
>> legacy functionality, the clients cannot be updated to change the
>> expectation that a partial match will return results.
>>
>> I can modify the schema and re-index so that we aggregate and sort over a
>> not_analyzed subfield instead, while executing any queries on the parent
>> field, but I wanted to verify that there is no other way to filter out
>> terms prior to loading them into the fielddata cache.
>>
>> The kind of filtering I'm looking for would be something like, "only
>> consider terms in field1 from documents where field2=valueA".
>>
>> -Lindsey
>>
>> On Sunday, March 15, 2015 at 4:43:56 PM UTC-7, Jörg Prante wrote:
>>>
>>> I mean, I do not understand what you mean by "I'm caught up on the
>>> advice to use doc_values where possible, but we have a use case where we do
>>> light analysis on a particular set of fields in our document" - what
>>> exactly prevents you from doc values?
>>>
>>> Jörg
>>>
>>> On Mon, Mar 16, 2015 at 12:41 AM, joerg...@gmail.com >> > wrote:
>>>
>>>> Have you considered doc values?
>>>>
>>>> http://www.elastic.co/guide/en/elasticsearch/guide/
>>>> current/doc-values.html
>>>>
>>>> Jörg
>>>>
>>>> On Sun, Mar 15, 2015 at 11:11 PM, Lindsey Poole 
>>>> wrote:
>>>>
>>>>> Hey guys,
>>>>>
>>>>> I have a question about the mechanics of aggregation and sorting
>>>>> w.r.t. the fielddata cache. I know this has been covered in some detail
>>>>> previously, and I'm caught up on the advice to use doc_values where
>>>>> possible, but we have a use case where we do light analysis on a 
>>>>> particular
>>>>> set of fields in our document, but also allow sorting on those fields.
>>>>>
>>>>> While we'll probably modify our schema to solve the issue, I was first
>>>>> wondering whether it is possible to filter the set of documents that ES
>>>>> aggregates / sorts over *before* pulling them into the fielddata cache? We
>>>>> have extremely high cardinality fields, but very selective queries, and it
>>>>> seems very inefficient to pull multiple gigabytes into the fielddata cache
>>>>> to select relatively few matching documents.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Lindsey
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to elasticsearc...@googlegroups.com.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc

Re: Aggregation / Sort and CircuitBreakingException

[Lindsey Poole]

Also, if I understand correctly, there are negative implications when 
sorting over a column that has been analyzed - in our case, to remove 
stop-words.

Since the total cardinality of our sort field exceeds the heap available, 
we can't sort a single users documents when using stop word analysis since 
doc_values do not support analyzed fields.

It seems like we'll have to preprocess the field to remove stop-words?

On Sunday, March 15, 2015 at 7:01:21 PM UTC-7, Lindsey Poole wrote:
>
> Well, we have a field that is supporting a backward compatibility use 
> case. Clients are executing a partial match query on this field, so we used 
> the keyword tokenizer instead of not_analyzed. Since this is supporting 
> legacy functionality, the clients cannot be updated to change the 
> expectation that a partial match will return results.
>
> I can modify the schema and re-index so that we aggregate and sort over a 
> not_analyzed subfield instead, while executing any queries on the parent 
> field, but I wanted to verify that there is no other way to filter out 
> terms prior to loading them into the fielddata cache.
>
> The kind of filtering I'm looking for would be something like, "only 
> consider terms in field1 from documents where field2=valueA".
>
> -Lindsey
>
> On Sunday, March 15, 2015 at 4:43:56 PM UTC-7, Jörg Prante wrote:
>>
>> I mean, I do not understand what you mean by "I'm caught up on the 
>> advice to use doc_values where possible, but we have a use case where we do 
>> light analysis on a particular set of fields in our document" - what 
>> exactly prevents you from doc values?
>>
>> Jörg
>>
>> On Mon, Mar 16, 2015 at 12:41 AM, joerg...@gmail.com  
>> wrote:
>>
>>> Have you considered doc values?
>>>
>>>
>>> http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html
>>>
>>> Jörg
>>>
>>> On Sun, Mar 15, 2015 at 11:11 PM, Lindsey Poole  
>>> wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I have a question about the mechanics of aggregation and sorting w.r.t. 
>>>> the fielddata cache. I know this has been covered in some detail 
>>>> previously, and I'm caught up on the advice to use doc_values where 
>>>> possible, but we have a use case where we do light analysis on a 
>>>> particular 
>>>> set of fields in our document, but also allow sorting on those fields.
>>>>
>>>> While we'll probably modify our schema to solve the issue, I was first 
>>>> wondering whether it is possible to filter the set of documents that ES 
>>>> aggregates / sorts over *before* pulling them into the fielddata cache? We 
>>>> have extremely high cardinality fields, but very selective queries, and it 
>>>> seems very inefficient to pull multiple gigabytes into the fielddata cache 
>>>> to select relatively few matching documents.
>>>>
>>>> Thanks,
>>>>
>>>> Lindsey
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com
>>>>  
>>>> <https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8027c84c-dd00-490e-a845-7fb0bb2f6107%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation / Sort and CircuitBreakingException

[Lindsey Poole]

Well, we have a field that is supporting a backward compatibility use case. 
Clients are executing a partial match query on this field, so we used the 
keyword tokenizer instead of not_analyzed. Since this is supporting legacy 
functionality, the clients cannot be updated to change the expectation that 
a partial match will return results.

I can modify the schema and re-index so that we aggregate and sort over a 
not_analyzed subfield instead, while executing any queries on the parent 
field, but I wanted to verify that there is no other way to filter out 
terms prior to loading them into the fielddata cache.

The kind of filtering I'm looking for would be something like, "only 
consider terms in field1 from documents where field2=valueA".

-Lindsey

On Sunday, March 15, 2015 at 4:43:56 PM UTC-7, Jörg Prante wrote:
>
> I mean, I do not understand what you mean by "I'm caught up on the advice 
> to use doc_values where possible, but we have a use case where we do light 
> analysis on a particular set of fields in our document" - what exactly 
> prevents you from doc values?
>
> Jörg
>
> On Mon, Mar 16, 2015 at 12:41 AM, joerg...@gmail.com  <
> joerg...@gmail.com > wrote:
>
>> Have you considered doc values?
>>
>> http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html
>>
>> Jörg
>>
>> On Sun, Mar 15, 2015 at 11:11 PM, Lindsey Poole > > wrote:
>>
>>> Hey guys,
>>>
>>> I have a question about the mechanics of aggregation and sorting w.r.t. 
>>> the fielddata cache. I know this has been covered in some detail 
>>> previously, and I'm caught up on the advice to use doc_values where 
>>> possible, but we have a use case where we do light analysis on a particular 
>>> set of fields in our document, but also allow sorting on those fields.
>>>
>>> While we'll probably modify our schema to solve the issue, I was first 
>>> wondering whether it is possible to filter the set of documents that ES 
>>> aggregates / sorts over *before* pulling them into the fielddata cache? We 
>>> have extremely high cardinality fields, but very selective queries, and it 
>>> seems very inefficient to pull multiple gigabytes into the fielddata cache 
>>> to select relatively few matching documents.
>>>
>>> Thanks,
>>>
>>> Lindsey
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0c9dc986-cfe1-42f9-ac83-d1ca40699c3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation / Sort and CircuitBreakingException

[joergpra...@gmail.com]

I mean, I do not understand what you mean by "I'm caught up on the advice
to use doc_values where possible, but we have a use case where we do light
analysis on a particular set of fields in our document" - what exactly
prevents you from doc values?

Jörg

On Mon, Mar 16, 2015 at 12:41 AM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Have you considered doc values?
>
> http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html
>
> Jörg
>
> On Sun, Mar 15, 2015 at 11:11 PM, Lindsey Poole  wrote:
>
>> Hey guys,
>>
>> I have a question about the mechanics of aggregation and sorting w.r.t.
>> the fielddata cache. I know this has been covered in some detail
>> previously, and I'm caught up on the advice to use doc_values where
>> possible, but we have a use case where we do light analysis on a particular
>> set of fields in our document, but also allow sorting on those fields.
>>
>> While we'll probably modify our schema to solve the issue, I was first
>> wondering whether it is possible to filter the set of documents that ES
>> aggregates / sorts over *before* pulling them into the fielddata cache? We
>> have extremely high cardinality fields, but very selective queries, and it
>> seems very inefficient to pull multiple gigabytes into the fielddata cache
>> to select relatively few matching documents.
>>
>> Thanks,
>>
>> Lindsey
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGD8qRCq6k6MwK4ujnWYfYv%2BGzdqn45GA6a6Gv4jHcUWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation / Sort and CircuitBreakingException

[joergpra...@gmail.com]

Have you considered doc values?

http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html

Jörg

On Sun, Mar 15, 2015 at 11:11 PM, Lindsey Poole  wrote:

> Hey guys,
>
> I have a question about the mechanics of aggregation and sorting w.r.t.
> the fielddata cache. I know this has been covered in some detail
> previously, and I'm caught up on the advice to use doc_values where
> possible, but we have a use case where we do light analysis on a particular
> set of fields in our document, but also allow sorting on those fields.
>
> While we'll probably modify our schema to solve the issue, I was first
> wondering whether it is possible to filter the set of documents that ES
> aggregates / sorts over *before* pulling them into the fielddata cache? We
> have extremely high cardinality fields, but very selective queries, and it
> seems very inefficient to pull multiple gigabytes into the fielddata cache
> to select relatively few matching documents.
>
> Thanks,
>
> Lindsey
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFgpwVbkkAsKK11m74qqE_avwQ5mmMGb2z1w0-qH5hNMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation / Sort and CircuitBreakingException

[Lindsey Poole]

Hey guys,

I have a question about the mechanics of aggregation and sorting w.r.t. the 
fielddata cache. I know this has been covered in some detail previously, 
and I'm caught up on the advice to use doc_values where possible, but we 
have a use case where we do light analysis on a particular set of fields in 
our document, but also allow sorting on those fields.

While we'll probably modify our schema to solve the issue, I was first 
wondering whether it is possible to filter the set of documents that ES 
aggregates / sorts over *before* pulling them into the fielddata cache? We 
have extremely high cardinality fields, but very selective queries, and it 
seems very inefficient to pull multiple gigabytes into the fielddata cache 
to select relatively few matching documents.

Thanks,

Lindsey

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e32cf7c3-e2b3-48e9-bc7c-d7f2e0016835%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: children aggregation

[Adrien Grand]

Hi,

This aggregation works with parent/child functionality which requires that
parents and children are in the sane shard. So having parents and children
in different indexes is not possible.

See
http://www.elastic.co/guide/en/elasticsearch/guide/current/parent-child.html

On Tue, Mar 10, 2015 at 5:22 AM, kazoompa  wrote:

> Hi,
>
> I was wondering whether this aggregation will work with parent documents
> residing in different indexes than the children documents? Are there any
> limitations with respect to shards? I remember parent-child relationships
> have shard limitation as stated here: http://goo.gl/gU6Mcx.
>
> In short, I would like to know the limitations of this aggregation type
> since there were none described in the children agg documentation:
> http://goo.gl/Y3yfhc.
>
>
> Thanks a lot.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/055b8b28-d403-483e-92cb-a1b3ae04b252%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/055b8b28-d403-483e-92cb-a1b3ae04b252%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4AhbQsVTk%2BuzdVy2W_HK_5qk9YaDROxzgcYx3_fzv%3Dtw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: fielddata and date_histogram aggregation

[Anant Aneja]

Ignore the bit about high cardinality.

IMO, the use of fielddata for date_historgram is counter intuitive, since 
the buckets made a ranges as defined by the grain, and therefore the 
buckets are pre-defined in a sense which means the inverted index should 
itself be sufficient for building the buckets, fielddata is not required.

Is this how a date_histogram actually behaves under the covers ?

On Tuesday, March 10, 2015 at 1:43:13 PM UTC-7, Anant Aneja wrote:
>
> I was reading up on when ES builds fielddata : 
> https://www.elastic.co/guide/en/elasticsearch/guide/current/fielddata.html
>
> A footnote on this page reads 'It is required for any operation that needs 
> to look up the value contained in a specific document'.
>
> Would this also be true when building buckets in a date_histogram 
> aggregation ? Date fields are bound to have high cardinalilty so I would 
> imagine that if populating fielddata would be counterproductive.
>
> In what cases while building aggregations does the above mentioned pattern 
> not apply ?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fdd817e6-7ac2-49be-862b-947492da2dda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

fielddata and date_histogram aggregation

[Anant Aneja]

I was reading up on when ES builds fielddata 
: https://www.elastic.co/guide/en/elasticsearch/guide/current/fielddata.html

A footnote on this page reads 'It is required for any operation that needs 
to look up the value contained in a specific document'.

Would this also be true when building buckets in a date_histogram 
aggregation ? Date fields are bound to have high cardinalilty so I would 
imagine that if populating fielddata would be counterproductive.

In what cases while building aggregations does the above mentioned pattern 
not apply ?


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/846fb94b-d390-434b-bc7f-58885bf477ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

children aggregation

[kazoompa]

Hi,

I was wondering whether this aggregation will work with parent documents 
residing in different indexes than the children documents? Are there any 
limitations with respect to shards? I remember parent-child relationships 
have shard limitation as stated here: http://goo.gl/gU6Mcx.

In short, I would like to know the limitations of this aggregation type 
since there were none described in the children agg 
documentation: http://goo.gl/Y3yfhc.


Thanks a lot.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/055b8b28-d403-483e-92cb-a1b3ae04b252%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stats aggregation: value script broken due to Groovy scripting disabled

[BradVido]

Great. That worked and thanks for opening the github issue! I see that 
others agree this needs to be fixed.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6d8a438c-b1d4-4901-88b6-7d82d09a887c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stats aggregation: value script broken due to Groovy scripting disabled

[Adrien Grand]

Hi Brad,

Giving the name of the file as a script (without extension) should work.
However I agree this is confusing and opened
https://github.com/elasticsearch/elasticsearch/issues/10011.

On Thu, Mar 5, 2015 at 6:32 PM, BradVido  wrote:

> I understand that Groovy dynamic scripting has been disabled by default in
> 1.4.4 and I should either use a different language or put the script in the
> config/scripts directory.
>
> However, when calling a script for the value of a Stats aggregation
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-stats-aggregation.html#_script_5>,
> I cannot figure out how to specify the script_file parameter. I'm using the
> Java API and the StatsBuilder class has only 1 method for the script,
> which accepts a sting:
>  StatsBuilder.stats(String script)
>
> If I specify the script string, I get (as expected):
> ScriptException[dynamic scripting for [groovy] disabled]
>
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/83258db0-c815-4508-8adb-d5be4d7ee0a5%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/83258db0-c815-4508-8adb-d5be4d7ee0a5%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6uSqXRURWPStKpfdo%3DJ4JV9F4s-2Dw0UKSuR7XOpcUkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Stats aggregation: value script broken due to Groovy scripting disabled

[BradVido]

I understand that Groovy dynamic scripting has been disabled by default in 
1.4.4 and I should either use a different language or put the script in the 
config/scripts directory.

However, when calling a script for the value of a Stats aggregation 
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-stats-aggregation.html#_script_5>,
 
I cannot figure out how to specify the script_file parameter. I'm using the 
Java API and the StatsBuilder class has only 1 method for the script, which 
accepts a sting:
 StatsBuilder.stats(String script)

If I specify the script string, I get (as expected): 
ScriptException[dynamic scripting for [groovy] disabled]


Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/83258db0-c815-4508-8adb-d5be4d7ee0a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multisort on terms aggregation

[Colin Goodheart-Smithe]

In the currently released versions it is not possible to sort by multiple 
criteria in the terms aggregations. This 
functionality is coming in 1.5 from this PR: 
https://github.com/elasticsearch/elasticsearch/pull/7662.

The syntax will be a bit different from what you tried in your gist. have a 
look 
at 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order
 
to see the syntax that is coming in 1.5 for this.

Colin


On Tuesday, 3 March 2015 17:00:15 UTC, Demetrioca wrote:
>
> Is there a way to do a sorting by multiple metrics in the terms 
> aggregations?
> Here's the way I tried: https://gist.github.com/2e3s/599330bfef10e9375f6f 
> and this doesn't work.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/614fd9a1-5c88-4cd2-9596-b01a5d47cdad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multisort on terms aggregation

[Demetrioca]

Is there a way to do a sorting by multiple metrics in the terms 
aggregations?
Here's the way I tried: https://gist.github.com/2e3s/599330bfef10e9375f6f 
and this doesn't work.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f259c8ce-ac62-487b-af1c-21a19e230eac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sum in Nested Date Range Aggregation Not Handling Negative Values

[Nate Nash]

Hello - 

I have a an index with 30M documents, spread across 4 nodes, 30 shards, 
with 2 replicas each. Each document has a nested object, representing a 
transaction log entry. Each transaction log entry has a positive or 
negative dollar value and a timestamp. When I attempt to bucket the nested 
objects using the date range and sum aggregations, the sum aggregation 
appears to break on transactions with both positive and negative numbers. 

For example, I use the follow query. I am including a reverse nested agg to 
show that the nested objects are different than the output of the sum agg. 
Enter code here

  "size": 0,
  "query": {
"filtered": {
  "query": {
"match_all": {}
  },
  "filter": {
"bool": {
  "must": [
{
  "term": {
"foo._id": "12345"
  }
},
{
  "range": {
"timestamp": {
  "gte": "2011-10-01"
}
  }
},
{
  "term": {
"bar.name": "A Special Name"
  }
}
  ]
}
  }
}
  },
  "aggs": {
"BAR": {
  "terms": {
"field": "bar.name",
"size": 2,
"order": {
  "NESTED>TOTAL": "asc"
}
  },
  "aggs": {
"NESTED": {
  "nested": {
"path": "a"
  },
  "aggs": {
"TOTAL": {
  "sum": {
"field": "a.obligatedamount"
  }
},
"DATES": {
  "date_range": {
"field": "a.signeddate",
"keyed": true,
"ranges": [
  {
"key": "FY2012",
"from": "2011-10-01",
"to": "2012-09-30"
  },
  {
"key": "FY2013",
"from": "2012-10-01",
"to": "2013-09-30"
  },
  {
"key": "FY2014",
"from": "2013-10-01",
"to": "2014-09-30"
  },
  {
"key": "FY2015",
"from": "2014-10-01",
"to": "2015-09-30"
  }
]
  },
  "aggs": {
"DATEBUCKET_SUBTOTAL": {
  "sum": {
"field": "a.obligatedamount"
  }
},
"HITS_REVERSE" : { 
  "reverse_nested": {},
  "aggs": {
"HITS": {
  "top_hits": {
"_source": {
"include": [
"a.obligatedamount",
"a.signeddate"
]
}
  }
}
  }
}
  }
}
  }
}
  }
}
  }
}...

The following result is returned. Note the odd number in the 
DATEBUCKET_SUBTOTAL in FY2012 for "A Special Place". This appears to only 
be an issue when the transaction log contains both negative and positive 
numbers. 

{
   "took": 11,
   "timed_out": false,
   "_shards": {
  "total": 30,
  "successful": 30,
  "failed": 0
   },
   "hits": {
  "total": 2,
  "max_score": 0,
  "hits": []
   },
   "aggregations": {
  "BAR": {
 "doc_count_error_upper_bound": 0,
 "sum_other_doc_count": 0,
 "buckets": [
{
   "key": "A Special Place",
   "doc_count": 2,
   "NESTED": {
  "doc_count": 3,
  "DATES": {
 "buc

Help with aggregation

[tomas . mozes]

Hello,
suppose we have documents that only have a single entry (uid in form of a 
string like "12345678"). Now we can make an aggegation on terms to see how 
many of these occur. That is ok. And from that, we would like to know the 
distribution of counts of that uid

So the entries are:
... "uid": "1234"
... "uid": "5612"
... "uid": "1234"
... "uid": ""
... "uid": "5612"

if we make the aggregation we get:
"1234": 2
"5612": 2
"": 1

And the final step is that we would like numbers like:
2: 2 (there are 2 groups with count 2 - "1234" and "5612")
1: 1 (there is 1 group with count 1 - "")

How is that possible please?
Thank you!

Tomas

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a25ca9cb-54f9-47c1-ab02-0b17a95a89f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Modeling index for aggregation performance

[Adrien Grand]

The overhead of shards boils down to the fact that shards are Lucene
indices. There is not exact number for the right number of shards per node
although 3 is certainly OK and 100 probably too much. It's true that today
shards are processed in a single thread, so having fewer larger shards
might increase latency a bit (but it does not hurt throughput however).

The chapter about capacity planning from the book might be helpful too:
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scale.html

On Fri, Feb 27, 2015 at 6:39 AM, Justin Warkentin <
justin.warken...@gmail.com> wrote:

> How can I learn more about the overhead of shards? As I understand it, the
> more documents there are per shard, the slower query performance is. If
> there is only one shard per index then there is no parallelization of the
> query. Right now I have an average of 60 million records per month going
> into an index with the default of 5 shards and I'm creating a new index
> each month. That means there's an average of about 12 million records per
> shard. Perhaps I should decrease the number of shards, but I'm concerned
> about query performance if I get too many documents per shard. With this
> scheme that would leave me with 60 new shards per year. Of course, I have 4
> data nodes and I'm planning to expand this further so they don't all have
> to be on one node.
>
> I guess what I'd really like to understand then is, what's a reasonable
> number of shards per node? What's the overhead of a new active primary
> shard? I can continue to spin up servers and scale horizontally adding new
> nodes to handle the shards, I just need a good way to gauge how to tell
> when to add a node to the cluster. Also, I imagine if I'm searching across
> all indices it will wreck query performance just because at some point
> there will be so many that the parallelization itself introduces too much
> overhead. Hence, the idea of limiting article hits to a subset of indices
> based on both article id and date to prevent queries from having to touch
> everything as the data grows.
>
>
> On Thursday, February 26, 2015 at 4:31:44 PM UTC-7, Adrien Grand wrote:
>>
>> For 1., storing all hits for an article in the same index would not help
>> performance. ALso note that you mentioned numbers of indices but what
>> really matters to Elasticsearch is the total number of primary shards.
>> Having 60 indices with 1 shard is much better to Elasticsearch than one
>> index with 1000 shards.
>>
>> > Would it be better to store a hit counter on the article record itself
>> that gets updated occasionally?
>>
>> I think this is an option that you should consider indeed. Running a
>> query with a sort is much more efficient than running a terms aggregation
>> to compute the article that has most hits.
>>
>> --
>> Adrien Grand
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/6a462d71-2904-4d3b-b75d-c03d6de6a122%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/6a462d71-2904-4d3b-b75d-c03d6de6a122%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6BMp7%3DR-m3GWfzgnANsOFVySehuCB2BoAUA5m1Egzr5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Modeling index for aggregation performance

[Justin Warkentin]

How can I learn more about the overhead of shards? As I understand it, the 
more documents there are per shard, the slower query performance is. If 
there is only one shard per index then there is no parallelization of the 
query. Right now I have an average of 60 million records per month going 
into an index with the default of 5 shards and I'm creating a new index 
each month. That means there's an average of about 12 million records per 
shard. Perhaps I should decrease the number of shards, but I'm concerned 
about query performance if I get too many documents per shard. With this 
scheme that would leave me with 60 new shards per year. Of course, I have 4 
data nodes and I'm planning to expand this further so they don't all have 
to be on one node.

I guess what I'd really like to understand then is, what's a reasonable 
number of shards per node? What's the overhead of a new active primary 
shard? I can continue to spin up servers and scale horizontally adding new 
nodes to handle the shards, I just need a good way to gauge how to tell 
when to add a node to the cluster. Also, I imagine if I'm searching across 
all indices it will wreck query performance just because at some point 
there will be so many that the parallelization itself introduces too much 
overhead. Hence, the idea of limiting article hits to a subset of indices 
based on both article id and date to prevent queries from having to touch 
everything as the data grows.

On Thursday, February 26, 2015 at 4:31:44 PM UTC-7, Adrien Grand wrote:
>
> For 1., storing all hits for an article in the same index would not help 
> performance. ALso note that you mentioned numbers of indices but what 
> really matters to Elasticsearch is the total number of primary shards. 
> Having 60 indices with 1 shard is much better to Elasticsearch than one 
> index with 1000 shards.
>
> > Would it be better to store a hit counter on the article record itself 
> that gets updated occasionally?
>
> I think this is an option that you should consider indeed. Running a query 
> with a sort is much more efficient than running a terms aggregation to 
> compute the article that has most hits.
>
> -- 
> Adrien Grand
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a462d71-2904-4d3b-b75d-c03d6de6a122%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Modeling index for aggregation performance

[Adrien Grand]

For 1., storing all hits for an article in the same index would not help
performance. ALso note that you mentioned numbers of indices but what
really matters to Elasticsearch is the total number of primary shards.
Having 60 indices with 1 shard is much better to Elasticsearch than one
index with 1000 shards.

> Would it be better to store a hit counter on the article record itself
that gets updated occasionally?

I think this is an option that you should consider indeed. Running a query
with a sort is much more efficient than running a terms aggregation to
compute the article that has most hits.

-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5Vt6tbxqe3eG6OxY-L4NsnarfkjbmDp5oTP4T0E1xjuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: From ES aggregation result to a List of Maps (Java)

[Adrien Grand]

There is no automatic way to do that. I believe you need to have special
handling of this on client side for the aggregations that you plan to
support.

On Wed, Feb 25, 2015 at 11:09 PM, Sven Jörns  wrote:

> Hi,
>
> what is the best way to convert an ES Aggregation result to a List of Maps
> in Java using Java API?
>
> I want to display the results into a pivot-table or in the first step just
> print out the filled rows into the console.
>
> Thanks
> Sven
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5c172b60-602c-44d5-9ae7-e57949b01bf1%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/5c172b60-602c-44d5-9ae7-e57949b01bf1%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5diNS05BsSnP6bB7MzaeMgS1yhtiYfOC8E9ZAkbx5LKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Bisect buckets on filter aggregation

[Adrien Grand]

Hi Ross,

There is no way to do it today without repeating the filter and nesting it
inside of a `not` filter. We are considering adding info about missing and
other buckets to our terms aggregation, you can read the discussion at
https://github.com/elasticsearch/elasticsearch/issues/5324

On Thu, Feb 26, 2015 at 4:00 PM, Ross Duncan  wrote:

> Hi there,
>
> Im a little new to the nuances of building aggregations, but essentially I
> am trying to construct an aggregation which results in an "in" bucket and
> an "out" bucket with respect to some predicate (filter?) that I want to
> apply.
>
> I can easily achieve the in-bucket by using a filter aggregation, but if I
> also want to see the inverse to this filter I would rather not have to
> create (repeat) the filter to identify the "out" set.
>
> Is there an easy way to do this?
>
> Thanks,
> Ross
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b2a8e295-6db1-49ce-af2d-d78638f9cf48%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/b2a8e295-6db1-49ce-af2d-d78638f9cf48%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j74_iX33r-3n%2BzbwyXWGnTm_-Ci6gDCyv8d2u%3D-8CQNew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Bisect buckets on filter aggregation

[Ross Duncan]

Hi there,

Im a little new to the nuances of building aggregations, but essentially I 
am trying to construct an aggregation which results in an "in" bucket and 
an "out" bucket with respect to some predicate (filter?) that I want to 
apply.

I can easily achieve the in-bucket by using a filter aggregation, but if I 
also want to see the inverse to this filter I would rather not have to 
create (repeat) the filter to identify the "out" set.

Is there an easy way to do this? 

Thanks,
Ross

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b2a8e295-6db1-49ce-af2d-d78638f9cf48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

From ES aggregation result to a List of Maps (Java)

[Sven Jörns]

Hi,

what is the best way to convert an ES Aggregation result to a List of Maps 
in Java using Java API?

I want to display the results into a pivot-table or in the first step just 
print out the filled rows into the console.

Thanks
Sven

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5c172b60-602c-44d5-9ae7-e57949b01bf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kiban4 Issue: How to adjust interval value in the case of date histogram aggregation

[cong yue]

Hi
 I found for Kibana4, the interval value of date histogram aggregation can 
not be adjusted as kibana3. Now my visualization object json is like
--
visState
*{*
*  "aggs": [*
*{*
*  "id": "1",*
*  "params": {*
*"field": "cacheCode"*
*  },*
*  "schema": "metric",*
*  "type": "avg"*
*},*
*{*
*  "id": "2",*
*  "params": {*
*"extended_bounds": {},*
*"field": "accessTime",*
*"interval": "10minutes",*
*"min_doc_count": 1*
*  },*
*  "schema": "segment",*
*  "type": "date_histogram"*
*}*
*  ],*
*  "listeners": {},*
*  "params": {*
*"addLegend": true,*
*"addTooltip": true,*
*"defaultYExtents": false,*
*"shareYAxis": true*
*  },*
*  "type": "line"*
*}*


kibanaSavedObjectMeta.searchSourceJSON
*{*
*  "query": {*
*"query_string": {*
*  "analyze_wildcard": true,*
*  "query": "*"*
*}*
*  },*
*  "filter": []*
*}*

*--*
*I want to filter my query like*
*---*
*#Cache hit ratio for timeline baseGET /ats/_search{  "query": {
"filtered": {  "query": {"match_all": {}  },  "filter": 
{"range": {"accessTime": {  "gte": "now-1d/d"}  
}}}  },   "size": 0,  "aggs": {"accessTimes": {  
"date_histogram": {"field": "accessTime","interval": "10m"  
},  "aggs": {"hit_ratio": {  "avg": {
"field": "cacheCode"  }}  }}  }}*
*---*


*How I can do this from kiban4? Kibana is with real cool of new charts, new 
menu of discovery and visualization menus, but I still can not how I 
customize the query and filter for kibana4. Always the setting in the top 
bar will take effect. Is this the limitation of kibana4? I want to do some 
similar thing as marvel do for my server applications. May I have to roll 
back to kibana3 to do this?*

*thanks,*
*Cong*

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/96f76542-c2d6-409e-b6b1-5dbc9f97f2cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Modeling index for aggregation performance

[Justin Warkentin]

I'm a bit stuck trying to figure out the best way to model my indices for 
aggregations. I'm currently storing article hits in indices that roll over 
each month. Each index tends to have around 60M records. However, I have 
two concerns:

1. In the future I expect the number of indices will grow into the 
hundreds. If I'm trying to aggregate the total number of hits or the hits 
per month of an article across the many indices, will the query end up 
getting very slow since it has to aggregate across them all? Would it be 
better to store all the hits for an article in the same index and use a new 
index for blocks of article IDs instead of a new index per month to make 
the index predictable for a certain article?

2. What about when I want to see what the top 10 articles of all time are? 
This would require doing an aggregation of all articles across all indices, 
right? How slow will that get when there are hundreds of indices with 60M+ 
records per index? Would it be better to store a hit counter on the article 
record itself that gets updated occasionally?

Is there a better way to model the indices that would accommodate both of 
these use cases?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2758357b-dadf-4097-917a-0cc54ca2109e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: concerns on possible load of aggregation

[Jilles van Gurp]

You need to look into using an index template that uses optimal mapping for 
your data. For logstash, it really helps to use doc_values on all fields 
you aggregate on and turning off norms as well on those fields. Doc_values 
means elasticsearch uses memory mapped files instead of heap memory for the 
field values. WIth huge aggregations this means the system will get slower 
but less likely to run out of memory if you get a lot of requests. Without 
doc_values, you will want to configure field data circuitbreakers properly 
to ensure you don't run out of memory. This typically means that searches 
that would have run out of memory abort with an error instead, which is 
preferable to your cluster crashing but not great from an end user 
perspective.

Jilles

On Wednesday, February 25, 2015 at 9:09:43 AM UTC+1, Seungjin Lee wrote:
>
> We are running a PAAS built with elasticsearch and we want to provide 
> multi-column count aggregation feature through ES aggregation
>
> Let's take below as an example
>
> POST /INDEX_PATTREN-*/_search
> {
> "query":{"match":{"project":"dummyProject"}},
> "size":0,
>"aggs": {
>   "col1": {
>  "terms": {
> "field": "host",
> "size":5
>  },
>  "aggs": {
> "col2": {
>"terms": {
>   "field": "source",
>   "size":5
>},
>"aggs":{
>"col3":{
>"terms":{
>"field":"version",
>"size":5
>}
>}
>}
> }
>  }
>   }
>}
> }
>
>
> We use daily index, stores 30 days amount of data, approximately 500GB per 
> day index.
>
> So the example aggreagation will investigate huge data.
>
> But we found out that it's blazingly fast, we use 20 data nodes together 
> with several search/master nodes, and it responds within 10 minutes.
>
>
>
>
> OK, but what if there's many request at the same time, what can happen?
>
> Will those requests just make other requests to slow down(in this case, 
> increase # of machines will be a solution?) or possibly cause OOM or 
> whatever critical error on ES daemon? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/745a95f9-d963-472c-9ece-f326521707b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

concerns on possible load of aggregation

[Seungjin Lee]

We are running a PAAS built with elasticsearch and we want to provide
multi-column count aggregation feature through ES aggregation

Let's take below as an example

POST /INDEX_PATTREN-*/_search
{
"query":{"match":{"project":"dummyProject"}},
"size":0,
   "aggs": {
  "col1": {
 "terms": {
"field": "host",
"size":5
 },
 "aggs": {
"col2": {
   "terms": {
  "field": "source",
  "size":5
   },
   "aggs":{
   "col3":{
   "terms":{
   "field":"version",
   "size":5
   }
   }
   }
}
 }
  }
   }
}


We use daily index, stores 30 days amount of data, approximately 500GB per
day index.

So the example aggreagation will investigate huge data.

But we found out that it's blazingly fast, we use 20 data nodes together
with several search/master nodes, and it responds within 10 minutes.




OK, but what if there's many request at the same time, what can happen?

Will those requests just make other requests to slow down(in this case,
increase # of machines will be a solution?) or possibly cause OOM or
whatever critical error on ES daemon?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL3_U43m1UuZbAHPwSNzQHC-xpBxGsr%2B%3DGNt-GUeMCueoyTP0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java Client Aggregation question

[David Pilato]

Yes. Change your mapping and define the field as not_analyzed.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 




> Le 20 févr. 2015 à 12:32, Matt Williams  a écrit :
> 
> Hi all,
> 
> I am currently indexing tags (industries) for an entity with a data structure 
> like this:
> 
> industry: ["Consulting & Recruitment","Professional Services","Education & 
> Training"] 
> I am applying a termsAggregation to the query as:
> 
> AggregationBuilders.terms("industry").field("industry");
> What I expect to come out:
> 
> Key: "Consulting & Recruitment"
> 
> docCount: 100
> 
> What I actually get:
> 
> Key: "Consulting"
> 
> docCount: 100
> 
> Key: "Recruitment"
> 
> docCount: 100.
> 
> Is there a way to correct this?
> 
> Thanks
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/e86c02ab-9699-41ea-88b3-871ed761dad6%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6A33DAF3-C491-4EBA-8C1D-77F83C52160F%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Java Client Aggregation question

[Matt Williams]

Hi all,

I am currently indexing tags (industries) for an entity with a data 
structure like this:

industry: ["Consulting & Recruitment","Professional Services","Education & 
Training"] 

I am applying a termsAggregation to the query as:

AggregationBuilders.terms("industry").field("industry");

What I expect to come out:

Key: "Consulting & Recruitment"

docCount: 100

What I actually get:

Key: "Consulting"

docCount: 100

Key: "Recruitment"

docCount: 100.

Is there a way to correct this?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e86c02ab-9699-41ea-88b3-871ed761dad6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Range Aggregation - Default 'key' [ES 1.1.1]

[Colin Goodheart-Smithe]

How many nodes do you have in your cluster? 

If you have 2 nodes then it would suggest that there is something different 
between the two nodes which is causing this and your request is alternating 
between each of the nodes. 

My first guess would be that the nodes are accidentally on different 
versions. 

I would start by using the nodes API to confirm the nodes are all on the 
same version of Elasticsearch:

curl -XGET "http://localhost:9200/_nodes/settings";


Hope this helps

Colin 

On Tuesday, 17 February 2015 17:39:16 UTC, Scott Rallya wrote:
>
> In running the following range aggregation [choosing not to specify a 
> 'key' for each range]
>
> {
>   "query": {
> "match_all": {}
>   },
>   "aggs": {
> "duration": {
>   "range": {
> "field": "duration",
> "ranges": [
>   { "to": 60 },
>   { "from": 60, "to": 300},
>   { "from": 300, "to": 900},
>   { "from": 900, "to": 3600},
>   { "from": 3600 }
>  ]
>   }
> }
>   }
> }
>
> I seem to be alternating between two result sets, the first request will 
> return:
> "aggregations": {
> "duration": {
> "buckets": [
> {
> "to": 60,
> "doc_count": 157680
> },
> {
> "from": 60,
> "to": 300,
> "doc_count": 181398
> },
> {
> "from": 300,
> "to": 900,
> "doc_count": 39937
> },
> {
> "from": 900,
> "to": 3600,
> "doc_count": 8809
> },
> {
> "from": 3600,
> "doc_count": 298
> }
> ]
> }
> }
>
> And the subsequent request will return:
> "aggregations": {
> "duration": {
> "buckets": [
> {
> "key": "*-60.0",
> "to": 60,
> "doc_count": 157680
> },
> {
> "key": "60.0-300.0",
> "from": 60,
> "to": 300,
> "doc_count": 181398
> },
> {
> "key": "300.0-900.0",
> "from": 300,
> "to": 900,
> "doc_count": 39937
> },
> {
> "key": "900.0-3600.0",
> "from": 900,
> "to": 3600,
> "doc_count": 8809
> },
> {
> "key": "3600.0-*",
> "from": 3600,
> "doc_count": 298
> }
> ]
> }
> }
>
> Each request afterwards alternates between "key" being absent from each 
> bucket in the list and then being present. Was hoping someone might have 
> some insight as to what is going on just to satisfy my own curiosity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/65724db4-f1a0-4015-8a3d-924c7b1f73a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Range Aggregation - Default 'key' [ES 1.1.1]

[Scott R]

In running the following range aggregation [choosing not to specify a 'key' 
for each range]

{
  "query": {
"match_all": {}
  },
  "aggs": {
"duration": {
  "range": {
"field": "duration",
"ranges": [
  { "to": 60 },
  { "from": 60, "to": 300},
  { "from": 300, "to": 900},
  { "from": 900, "to": 3600},
  { "from": 3600 }
 ]
  }
}
  }
}

I seem to be alternating between two result sets, the first request will 
return:
"aggregations": {
"duration": {
"buckets": [
{
"to": 60,
"doc_count": 157680
},
{
"from": 60,
"to": 300,
"doc_count": 181398
},
{
"from": 300,
"to": 900,
"doc_count": 39937
},
{
"from": 900,
"to": 3600,
"doc_count": 8809
},
{
"from": 3600,
"doc_count": 298
}
]
}
}

And the subsequent request will return:
"aggregations": {
"duration": {
"buckets": [
{
"key": "*-60.0",
"to": 60,
"doc_count": 157680
},
{
"key": "60.0-300.0",
"from": 60,
"to": 300,
"doc_count": 181398
},
{
"key": "300.0-900.0",
"from": 300,
"to": 900,
"doc_count": 39937
},
{
"key": "900.0-3600.0",
"from": 900,
"to": 3600,
"doc_count": 8809
},
{
"key": "3600.0-*",
"from": 3600,
"doc_count": 298
}
]
}
}

Each request afterwards alternates between "key" being absent from each 
bucket in the list and then being present. Was hoping someone might have 
some insight as to what is going on just to satisfy my own curiosity.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/840a3619-2736-49e9-97b3-06a5b2fea9d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

aggregation with nested objects/

[Grigory Rubstein]

Hello

Few word about the schema , i have one type of document (Reviews) that 
contain list of reviews (nested object) each review has folowing fields : 
*polarity*(negative or opposite ), *keyword*(main word of the review) , 
*reviewer 
*.
my goal is to find top negative and positive keyword, and for each keyword 
find the count of its opposite(if the keyword was on top positive ,i need 
to find negative count for the keyword) 

for example (based on data presented below)

top negative
   iphone - 2
   opposite count (positive) - 2
   samsung - 1
   opposite count(positive)  - 0
top positive
iphone - 2
opposite count (negative) - 2


Thank you in advance for you time.


*The schema :*
curl -XPOST "http://localhost:9200/forum_poc"; -d '
{
  "settings": {
"number_of_shards": 9,
"number_of_replicas": 1
  },
  "mappings": {
"_default_": {
  "_all": {
"enabled": false
  },
  "_source": {
"enabled": true
  },
  "dynamic": "false"
},
"ReviewEvent": {
  
  "_source": {
"enabled": true
  },
  "properties": {
"Reviews": {
  "type": "nested",
  "include_in_parent": true,
  "properties": {
"polarity": {
  "type": "string",
  "index": "not_analyzed",
  "store": "true"
},
"reviewer": {
  "type": "string",
  "index": "not_analyzed",
  "store": "true"
},
"keyword": {
  "type": "string",
  "index": "not_analyzed",
  "store": "true"
}
  }
}
  }
}
  }
}'
}


*The Data :*

curl -XPOST "http://localhost:9200/forum_poc"; -d '
{"index":{"_index":"forum_poc","_type":"ReviewEvent","_id":0}}
{"Reviews":[{"polarity":"negative","reviewer":"jhon","keyword":"iphone"},{"polarity":"negative","reviewer":"kevin","keyword":"samsung"}]}
{"index":{"_index":"forum_poc","_type":"ReviewEvent","_id":1}}
{"Reviews":[{"polarity":"positive","reviewer":"Doron","keyword":"iphone"}]}
{"index":{"_index":"forum_poc","_type":"ReviewEvent","_id":2}}
{"Reviews":[{"polarity":"negative","reviewer":"Michel","keyword":"iphone"}]}
{"index":{"_index":"forum_poc","_type":"ReviewEvent","_id":4}}
{"Reviews":[{"polarity":"positive","reviewer":"Afi","keyword":"iphone"}]}
'

*My query:*



POST forum_poc/_search?search_type=count
{
  "aggs": {
"aggregation": {
  "nested": {
"path": "Reviews"
  },
  "aggs": {
"polarity": {
  "terms": {
"field": "polarity",
"size": 10
  },
  "aggs": {
"keyword": {
  "terms": {
"field": "keyword",
"size": 10
  }
}
  }
}
  }
}
  }
}


i need the opposite count for each keyword.

{
   "took": 7,
   "timed_out": false,
   "_shards": {
  "total": 9,
  "successful": 9,
  "failed": 0
   },
   "hits": {
  "total": 4,
  "max_score": 0,
  "hits": []
   },
   "aggregations": {
  "aggregation": {
 "doc_count": 5,
 "polarity": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
   {
  "key": "negative",
  "doc_count": 3,
  "keyword": {
 "doc_count_error_upper_bound": 0,
 "sum_other_doc_count": 0,
 "buckets"

Re: Problems with aggregation query

[Eike Dehling]

Assuming the source_ip field is unanalyzed, your query could be something 
like this:

{
   'query': {
  'terms': {
  'port': [ 23 ]
  }
   },
   'size': 0,
   'aggs': {
  'counts_per_ip': {
 'terms': {
'field': 'source_ip'
 }
  } 
   }
}

On Saturday, February 14, 2015 at 2:40:17 AM UTC+1, Ian Hayes wrote:
>
> I'm having a difficult time trying to get what is a simple SQL query to 
> work with Elasticsearch. I can query, I can get counts, but can't seem to 
> get the aggregation to work.
>
> For a simple SQL query: select source_ip, count(*) from my_table where 
> destination port='23' group by source_ip;
>
> I've spent the day looking over documentation and examples, but just about 
> all of the examples I have seen don't seem to utilize a filter and 
> aggregate the entire index.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a51a023a-c4e5-4e69-b26e-427ccba748c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Problems with aggregation query

[Ian Hayes]

I'm having a difficult time trying to get what is a simple SQL query to 
work with Elasticsearch. I can query, I can get counts, but can't seem to 
get the aggregation to work.

For a simple SQL query: select source_ip, count(*) from my_table where 
destination port='23' group by source_ip;

I've spent the day looking over documentation and examples, but just about 
all of the examples I have seen don't seem to utilize a filter and 
aggregate the entire index.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d049ac60-cb1e-4d66-9e94-8b8f9093b9c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Filter aggregation vs Term aggregation with filtering values

[Anant Aneja]

I'm building aggregations on a field which has bot a '.analyzed' and a 
'.raw' version.
The analyzer used is the standard analyzer

If I want to build filtered term buckets, what would be more performant
1. Using an 'include' specified on the terms aggregation on the '.raw' 
field (
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values
)
2. Using a filter aggregation on the '.analyzed' version, then building the 
terms aggregation on the '.raw' field.

How will the performance change
1. If the field is really large
2. If the field is small (< 300 characters)

I know that the semantics of querying on a analyzed field differs from a 
regex match; but if my users would be okay either way - what is the query 
which will have better perf on the cluster ?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1597e450-c46f-4d98-9321-235768c03e2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Geohash Grid Aggregation performance

[Sávio Salvarino Teles de Oliveira]

Setting geohash = true or geohash_prefix = true improve the Geohash Grid 
Aggregation performance?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1d69475a-9341-4798-8de2-d55a632bdd46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Date range aggregation vs filters aggregation with range filter

[Maxim Valyanskiy]

Hello!

I use date range aggregation to calculate counts for several fixed periods 
(like 'today', 'this week', 'this month' & etc). This variants are shown 
via radio-buttons in web interface. Also I need to add "total" variant that 
counts total number of documents .

My current solution uses total document count value from some upper level 
source (like upper level filter aggregation or bucket of terms 
aggregation). This causes a lot of complications in my source code.

I have an idea to simplify it by replacing date range aggregation with 
filters aggregation that holds range filters for my periods and matchall 
filter to count total value. I wonder is there any performance problems 
with this solution? What is faster to execute, date rage aggregation or 
filters aggregation with range filters? 

Is there any other solutions for this problem?

Maxim

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7ebe5b84-ca86-468c-bcf0-7fe128417274%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation of count of terms (possibly...)

['Clive Lawrence' via elasticsearch]

Hi all,

This is my first post as I'm relatively new to ElasticSearch, Logstash, 
Kibana etc. and I'm really enjoying the challenge of learning it all and 
applying it!

I'm reasonably familiar with basic aggregations now, but I'm trying to 
produce a particular report from an index and I would really appreciate 
some help or advice on how to approach it.

We index log entries from application servers and I'm capturing events like 
"login" and the "user id" that caused that event to fire. I can produce a 
report of total "login" events, a total count of unique "user ids" who 
logged in (based on a cardinality aggregation), a count of the top "user 
ids" who logged in the most (user "mike" logged in 23 times today etc. 
based on a terms aggregation), but I'd like to produce a report with the 
following data:

x users logged in 20 times today (assuming that 20 was the maximum 
frequency of any particular user id appearing in the logs)
.
y users logged in 18 times today
.
.
z users logged in only 1 time today.

So a breakdown of the count of the frequency of "user id" entries (filtered 
by the "login" event). I don't need the user ids in this report, just the 
frequency breakdown if you see what I mean. I'm not sure if this would 
require a script (not used them before...) or some sub-aggregation, or 
something else?

Any ideas or assistance would be appreciated!

Many thanks,

Clive

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a22c9066-6a99-4475-be49-42f4a3710b18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch C# client (NEST): access nested aggregation results

[Jay Hilden]

I'm trying to use the C# plugin to retrieve data from a nested aggregation. 
 If anyone could help that would be most appreciated.  Here is the Stack 
Overflow question.

http://stackoverflow.com/questions/28096723/elasticsearch-c-sharp-client-nest-access-nested-aggregation-results

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/73f7e411-123b-4e83-9806-a98b04cf8eb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: stats aggregation on list length

[Jilles van Gurp]

Thanks! Now it works.

Best,

Jilles

On Friday, January 23, 2015 at 3:04:55 AM UTC+1, Masaru Hasegawa wrote:
>
> Hi, 
>
> Objects are flattened in index level. Nothing is indexed as “member” 
> that’s why you get the exception. 
> Using doc[‘members.name'] instead of doc[‘members’] in script should 
> work. 
>
>
> Masaru 
>
>
> On January 22, 2015 at 19:10:25, Jilles van Gurp (jilles...@gmail.com 
> ) wrote: 
> > I'm trying to do a stats aggregation on the list length using a script 
> but 
> > I'm getting errors. For this data, 
> >   
> > PUT test_groups/group/1 
> > { 
> > "name":"1", 
> > "members":[ 
> > { 
> > "name":"m1" 
> > } 
> > ] 
> > } 
> >   
> > PUT test_groups/group/2 
> > { 
> > "name":"2", 
> > "members":[ 
> > { 
> > "name":"m1" 
> > }, 
> > { 
> > "name":"m2" 
> > } 
> > ] 
> > } 
> >   
> > and this query: 
> >   
> > GET test_groups/group/_search 
> > { 
> > "aggs": { 
> > "group_members": { 
> > "filter": { 
> > "exists": { 
> > "field": "members" 
> > } 
> > }, 
> > "aggs": { 
> > "length": { 
> > "stats": { 
> > "script": "doc['members'].values.length" 
> > } 
> > } 
> > } 
> > } 
> > } 
> > } 
> >   
> > I get an error stating that the members field does not exist in type 
> group: 
> >   
> > { 
> > "took": 4, 
> > "timed_out": false, 
> > "_shards": { 
> > "total": 5, 
> > "successful": 3, 
> > "failed": 2, 
> > "failures": [ 
> > { 
> > "index": "test_groups", 
> > "shard": 2, 
> > "status": 500, 
> > "reason": "QueryPhaseExecutionException[[test_groups][2]: 
> > query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed 
> > [Failed to execute main query]]; nested: 
> > GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No 
>   
> > field found for [members] in mapping with types [group]]]; " 
> > }, 
> > { 
> > "index": "test_groups", 
> > "shard": 3, 
> > "status": 500, 
> > "reason": "QueryPhaseExecutionException[[test_groups][3]: 
> > query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed 
> > [Failed to execute main query]]; nested: 
> > GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No 
>   
> > field found for [members] in mapping with types [group]]]; " 
> > } 
> > ] 
> > }, 
> > "hits": { 
> > "total": 0, 
> > "max_score": null, 
> > "hits": [] 
> > }, 
> > "aggregations": { 
> > "group_members": { 
> > "doc_count": 0, 
> > "length": { 
> > "count": 0, 
> > "min": null, 
> > "max": null, 
> > "avg": null, 
> > "sum": null 
> > } 
> > } 
> > } 
> > } 
> >   
> > Is there a way to do this? 
> >   
> > Best regards, 
> >   
> > Jilles 
> >   
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch"   
> > group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com .   
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/fcbce46c-6556-4e4e-b74a-2a4cbea915c6%40googlegroups.com.
>  
>   
> > For more options, visit https://groups.google.com/d/optout. 
> >   
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b3f877a9-633e-4fe8-b4ea-3869e833782f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reverse nested aggregation within nested filter aggregation fails

[Selvinaz Karahancer-Bouraga]

Hi Masaru,

the datamodel is correct, attributes are in the root element, thas was just
a copy paste failure.
But the reason why it fails was that the nested aggregation  around “LINE”
has missing.

Now I get the correct results.

Thank you s much :)

2015-01-23 5:22 GMT+01:00 Masaru Hasegawa :

> Hi,
>
> Not sure if it solves your issue but I think there are a few things to fix:
> - “attributes" is under “source". nested aggregation’s “path” would be
> “source.attributes”. You’d need to update field names accordingly as well.
> - reverse_nested aggregation’s “path” would be empty since it’s joined
> back to root.
> - nested aggregation is needed around “LINE” aggregation since you are in
> root level.
>
>
> Masaru
>
> On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga (
> selvinaz.karahance...@gmail.com) wrote:
> > I am using Elasticsearch 1.3.4.
> >
> > Nobody has an idea why the buckets of LINES are empty?
> > Is there another possibility to resolve this problem?
> >
> > Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
> > Karahancer-Bouraga:
> > >
> > > Hello,
> > >
> > > I have to realize distinct data queries on data persisted in
> ElasticSearch.
> > > My data model looks like:
> > >
> > >
> {"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
> > > "severityLevel" : "SL_EVENT",
> > > "source" : {
> > > "sourceId" : "1",
> > > "sourceType" : "VEHICLE",
> > > "description" : null,
> > > "mandator" : {
> > > "mandatorId" : "DEF",
> > > "mandatorName" : null,
> > > "priority" : null
> > > },
> > > "eventTime" : 1410768722000,
> > > "version" : "Version_0_1",
> > > "attributes" : [ {
> > > "paramKey" : "COURSE",
> > > "value" : "123"
> > > }, {
> > > "paramKey" : "DRIVERNO",
> > > "value" : "111"
> > > }, {
> > > "paramKey" : "LINE",
> > > "value" : "101"
> > > }, {
> > > "paramKey" : "gps_x",
> > > "value" : ""
> > > }, {
> > > "paramKey" : "gps_y",
> > > "value" : "87654321"
> > > } ]
> > > }}
> > > where attributes are nested objects of EventBeans.
> > > Now I want to have all distinct values of mandatorId, LINE and gps_x.
> > > The aggregationbuilder looks like:
> > >
> > > "aggregations" : {
> > > "source.mandator.mandatorId" : {
> > > "terms" : {
> > > "field" : "source.mandator.mandatorId",
> > > "size" : 2147483647,
> > > "min_doc_count" : 1
> > > },
> > > "aggregations" : {
> > > "attributes" : {
> > > "nested" : {
> > > "path" : "attributes"
> > > },
> > > "aggregations" : {
> > > "gps_x" : {
> > > "filter" : {
> > > "term" : {
> > > "attributes.paramKey" : "gps_x"
> > > }
> > > },
> > > "aggregations" : {
> > > "gps_x" : {
> > > "terms" : {
> > > "field" : "attributes.value",
> > > "size" : 2147483647,
> > > "order" : {
> > > "_count" : "desc"
> > > }
> > > },
> > > "aggregations" : {
> > > "attributes" : {
> > > "reverse_nested" : {
> > > "path" : "attributes"
> > > },
> > > "aggregations" : {
> > > "LINE" : {
> > > "filter" : {
> > > "term" : {
> > > "attributes.paramKey" : "LINE"
> > > }
> > > },
> > > "aggregations" : {
> > > "LINE&

Re: Reverse nested aggregation within nested filter aggregation fails

[Masaru Hasegawa]

Hi,

Not sure if it solves your issue but I think there are a few things to fix:
- “attributes" is under “source". nested aggregation’s “path” would be 
“source.attributes”. You’d need to update field names accordingly as well.
- reverse_nested aggregation’s “path” would be empty since it’s joined back to 
root.
- nested aggregation is needed around “LINE” aggregation since you are in root 
level.


Masaru

On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga 
(selvinaz.karahance...@gmail.com) wrote:
> I am using Elasticsearch 1.3.4.
>  
> Nobody has an idea why the buckets of LINES are empty?
> Is there another possibility to resolve this problem?
>  
> Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
> Karahancer-Bouraga:
> >
> > Hello,
> >
> > I have to realize distinct data queries on data persisted in ElasticSearch.
> > My data model looks like:
> >
> > {"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
> >   
> > "severityLevel" : "SL_EVENT",
> > "source" : {
> > "sourceId" : "1",
> > "sourceType" : "VEHICLE",
> > "description" : null,
> > "mandator" : {
> > "mandatorId" : "DEF",
> > "mandatorName" : null,
> > "priority" : null
> > },
> > "eventTime" : 1410768722000,
> > "version" : "Version_0_1",
> > "attributes" : [ {
> > "paramKey" : "COURSE",
> > "value" : "123"
> > }, {
> > "paramKey" : "DRIVERNO",
> > "value" : "111"
> > }, {
> > "paramKey" : "LINE",
> > "value" : "101"
> > }, {
> > "paramKey" : "gps_x",
> > "value" : ""
> > }, {
> > "paramKey" : "gps_y",
> > "value" : "87654321"
> > } ]
> > }}
> > where attributes are nested objects of EventBeans.
> > Now I want to have all distinct values of mandatorId, LINE and gps_x.
> > The aggregationbuilder looks like:
> >
> > "aggregations" : {
> > "source.mandator.mandatorId" : {
> > "terms" : {
> > "field" : "source.mandator.mandatorId",
> > "size" : 2147483647,
> > "min_doc_count" : 1
> > },
> > "aggregations" : {
> > "attributes" : {
> > "nested" : {
> > "path" : "attributes"
> > },
> > "aggregations" : {
> > "gps_x" : {
> > "filter" : {
> > "term" : {
> > "attributes.paramKey" : "gps_x"
> > }
> > },
> > "aggregations" : {
> > "gps_x" : {
> > "terms" : {
> > "field" : "attributes.value",
> > "size" : 2147483647,
> > "order" : {
> > "_count" : "desc"
> > }
> > },
> > "aggregations" : {
> > "attributes" : {
> > "reverse_nested" : {
> > "path" : "attributes"
> > },
> > "aggregations" : {
> > "LINE" : {
> > "filter" : {
> > "term" : {
> > "attributes.paramKey" : "LINE"
> > }
> > },
> > "aggregations" : {
> > "LINE" : {
> > "terms" : {
> > "field" : "attributes.value",
> > "size" : 2147483647,
> > "order" : {
> > "_count" : "desc"
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> > }
> >
> > and the response looks like:
> >
> > "aggregations" : {
> > "source.mandator.mandatorId" : {
> > "doc_count_error_upper_bound" : 0,
> > "sum_other_doc_count" : 0,
> > "buckets" : [ {
> > "key" : "def",
> > "doc_count" : 3,
> > "attrib

Re: stats aggregation on list length

[Masaru Hasegawa]

Hi,

Objects are flattened in index level. Nothing is indexed as “member” that’s why 
you get the exception.
Using doc[‘members.name'] instead of doc[‘members’] in script should work.


Masaru


On January 22, 2015 at 19:10:25, Jilles van Gurp (jillesvang...@gmail.com) 
wrote:
> I'm trying to do a stats aggregation on the list length using a script but
> I'm getting errors. For this data,
>  
> PUT test_groups/group/1
> {
> "name":"1",
> "members":[
> {
> "name":"m1"
> }
> ]
> }
>  
> PUT test_groups/group/2
> {
> "name":"2",
> "members":[
> {
> "name":"m1"
> },
> {
> "name":"m2"
> }
> ]
> }
>  
> and this query:
>  
> GET test_groups/group/_search
> {
> "aggs": {
> "group_members": {
> "filter": {
> "exists": {
> "field": "members"
> }
> },
> "aggs": {
> "length": {
> "stats": {
> "script": "doc['members'].values.length"
> }
> }
> }
> }
> }
> }
>  
> I get an error stating that the members field does not exist in type group:
>  
> {
> "took": 4,
> "timed_out": false,
> "_shards": {
> "total": 5,
> "successful": 3,
> "failed": 2,
> "failures": [
> {
> "index": "test_groups",
> "shard": 2,
> "status": 500,
> "reason": "QueryPhaseExecutionException[[test_groups][2]:
> query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed
> [Failed to execute main query]]; nested:
> GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No  
> field found for [members] in mapping with types [group]]]; "
> },
> {
> "index": "test_groups",
> "shard": 3,
> "status": 500,
> "reason": "QueryPhaseExecutionException[[test_groups][3]:
> query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed
> [Failed to execute main query]]; nested:
> GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No  
> field found for [members] in mapping with types [group]]]; "
> }
> ]
> },
> "hits": {
> "total": 0,
> "max_score": null,
> "hits": []
> },
> "aggregations": {
> "group_members": {
> "doc_count": 0,
> "length": {
> "count": 0,
> "min": null,
> "max": null,
> "avg": null,
> "sum": null
> }
> }
> }
> }
>  
> Is there a way to do this?
>  
> Best regards,
>  
> Jilles
>  
> --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch"  
> group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.  
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/fcbce46c-6556-4e4e-b74a-2a4cbea915c6%40googlegroups.com.
>   
> For more options, visit https://groups.google.com/d/optout.
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.54c1ac3b.7fdcc233.1877%40citra.local.
For more options, visit https://groups.google.com/d/optout.

Performance - Very large list of buckets in an aggregation field

['Sandeep Ramesh Khanzode' via elasticsearch]

Hi,

I have to run aggregation on a very large corpus and pull out facets for 
~10-12 fields. All fields except one have decent sized buckets (like, not 
more than ~1K at a maximum), however, one field may have a very large 
number of buckets. Probably in millions. Will that turn out to be a 
performance issue?

All I am interested is in the grouping of the records based on that field.

Is there any best practice on how to achieve this, or is this not a normal 
scenario?

Thanks,
SRK

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/82c68775-f0a8-4044-bf6b-f2a975754013%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reverse nested aggregation within nested filter aggregation fails

[Selvinaz Karahancer-Bouraga]

ot; : 0,
>   "buckets" : [ {
> "key" : "def",
> "doc_count" : 3,
> "attributes" : {
>   "doc_count" : 15,
>   "gps_x" : {
> "doc_count" : 3,
> "gps_x" : {
>   "doc_count_error_upper_bound" : 0,
>   "sum_other_doc_count" : 0,
>   "buckets" : [ {
> "key" : "1111",
> "doc_count" : 1,
> "attributes" : {
>   "doc_count" : 1,
>   "LINE" : {
> "doc_count" : 0,
> "LINE" : {
>   "doc_count_error_upper_bound" : 0,
>   "sum_other_doc_count" : 0,
>   "buckets" : [ ]
> }
>   }
> }
>   }, {
> "key" : "",
> "doc_count" : 1,
> "attributes" : {
>   "doc_count" : 1,
>   "LINE" : {
> "doc_count" : 0,
> "LINE" : {
>   "doc_count_error_upper_bound" : 0,
>   "sum_other_doc_count" : 0,
>   "buckets" : [ ]
> }
>   }
> }
>   }, {
> "key" : "",
> "doc_count" : 1,
> "attributes" : {
>   "doc_count" : 1,
>   "LINE" : {
> "doc_count" : 0,
> "LINE" : {
>   "doc_count_error_upper_bound" : 0,
>   "sum_other_doc_count" : 0,
>   "buckets" : [ ]
> }
>   }
> }
>   } ]
> }
>   }
> }
>   } ]
> }
>   }
>
> The buckets of LINEs are empty, I am using reverse_nested aggregation, but 
> I think I am still in the filter of attributes.paramKey=gps_x.
> How can I solve this problem?
> Anyone an idea?
>
> Thank you for your help.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

stats aggregation on list length

[Jilles van Gurp]

I'm trying to do a stats aggregation on the list length using a script but 
I'm getting errors. For this data, 

PUT test_groups/group/1
{
  "name":"1",
  "members":[
{
  "name":"m1"
}
  ]
}

PUT test_groups/group/2
{
  "name":"2",
  "members":[
{
  "name":"m1"
},
{
  "name":"m2"
}
  ]
}

and this query:

GET test_groups/group/_search
{
  "aggs": {
"group_members": {
  "filter": {
"exists": {
  "field": "members"
}
  },
  "aggs": {
"length": {
  "stats": {
"script": "doc['members'].values.length"
  }
}
  }
}
  }
}

I get an error stating that the members field does not exist in type group:

{
   "took": 4,
   "timed_out": false,
   "_shards": {
  "total": 5,
  "successful": 3,
  "failed": 2,
  "failures": [
 {
"index": "test_groups",
"shard": 2,
"status": 500,
"reason": "QueryPhaseExecutionException[[test_groups][2]: 
query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed 
[Failed to execute main query]]; nested: 
GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No 
field found for [members] in mapping with types [group]]]; "
 },
 {
"index": "test_groups",
"shard": 3,
"status": 500,
"reason": "QueryPhaseExecutionException[[test_groups][3]: 
query[ConstantScore(cache(_type:group))],from[0],size[10]: Query Failed 
[Failed to execute main query]]; nested: 
GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No 
field found for [members] in mapping with types [group]]]; "
 }
  ]
   },
   "hits": {
  "total": 0,
  "max_score": null,
  "hits": []
   },
   "aggregations": {
  "group_members": {
 "doc_count": 0,
 "length": {
"count": 0,
"min": null,
"max": null,
"avg": null,
"sum": null
 }
  }
   }
}

Is there a way to do this?

Best regards,

Jilles

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fcbce46c-6556-4e4e-b74a-2a4cbea915c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reverse nested aggregation within nested filter aggregation fails

[selvinaz . karahancer84]

Hello,

I have to realize distinct data queries on data persisted in ElasticSearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
  "severityLevel" : "SL_EVENT",
  "source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
  "mandatorId" : "DEF",
  "mandatorName" : null,
  "priority" : null
},
  "eventTime" : 1410768722000,
  "version" : "Version_0_1",
  "attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
  }, {
"paramKey" : "DRIVERNO",
"value" : "111"
  }, {
"paramKey" : "LINE",
"value" : "101"
  }, {
"paramKey" : "gps_x",
"value" : ""
  }, {
"paramKey" : "gps_y",
"value" : "87654321"
  } ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
  "terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
  },
  "aggregations" : {
"attributes" : {
  "nested" : {
"path" : "attributes"
  },
  "aggregations" : {
"gps_x" : {
  "filter" : {
"term" : {
  "attributes.paramKey" : "gps_x"
}
  },
  "aggregations" : {
"gps_x" : {
  "terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
  "_count" : "desc"
}
  },
  "aggregations" : {
"attributes" : {
  "reverse_nested" : {
"path" : "attributes"
  },
  "aggregations" : {
"LINE" : {
  "filter" : {
"term" : {
  "attributes.paramKey" : "LINE"
}
  },
  "aggregations" : {
"LINE" : {
  "terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
  "_count" : "desc"
}
  }
}
  }
}
  }
}
  }
}
  }
}
  }
}
  }
}
  }

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
  "doc_count_error_upper_bound" : 0,
  "sum_other_doc_count" : 0,
  "buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
  "doc_count" : 15,
  "gps_x" : {
"doc_count" : 3,
"gps_x" : {
  "doc_count_error_upper_bound" : 0,
  "sum_other_doc_count" : 0,
  "buckets" : [ {
"key" : "",
"doc_count" : 1,
"attributes" : {
  "doc_count" : 1,
  "LINE" : {
"doc_count" : 0,
"LINE" : {
  "doc_co

Re: Aggregation - Blank and date aggregation

[buddarapu nagaraju]

I was able to figure out through fiddler ...date histrograms are returns in
seperate nested object in the result .. Now works

On Friday, January 16, 2015, Adrien Grand 
wrote:

> This looks good, what error did you get?
>
> On Fri, Jan 16, 2015 at 9:41 AM, buddarapu nagaraju  > wrote:
>
>> Index mapping here
>>
>> "mappings": {
>>
>>- "document": {
>>   - "properties": {
>>  - "createdDateTime": {
>> - "format": "dateOptionalTime",
>> - "type": "date"
>>  },
>>  - "doubleSort1": {
>> - "type": "double"
>>  },
>>  - "stringSort3": {
>> - "type": "string"
>>  },
>>  - "doubleSort2": {
>> - "type": "double"
>>  },
>>  - "doubleSort3": {
>> - "type": "double"
>>  },
>>  - "numSort1": {
>> - "type": "long"
>>  },
>>  - "stringSort2": {
>> - "type": "string"
>>  },
>>  - "dcn": {
>> - "type": "string"
>>  },
>>  - "numSort2": {
>> - "type": "long"
>>  },
>>  - "numSort3": {
>> - "type": "long"
>>  },
>>  - "path": {
>> - "type": "string"
>>  },
>>  - "numField": {
>> - "type": "long"
>>  },
>>  - "dateSort3": {
>> - "format": "dateOptionalTime",
>> - "type": "date"
>>  },
>>  - "dateSort2": {
>> - "format": "dateOptionalTime",
>> - "type": "date"
>>  },
>>  - "rank": {
>> - "type": "double"
>>  },
>>  - "id": {
>> - "type": "long"
>>  },
>>  - "text": {
>> - "type": "string"
>>  },
>>  - "fields": {
>> - "properties": {
>>- "isAnalyzed": {
>>   - "type": "boolean"
>>},
>>- "name": {
>>   - "type": "string"
>>},
>>- "isFullText": {
>>   - "type": "boolean"
>>},
>>- "isStored": {
>>   - "type": "boolean"
>>},
>>- "value": {
>>   - "type": "string"
>>}
>> }
>>  }
>>   }
>>}
>>
>>
>> Regards
>> Nagaraju
>> 908 517 6981
>>
>> On Fri, Jan 16, 2015 at 3:23 AM, buddarapu nagaraju <
>> budda08n...@gmail.com
>> > wrote:
>>
>>> Hi ,
>>>
>>> I tried but date histrogram didnt work not sure what is the mistake am
>>> doing
>>>
>>> here is date histrogram request(json) am passing and also pasted sample
>>> doc structure
>>>
>>>
>>>
>>>
>>>
>>> date histogram request
>>>
>>> {
>>>   "aggs": {
>>> "createddatetime": {
>>>   "date_histogram": {
>>> "field": "createddatetime",
>>> "interval": "day"
>>>   }
>>> }
>>>   }
>>> }
>>>
>>> Document in index has fields
>>>
>>>
>>>
>>>-
>>>   - "id": 79,
>>>   - "rank": 0,
>>>   - "dateSort2": "2015-01-15T06:08:06.7091884Z",
>>>   - "dateSort3": "0001-01-01T00:00:00",
>>>   - "do

Re: Aggregation - Blank and date aggregation

[Adrien Grand]

This looks good, what error did you get?

On Fri, Jan 16, 2015 at 9:41 AM, buddarapu nagaraju 
wrote:

> Index mapping here
>
> "mappings": {
>
>- "document": {
>   - "properties": {
>  - "createdDateTime": {
> - "format": "dateOptionalTime",
> - "type": "date"
>  },
>  - "doubleSort1": {
> - "type": "double"
>  },
>  - "stringSort3": {
> - "type": "string"
>  },
>  - "doubleSort2": {
> - "type": "double"
>  },
>  - "doubleSort3": {
> - "type": "double"
>  },
>  - "numSort1": {
> - "type": "long"
>  },
>  - "stringSort2": {
> - "type": "string"
>  },
>  - "dcn": {
> - "type": "string"
>  },
>  - "numSort2": {
> - "type": "long"
>  },
>  - "numSort3": {
> - "type": "long"
>  },
>  - "path": {
> - "type": "string"
>  },
>  - "numField": {
> - "type": "long"
>  },
>  - "dateSort3": {
> - "format": "dateOptionalTime",
> - "type": "date"
>  },
>  - "dateSort2": {
> - "format": "dateOptionalTime",
> - "type": "date"
>  },
>  - "rank": {
> - "type": "double"
>  },
>  - "id": {
> - "type": "long"
>  },
>  - "text": {
> - "type": "string"
>  },
>  - "fields": {
> - "properties": {
>- "isAnalyzed": {
>   - "type": "boolean"
>},
>- "name": {
>   - "type": "string"
>},
>- "isFullText": {
>   - "type": "boolean"
>},
>- "isStored": {
>   - "type": "boolean"
>},
>- "value": {
>   - "type": "string"
>}
> }
>  }
>   }
>}
>
>
> Regards
> Nagaraju
> 908 517 6981
>
> On Fri, Jan 16, 2015 at 3:23 AM, buddarapu nagaraju  > wrote:
>
>> Hi ,
>>
>> I tried but date histrogram didnt work not sure what is the mistake am
>> doing
>>
>> here is date histrogram request(json) am passing and also pasted sample
>> doc structure
>>
>>
>>
>>
>>
>> date histogram request
>>
>> {
>>   "aggs": {
>> "createddatetime": {
>>   "date_histogram": {
>> "field": "createddatetime",
>> "interval": "day"
>>   }
>> }
>>   }
>> }
>>
>> Document in index has fields
>>
>>
>>
>>-
>>   - "id": 79,
>>   - "rank": 0,
>>   - "dateSort2": "2015-01-15T06:08:06.7091884Z",
>>   - "dateSort3": "0001-01-01T00:00:00",
>>   - "doubleSort1": 118.5,
>>   - "doubleSort2": 67884.18,
>>   - "doubleSort3": 54262.6006,
>>   - "numField": 0,
>>   - "createdDateTime": "2015-01-16T06:08:06.7091884Z",
>>   -
>>
>>
>> Regards
>> Nagaraju
>> 908 517 6981
>>
>> On Thu, Jan 15, 2015 at 12:38 PM, Adrien Grand <
>> adrien.gr...@elasticsearch.com> wrote:
>>
>>> Then it means that you want to use a date_histogram aggregation with
>>> interval=day. See
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html
>

Re: Aggregation - Blank and date aggregation

[buddarapu nagaraju]

Index mapping here

"mappings": {

   - "document": {
  - "properties": {
 - "createdDateTime": {
- "format": "dateOptionalTime",
- "type": "date"
 },
 - "doubleSort1": {
- "type": "double"
 },
 - "stringSort3": {
- "type": "string"
 },
 - "doubleSort2": {
- "type": "double"
 },
 - "doubleSort3": {
- "type": "double"
 },
 - "numSort1": {
- "type": "long"
 },
 - "stringSort2": {
- "type": "string"
 },
 - "dcn": {
- "type": "string"
 },
 - "numSort2": {
- "type": "long"
 },
 - "numSort3": {
- "type": "long"
 },
 - "path": {
- "type": "string"
 },
 - "numField": {
- "type": "long"
 },
 - "dateSort3": {
- "format": "dateOptionalTime",
- "type": "date"
 },
 - "dateSort2": {
- "format": "dateOptionalTime",
- "type": "date"
 },
 - "rank": {
- "type": "double"
 },
 - "id": {
- "type": "long"
 },
 - "text": {
- "type": "string"
 },
 - "fields": {
- "properties": {
   - "isAnalyzed": {
  - "type": "boolean"
   },
   - "name": {
  - "type": "string"
   },
   - "isFullText": {
  - "type": "boolean"
   },
   - "isStored": {
  - "type": "boolean"
   },
   - "value": {
  - "type": "string"
   }
}
 }
  }
   }


Regards
Nagaraju
908 517 6981

On Fri, Jan 16, 2015 at 3:23 AM, buddarapu nagaraju 
wrote:

> Hi ,
>
> I tried but date histrogram didnt work not sure what is the mistake am
> doing
>
> here is date histrogram request(json) am passing and also pasted sample
> doc structure
>
>
>
>
>
> date histogram request
>
> {
>   "aggs": {
> "createddatetime": {
>   "date_histogram": {
> "field": "createddatetime",
> "interval": "day"
>   }
> }
>   }
> }
>
> Document in index has fields
>
>
>
>-
>   - "id": 79,
>   - "rank": 0,
>   - "dateSort2": "2015-01-15T06:08:06.7091884Z",
>   - "dateSort3": "0001-01-01T00:00:00",
>   - "doubleSort1": 118.5,
>   - "doubleSort2": 67884.18,
>   - "doubleSort3": 54262.6006,
>   - "numField": 0,
>   - "createdDateTime": "2015-01-16T06:08:06.7091884Z",
>   -
>
>
> Regards
> Nagaraju
> 908 517 6981
>
> On Thu, Jan 15, 2015 at 12:38 PM, Adrien Grand <
> adrien.gr...@elasticsearch.com> wrote:
>
>> Then it means that you want to use a date_histogram aggregation with
>> interval=day. See
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html
>>
>> On Thu, Jan 15, 2015 at 4:43 PM, buddarapu nagaraju <
>> budda08n...@gmail.com> wrote:
>>
>>> Hey Adrien ,Thank you.I have one more question on aggregating on dates .
>>>
>>> We actually stored date time in a field called "createdDateTime" but I
>>> need only aggregates on date part of date time .
>>>
>>> Any ideas ? Or sample code  can help us ?
>>>
>>> Regards
>>> Nagaraju
>>> 908 517 6981
>>>
>>> On Wed, Jan 14, 2015 at 6:10 AM, Adrien Grand <
>>> adrien.gr...@elasticsearch.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Jan 14, 2015 at 10:

Re: Aggregation - Blank and date aggregation

[buddarapu nagaraju]

Hi ,

I tried but date histrogram didnt work not sure what is the mistake am doing

here is date histrogram request(json) am passing and also pasted sample doc
structure





date histogram request

{
  "aggs": {
"createddatetime": {
  "date_histogram": {
"field": "createddatetime",
"interval": "day"
  }
}
  }
}

Document in index has fields



   -
  - "id": 79,
  - "rank": 0,
  - "dateSort2": "2015-01-15T06:08:06.7091884Z",
  - "dateSort3": "0001-01-01T00:00:00",
  - "doubleSort1": 118.5,
  - "doubleSort2": 67884.18,
  - "doubleSort3": 54262.6006,
  - "numField": 0,
  - "createdDateTime": "2015-01-16T06:08:06.7091884Z",
  -


Regards
Nagaraju
908 517 6981

On Thu, Jan 15, 2015 at 12:38 PM, Adrien Grand <
adrien.gr...@elasticsearch.com> wrote:

> Then it means that you want to use a date_histogram aggregation with
> interval=day. See
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html
>
> On Thu, Jan 15, 2015 at 4:43 PM, buddarapu nagaraju  > wrote:
>
>> Hey Adrien ,Thank you.I have one more question on aggregating on dates .
>>
>> We actually stored date time in a field called "createdDateTime" but I
>> need only aggregates on date part of date time .
>>
>> Any ideas ? Or sample code  can help us ?
>>
>> Regards
>> Nagaraju
>> 908 517 6981
>>
>> On Wed, Jan 14, 2015 at 6:10 AM, Adrien Grand <
>> adrien.gr...@elasticsearch.com> wrote:
>>
>>>
>>>
>>> On Wed, Jan 14, 2015 at 10:37 AM, buddarapu nagaraju <
>>> budda08n...@gmail.com> wrote:
>>>
>>>> Does term aggregation counts on blank field values ?
>>>>
>>>>
>>> Yes, an empty value "" counts as a term. Note that you need the field to
>>> be not analyzed for it to work (or to use an analyzer that emits empty
>>> strings). Otherwise the standard analyzer would analyzer "" as an empty
>>> list of tokens, so a field value of "" would not actually count...
>>>
>>>
>>>> Does term aggregation is enough for doing date aggregation ? Or there
>>>> any specific aggregations we have ?All I need in date aggregation is to
>>>> know different dates and its counts ?
>>>>
>>>
>>> A terms aggregation is enough, but a date_histogram aggregation is
>>> generally more useful on dates as there are lots of unique values and it's
>>> often more useful to group them based on the year, month or day.
>>>
>>> --
>>> Adrien Grand
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/i9N09n_-n38/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j74ZqbBN0zNW6-5Feu7xYTKkomzx%3DDMhx28inFVYLSu5Q%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j74ZqbBN0zNW6-5Feu7xYTKkomzx%3DDMhx28inFVYLSu5Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAFtuXXKp0JycJfNvLxPGN_5YL7P-X%3DGDzvmYJQ9NFN7Q%2BaJjQw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elasticsearch/CAFtuXXKp0JycJfNvLxPGN_5YL7P-X%3DGDzvmYJQ9NFN7Q%2BaJjQw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adrien Grand
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/e