date:20140603

Can you show your test code?

You seem to look at the wrong settings - by adjusting node number, shard
number, replica number alone, you can not find out the maximum node
performance. E.g. concurrency settings, index optimizations, query
optimizations, thread pooling, and most of all, fast disk subsystem I/O is
important.

Jörg


On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech <
marcelopaesr...@gmail.com> wrote:

> Thanks for your reply Nikolas. It helps a lot.
>
> And about the quantity of documents of each shard, or size of each shard.
> And the need of no data nodes or only master nodes. When is it necessary?
>
> Some tests I did, when I increased request's number (like 100 users at
> same moment, and redo it again and again), 5 nodes with 1 shard and 2
> replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The
> response time start to increase more than 5s (I think less than 1s,  in
> this case, would be acceptable) .
>
> This test has a lot of documents (something like 14 millions).
>
>
> Thanks. Regards.
>
> Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett
> escreveu:
>
>>
>>
>>
>> On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech 
>> wrote:
>>
>> Hi guys,
>>>
>>> I'm looking for an article or a guide for the best cluster
>>> configuration. I read a lot of articles like "change this configuration"
>>> and "you must create X shards per node" but I didn't saw nothing like
>>> ElasticSearch Official guide for creating a cluster.
>>>
>>> What I would like to know are informations like.
>>> - How to calculate how many shards will be good for the cluster.
>>> - How many shards do we need per node? And if this is variable, how do I
>>> calculate this?
>>> - How much memory do I need per node and how many nodes?
>>>
>>> I think ElasticSearch is well documentated. But it is very fragmented.
>>>
>>>
>>>
>> For some of these that is because "it depends" is the answer.  For
>> example, you'll want larger heaps for aggregations and faceting.
>>
>> There are some rules of thumb:
>> 1.  Set Elasticsearch's heap memory to 1/2 of ram but not more then
>> 30GB.  Bigger then that and the JVM can't do pointer compression and you
>> effectively lose ram.
>> 2.  #1 implies that having much more then 60GB of ram on each node
>> doesn't make a big difference.  It helps but its not really as good as
>> having more nodes.
>> 3.  The most efficient efficient way of sharding is likely one shard on
>> each node.  So if you have 9 nodes and a replication factor of 2 (so 3
>> total copies) then 3 shards is likely to be more efficient then having 2 or
>> 4.  But this only really matters when those shards get lots of traffic.
>> And it breaks down a bit when you get lots of nodes.  And the in presence
>> of routing.  Its complicated.
>>
>> But these are really just starting points, safe-ish defaults.
>>
>> Nik
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBBwbpCnXPR%3D9r-yTngRbAUcG_wiqMTN8Hk6XeggPHLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: iptablex trojan experiences?

The script.disable_dynamic is an important one for anyone running <1.2.0.

You can also look at setting http.enabled for all your nodes, then use a
front end client with authentication.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 4 June 2014 08:49, Hassan Schroeder  wrote:

> On Tue, Jun 3, 2014 at 3:33 PM, 'Adolfo Rodriguez' via elasticsearch
>  wrote:
> > i was using release elasticsearch-0.90.5 in my exploited server, so maybe
> > this is already fixed in current release by disabling
> script.disable_dynamic
> > by default
>
> I got caught by this a week ago using 1.1.0 on Ubuntu 12.04. Had
> not even thought about a high port like 9200 being open by default.
> (And no, there's no Tomcat or Struts app on that box.)
>
> Luckily NewRelic tipped me off right away and I was able to put it
> into rescue mode while I provisioned a new server.
>
> One more item for the checklist :-)
>
> --
> Hassan Schroeder  hassan.schroe...@gmail.com
> http://about.me/hassanschroeder
> twitter: @hassan
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CACmC4yC%3D24X-0OBT3weju9s_9v--RJ4yLBahPn6dSuKwBho2ig%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624a75uoa4PXU6WW0_RHDBozFUE9-xO8wNCDsqN4w5%2BZuRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: iptablex trojan experiences?

2014-06-03 Thread Hassan Schroeder

On Tue, Jun 3, 2014 at 3:33 PM, 'Adolfo Rodriguez' via elasticsearch
 wrote:
> i was using release elasticsearch-0.90.5 in my exploited server, so maybe
> this is already fixed in current release by disabling script.disable_dynamic
> by default

I got caught by this a week ago using 1.1.0 on Ubuntu 12.04. Had
not even thought about a high port like 9200 being open by default.
(And no, there's no Tomcat or Struts app on that box.)

Luckily NewRelic tipped me off right away and I was able to put it
into rescue mode while I provisioned a new server.

One more item for the checklist :-)

-- 
Hassan Schroeder  hassan.schroe...@gmail.com
http://about.me/hassanschroeder
twitter: @hassan

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CACmC4yC%3D24X-0OBT3weju9s_9v--RJ4yLBahPn6dSuKwBho2ig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch 1.2.1 released

http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/

This includes a fix for a severe routing bug and is a recommended update
for anyone on 1.2.0. 1.2.0 has also been removed from the repos due to the
severity of issue.

More at the link.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YzGjYFfRj5NC6d5d0Jyv0vWA408m1J_R0T%2B3EhU52rmw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: iptablex trojan experiences?

2014-06-03 Thread 'Adolfo Rodriguez' via elasticsearch

i was using release *elasticsearch-0.90.5* in my exploited server, so maybe 
this is already fixed in current release by disabling script.disable_dynamic 
by default

https://github.com/elasticsearch/elasticsearch/issues/5853

(besides not exposing port 9200 outside)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3772e3b3-9b82-4018-8468-392ee2f1c4b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: iptablex trojan experiences?

There has been a few comments in IRC about similar things happening, all
due to ports 9200 and/or 9300 being open to the internet.

However, as you mentioned, you really shouldn't have ES directly accessible
to the outside world

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 4 June 2014 05:38, 'Adolfo Rodriguez' via elasticsearch <
elasticsearch@googlegroups.com> wrote:

> Hi, I had a couple of exploits in the last 2 weeks in my CentOS 5.7 with a
> trojan iptablex. Apparently it does a DDoS and, after, opens connections
> somewhere else. There are reported cases of connections open to someone at
> China Telecom.
>
> If you look processes in your server, you will find something as:
>
> root 4252 632 0 18:44 ? 00:00:00 /boot/.IptabLex
> root 4260 624 0 18:45 ? 00:00:00 /boot/.IptabLes
>
> This is the second time happening to me and in both cases root is
> compromised so it requires a full server reinstall. In the first case, I
> though the problem could come from Tomcat 7 which is having quite a few
> vulnerabilities last months (http://tomcat.apache.org/security-7.html) so
> I upgraded to Tomcat 8.0.8, latest release.
>
> However, problem reproduced again after fully reinstalling the server. In
> this second time I have found that ports 9200 and 9300 are open in my VPS
> by my hosting provider and I found some other cases of iptablex trojan
> attacking machines though Elastic Search ports. I know, they should not be
> open.
>
> You can find an increasingly number of reported cases on internet pointing
> to ES (and also Tomcat/struts)
>
> http://nerdanswer.com/answer.php?q=524925
>
> http://security.stackexchange.com/questions/58862/logging-server-compromised-iptables-and-iptablex
>
> So, has any other user in this group experienced the same?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f96fa6c7-a722-4bc3-9a4e-84385ceb11ac%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624aU%3DGZ6fH3fUVuD4eo5g%2BsFVFuCUTKeWhP4AYRA8Pd%3D0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Best cluster environment for search

2014-06-03 Thread Marcelo Paes Rech

Thanks for your reply Nikolas. It helps a lot.

And about the quantity of documents of each shard, or size of each shard. 
And the need of no data nodes or only master nodes. When is it necessary?

Some tests I did, when I increased request's number (like 100 users at same 
moment, and redo it again and again), 5 nodes with 1 shard and 2 replicas 
each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The response 
time start to increase more than 5s (I think less than 1s,  in this case, 
would be acceptable) .

This test has a lot of documents (something like 14 millions).


Thanks. Regards.

Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett 
escreveu:
>
>
>
>
> On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech  > wrote:
>
> Hi guys,
>>
>> I'm looking for an article or a guide for the best cluster configuration. 
>> I read a lot of articles like "change this configuration" and "you must 
>> create X shards per node" but I didn't saw nothing like ElasticSearch 
>> Official guide for creating a cluster.
>>
>> What I would like to know are informations like. 
>> - How to calculate how many shards will be good for the cluster.
>> - How many shards do we need per node? And if this is variable, how do I 
>> calculate this?
>> - How much memory do I need per node and how many nodes?
>>
>> I think ElasticSearch is well documentated. But it is very fragmented.
>>
>>
>>
> For some of these that is because "it depends" is the answer.  For 
> example, you'll want larger heaps for aggregations and faceting.
>
> There are some rules of thumb:
> 1.  Set Elasticsearch's heap memory to 1/2 of ram but not more then 30GB.  
> Bigger then that and the JVM can't do pointer compression and you 
> effectively lose ram.
> 2.  #1 implies that having much more then 60GB of ram on each node doesn't 
> make a big difference.  It helps but its not really as good as having more 
> nodes.
> 3.  The most efficient efficient way of sharding is likely one shard on 
> each node.  So if you have 9 nodes and a replication factor of 2 (so 3 
> total copies) then 3 shards is likely to be more efficient then having 2 or 
> 4.  But this only really matters when those shards get lots of traffic.  
> And it breaks down a bit when you get lots of nodes.  And the in presence 
> of routing.  Its complicated.
>
> But these are really just starting points, safe-ish defaults.
>
> Nik
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Simple Action Plugin

Not sure if I understand your concern completely - as long as you're doing
things right in your code, it should be possible to allocate resources only
when required - this holds also for plugins.

Jörg


On Tue, Jun 3, 2014 at 11:48 PM, virgil  wrote:

> Thank you Jörg. I see the point. But if the plugin consumes memory (ex.
> hold
> a HashMap for customized score), installing it on all nodes may waste
> memory
> of the cluster. Is there any way to deal with this issue?
>
>
>
> --
> View this message in context:
> http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057003.html
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1401832124609-4057003.post%40n3.nabble.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFpp8ybSWqxya9s-hK6qhhBUOTno8pdgX-Ba6YT_oAHFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Controlling IDF calculation to use num docs of matched type rather than all docs in Index

2014-06-03 Thread Riyaz

Facing the same issue. Is there a way to make elasticsearch compute idf 
with maxDoc =  
instead of maxDoc = 

On Monday, January 6, 2014 3:12:20 PM UTC-5, Michael Lawler wrote:
>
> Hi Adrien,
>
> Thanks for your comment. 
>
> Yes, I am already using different fields names across parents/child where 
> I want to isolate IDF from each other.
>
> However my understanding of my particular issue here is that its not about 
> the isolation provided by the field names, its about the maxDocs number 
> that is a divisor in the maths calculating the IDF. The maxDocs number 
> being used is the total number of documents in the index. I want it to use 
> the total number of documents of a particular document type.
>
> Michael
>
> On Monday, January 6, 2014 10:01:03 PM UTC+11, Adrien Grand wrote:
>>
>> Hi Michael,
>>
>> Unfortunately this is not possible. Maybe you could use different field 
>> names for parents and children to work around this issue?
>>
>>
>> On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler  
>> wrote:
>>
>>> Hi,
>>>
>>> Subject line says it all. Is there an easy way to change the IDF 
>>> calculation so that it uses the number of documents of the matched type (or 
>>> of a predetermined type specified at query time) rather than the total 
>>> number of all documents in the index.
>>>
>>> I am using documents with parent/child relationships, and as we scale 
>>> up, we could have orders of magnitude more children than the parents which 
>>> are important for IDF scoring.
>>>
>>> There are other reasons why we can't use separate indexes to control 
>>> this.
>>>
>>> regards,
>>> Michael
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> -- 
>> Adrien Grand
>>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/94d13530-766e-4062-a3cc-7635e2238a29%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Controlling IDF calculation to use num docs of matched type rather than all docs in Index

2014-06-03 Thread Riyaz

Facing the same issue. Is there a way to make elasticsearch compute idf 
with maxDoc =  
instead of maxDoc = 

On Monday, January 6, 2014 3:12:20 PM UTC-5, Michael Lawler wrote:
>
> Hi Adrien,
>
> Thanks for your comment. 
>
> Yes, I am already using different fields names across parents/child where 
> I want to isolate IDF from each other.
>
> However my understanding of my particular issue here is that its not about 
> the isolation provided by the field names, its about the maxDocs number 
> that is a divisor in the maths calculating the IDF. The maxDocs number 
> being used is the total number of documents in the index. I want it to use 
> the total number of documents of a particular document type.
>
> Michael
>
> On Monday, January 6, 2014 10:01:03 PM UTC+11, Adrien Grand wrote:
>>
>> Hi Michael,
>>
>> Unfortunately this is not possible. Maybe you could use different field 
>> names for parents and children to work around this issue?
>>
>>
>> On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler  
>> wrote:
>>
>>> Hi,
>>>
>>> Subject line says it all. Is there an easy way to change the IDF 
>>> calculation so that it uses the number of documents of the matched type (or 
>>> of a predetermined type specified at query time) rather than the total 
>>> number of all documents in the index.
>>>
>>> I am using documents with parent/child relationships, and as we scale 
>>> up, we could have orders of magnitude more children than the parents which 
>>> are important for IDF scoring.
>>>
>>> There are other reasons why we can't use separate indexes to control 
>>> this.
>>>
>>> regards,
>>> Michael
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> -- 
>> Adrien Grand
>>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/331c711d-97f6-4199-b6d6-1ad7869e87db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-03 Thread virgil

Thank you Jörg. I see the point. But if the plugin consumes memory (ex. hold
a HashMap for customized score), installing it on all nodes may waste memory
of the cluster. Is there any way to deal with this issue?



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057003.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1401832124609-4057003.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it possible to get a bucketed aggregation based on the count of values for a field?

2014-06-03 Thread Michael Nilsson

Thanks.  User is just 1 field in my docs, which actually represent all
requests made to my system.  What I would really like is to just get a
count of the "heavy", "medium", and "light" users, where heavy would be
users that have made say > 10 requests.  I guess what I would need would be
something like a terms_range agg.


On Fri, May 30, 2014 at 2:09 PM, 'Binh Ly' via elasticsearch <
elasticsearch@googlegroups.com> wrote:

> I don't believe this is possible at the moment. If you can pre-process
> your data and produce this summarization indexed into ES:
>
> Sara: 3
> Mike: 2
> John: 1
>
> Then you can use the range (or filter) agg as you already mentioned.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/CtDhs0HDK2Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a6741478-ff3d-4b47-a39a-5cb7332759bd%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEhMYmMv5VxLFB0676QX4vpQW%3D%3DYFYk4Lzi2M2yrGMzCzet0Rg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Curl commands to set Marvel settings

2014-06-03 Thread Boaz Leskes

At the moment the `enabled` option is only read on restart. What I suggest
you do is add that setting to all your elasticsearch.yml files and also run
the following command

curl -XPUT "http://localhost:9200/_cluster/settings"; -d'
{
  "transient": {
"marvel.agent.interval": "-1"
  }
}'

This will make the agent not sample data (and thus effectively be disabled)
until you restart the node - at which point the enabled: false setting will
be picked up


On Tue, Jun 3, 2014 at 11:01 PM, Ramdev Wudali  wrote:

> Hi Boaz:
> Couldn't I do the same curl command to disable the agent ? or is that
> setting something  that inly recognized on a ES  (re)start ?
>
> Thanks
>
> Ramdev
>
>
> On Tue, Jun 3, 2014 at 3:55 PM, Boaz Leskes  wrote:
>
>> Hi Ramdev,
>>
>> Do you want marvel to stop sending data? The best settings is
>> `marvel.agent.enabled: false` but that can only be set in the
>> elasticsearch.yml and requires a node restart. As an alternative you can
>> this (assuming you have a decently recent marvel version):
>>
>> curl -XPUT "http://localhost:9200/_cluster/settings"; -d'
>> {
>>   "persistent": {
>> "marvel.agent.interval": "-1"
>>   }
>> }'
>>
>> This will not disable the agent but stop it from sampling and shipping
>> data.
>>
>> Cheers,
>> Boaz
>>
>> On Tuesday, June 3, 2014 10:24:15 PM UTC+2, Ramdev Wudali wrote:
>>>
>>> Hi All:
>>>I am trying to prevent Marvel from creating the indices for each day
>>> automatically. I know there is a curl command to do that. I am not able to
>>> find it in any of the documentation that is available.  Other than setting
>>> it in the config file and have to restart my instance is there a way to do
>>> it ?
>>>
>>> (and can various ES settings be changed the same way ?)
>>>
>>> Thanks
>>>
>>> Ramdev
>>>
>>>
>>>
>>>
>>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/Celm5RDoC-0/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/857ee53a-7d03-4213-a1a2-364c6af78886%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/Celm5RDoC-0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGbqZ7goqRhjQvZ4giw9XF54%2B9tdh2cacFhy9vVkxG2Kk-j9iw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKzwz0q9bMVQ-XKmER%3DKSTA4cRX9tT4KNOgvvDzDgAWR-uhJcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Curl commands to set Marvel settings

2014-06-03 Thread Ramdev Wudali

Hi Boaz:
Couldn't I do the same curl command to disable the agent ? or is that
setting something  that inly recognized on a ES  (re)start ?

Thanks

Ramdev


On Tue, Jun 3, 2014 at 3:55 PM, Boaz Leskes  wrote:

> Hi Ramdev,
>
> Do you want marvel to stop sending data? The best settings is
> `marvel.agent.enabled: false` but that can only be set in the
> elasticsearch.yml and requires a node restart. As an alternative you can
> this (assuming you have a decently recent marvel version):
>
> curl -XPUT "http://localhost:9200/_cluster/settings"; -d'
> {
>   "persistent": {
> "marvel.agent.interval": "-1"
>   }
> }'
>
> This will not disable the agent but stop it from sampling and shipping
> data.
>
> Cheers,
> Boaz
>
> On Tuesday, June 3, 2014 10:24:15 PM UTC+2, Ramdev Wudali wrote:
>>
>> Hi All:
>>I am trying to prevent Marvel from creating the indices for each day
>> automatically. I know there is a curl command to do that. I am not able to
>> find it in any of the documentation that is available.  Other than setting
>> it in the config file and have to restart my instance is there a way to do
>> it ?
>>
>> (and can various ES settings be changed the same way ?)
>>
>> Thanks
>>
>> Ramdev
>>
>>
>>
>>
>>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/Celm5RDoC-0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/857ee53a-7d03-4213-a1a2-364c6af78886%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGbqZ7goqRhjQvZ4giw9XF54%2B9tdh2cacFhy9vVkxG2Kk-j9iw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Curl commands to set Marvel settings

2014-06-03 Thread Boaz Leskes

Hi Ramdev,

Do you want marvel to stop sending data? The best settings is 
`marvel.agent.enabled: false` but that can only be set in the 
elasticsearch.yml and requires a node restart. As an alternative you can 
this (assuming you have a decently recent marvel version):

curl -XPUT "http://localhost:9200/_cluster/settings"; -d'
{
  "persistent": {
"marvel.agent.interval": "-1"
  }
}'

This will not disable the agent but stop it from sampling and shipping data.

Cheers,
Boaz

On Tuesday, June 3, 2014 10:24:15 PM UTC+2, Ramdev Wudali wrote:
>
> Hi All:
>I am trying to prevent Marvel from creating the indices for each day 
> automatically. I know there is a curl command to do that. I am not able to 
> find it in any of the documentation that is available.  Other than setting 
> it in the config file and have to restart my instance is there a way to do 
> it ?
>
> (and can various ES settings be changed the same way ?)
>
> Thanks
>
> Ramdev
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/857ee53a-7d03-4213-a1a2-364c6af78886%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation vs Search/Filter discrepancy - caching issue?

2014-06-03 Thread Adrien Grand

Can you share your test case?


On Tue, Jun 3, 2014 at 1:00 PM, mooky  wrote:

> Update elastic to 1.2 - still seeing the same issue...
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2805708d-57dd-4977-a17c-2c27d9ee98d0%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7T%2B%3DgXO%2BGHGFw11RBjpTOkXCUJ%2Bz2yVauU_YOo6Pmwxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation bug? Or user error?

2014-06-03 Thread Adrien Grand

A recreation would be really great! If you can zip it and upload it to any
file sharing service, that would work for me.


On Tue, Jun 3, 2014 at 6:41 PM, mooky  wrote:

>
> By the way this test fails with elastic 1.2 also.
>
> How do I go about uploading an index with aggregation request json, etc?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j68foACRpstuJ-nGTgANgR%3Dqk%2Bh%3DTaSw2mRgRQiauY9%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Curl commands to set Marvel settings

2014-06-03 Thread Ramdev Wudali

Hi All:
   I am trying to prevent Marvel from creating the indices for each day 
automatically. I know there is a curl command to do that. I am not able to 
find it in any of the documentation that is available.  Other than setting 
it in the config file and have to restart my instance is there a way to do 
it ?

(and can various ES settings be changed the same way ?)

Thanks

Ramdev




-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6383efbe-cafe-48cb-8ee9-03b8f1dd1b0b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: All primary shards are in same node. Why? Version 1.1.1

2014-06-03 Thread Santiago Ferrer Deheza

Jörg, Thanks for the quick answer!

On Tuesday, June 3, 2014 4:53:16 PM UTC-3, Jörg Prante wrote:
>
> Primary shards are addressed first when writing, but it is a myth they do 
> all the writing. Secondary shards do the writing too, but only some milli 
> seconds later. There is nothing to worry about.
>
> Jörg
>
>
> On Tue, Jun 3, 2014 at 9:49 PM, Santiago Ferrer Deheza <
> sa.ferre...@gmail.com > wrote:
>
>> Hi everyone!
>>
>> Im having a doubt regards my elastic cluster. Here is a picture of the 
>> head plugin
>>
>>
>> 
>> Here you can see that all *primary shards are in the same node* and not 
>> distributed. I shutdown the master node to see if the primary shards were 
>> going to be distributed in both nodes left but this is what happened.
>>  
>>
>>
>> 
>>
>> *All primary shards were allocated in a single node*. Is this behaviour 
>> right? I would prefer them to be distributed equally among all nodes so the 
>> writing operations are not handled by a single one. Know how to do that?
>>
>>
>> This is my elasticsearch.yml
>>
>> *# ElasticSearch Configuration Example 
>> #*
>>
>> *# Cluster name identifies your cluster for auto-discovery. If you're 
>> running*
>> *# multiple clusters on the same network, make sure you're using unique 
>> names.*
>> *#*
>> *cluster.name : mclics*
>>
>> *action.auto_create_index: false*
>>
>> *index.number_of_shards: 16*
>>
>> *index.number_of_replicas: 2*
>>
>> *action.disable_delete_all_indices: true*
>>
>> *path.data: /mnt/elasticsearch*
>>
>> *path.logs: /usr/local/log*
>>
>> *bootstrap.mlockall: true*
>>
>> *gateway.recover_after_nodes: 2*
>>
>> *gateway.expected_nodes: 3*
>>
>> *discovery.zen.minimum_master_nodes: 1*
>>
>> *discovery.zen.ping.unicast.hosts: [##hosts##]*
>>
>>
>> Thanks,
>> Santiago!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/93af689a-e14d-4047-95be-7f32ed730a1e%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/29833671-7b72-423c-9b21-1a42a0e72e97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: All primary shards are in same node. Why? Version 1.1.1

Primary shards are addressed first when writing, but it is a myth they do
all the writing. Secondary shards do the writing too, but only some milli
seconds later. There is nothing to worry about.

Jörg


On Tue, Jun 3, 2014 at 9:49 PM, Santiago Ferrer Deheza <
sa.ferrer.deh...@gmail.com> wrote:

> Hi everyone!
>
> Im having a doubt regards my elastic cluster. Here is a picture of the
> head plugin
>
>
> 
> Here you can see that all *primary shards are in the same node* and not
> distributed. I shutdown the master node to see if the primary shards were
> going to be distributed in both nodes left but this is what happened.
>
>
>
> 
>
> *All primary shards were allocated in a single node*. Is this behaviour
> right? I would prefer them to be distributed equally among all nodes so the
> writing operations are not handled by a single one. Know how to do that?
>
>
> This is my elasticsearch.yml
>
> *# ElasticSearch Configuration Example
> #*
>
> *# Cluster name identifies your cluster for auto-discovery. If you're
> running*
> *# multiple clusters on the same network, make sure you're using unique
> names.*
> *#*
> *cluster.name : mclics*
>
> *action.auto_create_index: false*
>
> *index.number_of_shards: 16*
>
> *index.number_of_replicas: 2*
>
> *action.disable_delete_all_indices: true*
>
> *path.data: /mnt/elasticsearch*
>
> *path.logs: /usr/local/log*
>
> *bootstrap.mlockall: true*
>
> *gateway.recover_after_nodes: 2*
>
> *gateway.expected_nodes: 3*
>
> *discovery.zen.minimum_master_nodes: 1*
>
> *discovery.zen.ping.unicast.hosts: [##hosts##]*
>
>
> Thanks,
> Santiago!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/93af689a-e14d-4047-95be-7f32ed730a1e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGAWMbxKMHQnVOwtq-1jnSu9p2D_rUD8LhydeyJuQjLXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

All primary shards are in same node. Why? Version 1.1.1

2014-06-03 Thread Santiago Ferrer Deheza

Hi everyone!

Im having a doubt regards my elastic cluster. Here is a picture of the head
plugin

Here you can see that all *primary shards are in the same node* and not
distributed. I shutdown the master node to see if the primary shards were
going to be distributed in both nodes left but this is what happened.

*All primary shards were allocated in a single node*. Is this behaviour
right? I would prefer them to be distributed equally among all nodes so the
writing operations are not handled by a single one. Know how to do that?

This is my elasticsearch.yml

*# ElasticSearch Configuration Example
#*

*# Cluster name identifies your cluster for auto-discovery. If you're
running*
*# multiple clusters on the same network, make sure you're using unique
names.*
*#*
*cluster.name: mclics*

*action.auto_create_index: false*

*index.number_of_shards: 16*

*index.number_of_replicas: 2*

*action.disable_delete_all_indices: true*

*path.data: /mnt/elasticsearch*

*path.logs: /usr/local/log*

*bootstrap.mlockall: true*

*gateway.recover_after_nodes: 2*

*gateway.expected_nodes: 3*

*discovery.zen.minimum_master_nodes: 1*

*discovery.zen.ping.unicast.hosts: [##hosts##]*

Thanks,
Santiago!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/93af689a-e14d-4047-95be-7f32ed730a1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What's using memory in ElasticSearch? (Details to follow...)

What ES version is this?

Your segment count is very high (>1000) which is not efficient.

Maybe index.codec.bloom.load: false can help reducing heap mem usage.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html

Jörg

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGLMa1zV-8JM5zejxsVoTHmW93Tbt_Vs9hjD5t0NcOd4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

iptablex trojan experiences?

2014-06-03 Thread 'Adolfo Rodriguez' via elasticsearch

Hi, I had a couple of exploits in the last 2 weeks in my CentOS 5.7 with a
trojan iptablex. Apparently it does a DDoS and, after, opens connections
somewhere else. There are reported cases of connections open to someone at
China Telecom.

If you look processes in your server, you will find something as:

root 4252 632 0 18:44 ? 00:00:00 /boot/.IptabLex
root 4260 624 0 18:45 ? 00:00:00 /boot/.IptabLes

This is the second time happening to me and in both cases root is
compromised so it requires a full server reinstall. In the first case, I
though the problem could come from Tomcat 7 which is having quite a few
vulnerabilities last months (http://tomcat.apache.org/security-7.html) so I
upgraded to Tomcat 8.0.8, latest release.

However, problem reproduced again after fully reinstalling the server. In
this second time I have found that ports 9200 and 9300 are open in my VPS
by my hosting provider and I found some other cases of iptablex trojan
attacking machines though Elastic Search ports. I know, they should not be
open.

You can find an increasingly number of reported cases on internet pointing
to ES (and also Tomcat/struts)

http://nerdanswer.com/answer.php?q=524925
http://security.stackexchange.com/questions/58862/logging-server-compromised-iptables-and-iptablex

So, has any other user in this group experienced the same?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f96fa6c7-a722-4bc3-9a4e-84385ceb11ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Simple Action Plugin

Usually, plugins that extend internal ES functionality should be installed
on all nodes. This is easy to remember and preferable from an
administrative view. All the nodes in the ES cluster must have access to
plugin code under all circumstances, especially when executing actions,
mappers, routers, discovery helpers, analyzer code, indexing helpers...

In this case, for the simple action demo plugin, you can install it just on
the node of the cluster where you want to execute the demo "match_all"
search from. The "match_all" search is then searching on all the indexes of
the cluster.

If you want to execute the demo plugin "match_all" search from other nodes,
you would have to install the plugin on those other nodes, too.

Jörg

On Tue, Jun 3, 2014 at 7:33 PM, virgil  wrote:

> Thank you. About the plugin, I wonder if you install it on all nodes in
> cluster to make it work or install it in a nondata node?
>
>
>
> --
> View this message in context:
> http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4056981.html
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1401816815837-4056981.post%40n3.nabble.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkQ4xL5xqrXfH%2BzwSJ0WvyLcNw6S%2B7Z0pTSCuwXHnXWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch Node.Client Options

2014-06-03 Thread VB

We have client, data and master node configurations.

And we are seeing lots of OutOfMemory errors on node.client nodes.

One of the thing which we have some doubt is, node.client caches data for 
some reason? I am hoping not.

If yes then what are the ways we can control this caching so client does 
not cache any data.

And in general what are setting options which we can control through yml 
for client nodes (node.client).

And what all operations happens on Node.Client when search with some query.

Regards,
VB.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-03 Thread virgil

Thank you. About the plugin, I wonder if you install it on all nodes in
cluster to make it work or install it in a nondata node?



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4056981.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1401816815837-4056981.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana "bettermap" widget

2014-06-03 Thread Steven Pisarski

Thanks so much for the tip! You were correct as did format my location 
incorrectly. 

On Monday, June 2, 2014 12:51:28 PM UTC-6, Binh Ly wrote:
>
> Steve, if you can show a copy of your mapping here, I can take a quick 
> look.
>
> curl localhost:9200/your_index/_mapping?pretty
>
> It should work fine, the only time it doesn't work for me is when I messed 
> up in the mapping.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54b77c66-8d61-4c60-8970-caed98d767c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting up alerts

2014-06-03 Thread Ivan Brusic

With logstash, you can use either the email or pagerduty outputs. You can
setup a conditional for your specific log entries.

http://logstash.net/docs/1.4.1/outputs/email
http://logstash.net/docs/1.4.1/outputs/pagerduty

-- 
Ivan


On Sat, May 31, 2014 at 10:11 AM, Joshua Hall 
wrote:

> Vinneth thanks for your response.
>
> I guess I don't understand how the percolater works.  I get that you store
> a query, In my case I think I would want to store a query that looks at the
> fields I want for a specific value over the last timeperiod.
>
> Does the percolater actually attach to an index and run against all new
> documents added to the index?
>
> When I look at the documentation and see this I get very confused
>
> curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{
> "doc" : {
> "message" : "A new bonsai tree in the office"
> }
> }'
>
>
> To me this looks like asking the percolator if this message matches a
> percolator, but what I want to know is the count of the number of documents
> added to an index in the last 15 minutes matches a percolator.
>
> I would have expected to see something more similiar to
>
> curl -XGET 'localhost:9200/index/type/_percolate/count'
>
>
>
> Hopefully this makes sense as I am very confused about how this works,
> looks to me like it is just a simple does this document match any of the
> predefined queries and what I think I am after is does any documents in
> this index match this predefined query.
>
>
>
>
>
>
> On Saturday, May 31, 2014 2:18:33 AM UTC-4, vineeth mohan wrote:
>
>> Hello Joshua ,
>>
>> Percolater is the usual choice here.
>> You can register queries against poercolator and when you index some feed
>> , the peorcolater informs you that such a search query matches against your
>> index.
>> This information is passed when indexing.
>>
>> With logstash , I am not sure how this can be done.
>>
>> Thanks
>>  Vineeth
>>
>>
>> On Sat, May 31, 2014 at 8:31 AM, Joshua Hall 
>> wrote:
>>
>>> I am working on a demo using Elasticsearch, Logstash, Kibana and one of
>>> the key features that I am looking for is the ability to setup alerts to
>>> send out emails.
>>>
>>> Specifically I want to setup an alert to be emailed when a log is
>>> recorded with "severity=error" and "category=category1" occurs more than 20
>>> times in 15 minutes.
>>>
>>> I am limited to a windows only solution so that may limit some things.
>>>
>>> I have considered the following approaches
>>>
>>>1. An app/service that simply does a search to get the number of
>>>results every minute or so
>>>2. Doing something with logstash
>>>3. Doing something with Percolator (Logstash or an app)
>>>
>>> I really don't understand the percolator so I am not sure if it is even
>>> a viable approach for what doing this.
>>>
>>> Is there any guide / general guidance for doing something similar to
>>> this?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/98e8ec88-7ef9-45e6-875b-e72f72a4731a%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDCoV_vpb9uT2F_7Xzg%2ByMfgHx6qpkU6vqC6%2BAWcbL-ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: recovery took long

2014-06-03 Thread Ivan Brusic

Which version of 0.90 were you using? Starting with 0.90.1, recovery is
throttled to 20 mb per sec, which is very low IMHO. Is this setting set in
your new cluster?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html

-- 
Ivan


On Tue, Jun 3, 2014 at 6:00 AM, jaguarx  wrote:

> Hi,
> We just upgrade ES from 0.90 to 1.0.1, and met some strange issue. The
> cluster status is green, but /_cat/recovery shows one ongoing task on
> index1. When trying to flush index1, we got error saying "...recoverying,
> flush not allowed...". And the recovery took almost one day, but still
> didn't complete.
> Is this a known issue for 1.0.1? If it's, which version contains the
> bug fix?
>
> Thanks!
> Jaguar
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/538DC6D5.3090003%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAYq3qEwscin5MuF2vd550XPNsS6CCmc%2Bw_afJ431WV9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-03 Thread Luiz Guilherme Pais dos Santos

Hi Jörg,

Thanks a lot! Some month ago I started to write one plugin and it was
really difficult. This skeleton is perfect for a cold start :)


On Tue, Jun 3, 2014 at 12:15 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Hi,
>
> many of us want to start writing extensions for Elasticsearch.
>
> Except submitting pull requests to the core code, one great advantage of
> Elasticsearch is the plugin mechanism. Here, custom code can be hooked into
> Elasticsearch, without having to ask for inclusion into the core code.
> Nevertheless, plugin code can be published on Github and easily included
> into a running ES instance by using the ES plugin command line tool.
>
> Unfortunately, writing plugins is not so easy as it seems. There are many
> plugins, some of them are very advanced, and finding a starting point for a
> personal project could be quite hard.
>
> Hence, for educational purposes, I wrote a tiny plugin, as a starting
> point, to demonstrate how a plugin works.
>
> The simple plugin is indeed very simple. It makes reuse of the standard
> search action:
>
> - it defines a built-in query (a "match all" query)
>
> - it creates a custom action for it
>
> - the action is called from Java API
>
> - the result of the action (the search response of the "match all" query)
> is logged
>
> The plugin code comes with a junit test. It is available at
>
> https://github.com/jprante/elasticsearch-simple-action-plugin
>
> In the hope it is useful,
>
> Jörg
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Luiz Guilherme P. Santos

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAMdL%3DZEcM%3DzRmLk9%2BK7fiZ1hoT1zbcy_ScYT0F_8Kx0RzPkm%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migration from Solr to ElasticSearch

If you can iterate over the Solr index doc ids and fetch the source docs
from a secondary storage, you should consider doing this first. This is the
most straightforward method for reindexing.

Otherwise, if you can not access the filesystem storage for the docs (for
whatever reason), the idea would be to create a more complex tool, maybe
with help of https://github.com/DmitryKey/luke/ The Luke code should be
useful for document reconstruction, but there is no code I am aware of for
reindexing results into Elasticsearch. Such a reconstructor should take
also the Solr schema as input. But as said, such a tool heavily depends on
the Solr analyzers, so it must be evaluated first if the Solr index is
usable at all for reindexing.

Jörg


On Tue, Jun 3, 2014 at 7:24 PM, Diego Marchi 
wrote:

> Thank you Jorg,
>
> I'll start from the second question: Thanks! My problem was that I didn't
> know about the _shutdown option so I was simply killing the process
> therefore forcing the system to recover the indices.
>
> As far as the migration from solr to elasticsearch is concerned, I
> basically want the indexed/analyzed but unstored field to be transferred
> from solr to ES, so I can perform a full-text search on it.
> So are there tools allowing me to copy the lucene indexes over to
> elasticsearch and allow me to have the same functionality?
>
> To retrieve the actual document, I'll simply take the id and retrieve the
> document from the storage. This is how the system was built before and how
> I have to test it: indexed but unstored fields are kept inside solr, which
> is queried for full-text searches. Actual documents are kept in a separate
> filesystem. The results of the queries are taken and used to retrieve the
> actual documents from this filesystem.
>
> If we decide to move with ES, then we could change the approach and have
> everything stored inside ES and reindex our full archive.
>
> Thanks for the sharding advice, I realize I cannot use sharding with the
> current configuration. The current system in solr has just 1 collection
> with 1 core and 1 instance.
>
> We are confronting performances between ES and SOLR multicore on
> distributed system (not cloud, but simply having several instances and
> balance the load using a custom algorithm, to have more control on where
> the data goes) and after this we'll decide where we should go.
>
> Thanks
>
> Il giorno martedì 3 giugno 2014 09:55:21 UTC-7, Jörg Prante ha scritto:
>>
>>  If you have indexed the data in Solr, you should consider a tool that
>> can traverse the Lucene index and reconstruct the documents. This is not a
>> straightforward process, as you know already, because analyzed fields look
>> different than the original input. The reconstruction may not recover the
>> original input, but could be used for input into Elasticsearch, when
>> transformed to JSON. It heavily depends on the Solr analyzers you used.
>>
>> You know that Elasticsearch index is sharded, so it is obvious you have
>> to reindex the documents in order to take advantage of ES sharding.
>>
>> What time intervals do you mean to be expected at ES startup? When
>> shutting down ES, you should use the _shutdown endpoint for a clean
>> shutdown. A clean shutdown writes checksums to disk for fast startup. When
>> starting with valid checksums, ES is available within a few seconds and
>> turns to state "green". Otherwise it performs indices recovery. After all
>> shards respond after invalid checksums, and this duration is due to the
>> shard sizes and disk I/O speed, an ES cluster starts usually within 30
>> seconds to 1 minute. It can not do much faster after unclean shutdowns
>> because of the index recovery. The recovery, like index/search depends on
>> the overall power of your ES cluster. There are tunables to increase
>> recovery speed, by suppressing search/index performance at the same time.
>>
>> Jörg
>>
>> Am 02.06.14 21:33, schrieb Diego Marchi:
>>
>> Hello all,
>>
>>  I'm testing the ES environment to see if a migration from Solr could
>> bring benefits to our system. We are considering a complete renovation of
>> our service, taking it from Java to Python plus a lot of new enhancements.
>>
>>  Currently we use Solr for indexing purposes. We store webpages from
>> customers and index them using solar. Within a solr document we have a
>> dozen of fields to keep track of the data, the data itself is indexed in
>> Solr in a *content *field which is set (in the schema.xml) to be
>> indexed="true" stored="false". In fact, I can do a text search on it but I
>> cannot retrieve the whole field (obviously..)
>>
>>  The actual content is saved on our server and it is a massive 22TB of
>> data. You'll understand we cannot reindex the whole thing just for testing
>> purposes. We're considering to use a subset of it but also this is time
>> consuming.
>>
>>  I was looking if there was any way to transfer the indexed but unstored
>> *content *field directly from solr to elastic sea

Re: Migration from Solr to ElasticSearch

2014-06-03 Thread Diego Marchi

Thank you Jorg,

I'll start from the second question: Thanks! My problem was that I didn't 
know about the _shutdown option so I was simply killing the process 
therefore forcing the system to recover the indices. 

As far as the migration from solr to elasticsearch is concerned, I 
basically want the indexed/analyzed but unstored field to be transferred 
from solr to ES, so I can perform a full-text search on it. 
So are there tools allowing me to copy the lucene indexes over to 
elasticsearch and allow me to have the same functionality?

To retrieve the actual document, I'll simply take the id and retrieve the 
document from the storage. This is how the system was built before and how 
I have to test it: indexed but unstored fields are kept inside solr, which 
is queried for full-text searches. Actual documents are kept in a separate 
filesystem. The results of the queries are taken and used to retrieve the 
actual documents from this filesystem. 

If we decide to move with ES, then we could change the approach and have 
everything stored inside ES and reindex our full archive.

Thanks for the sharding advice, I realize I cannot use sharding with the 
current configuration. The current system in solr has just 1 collection 
with 1 core and 1 instance.

We are confronting performances between ES and SOLR multicore on 
distributed system (not cloud, but simply having several instances and 
balance the load using a custom algorithm, to have more control on where 
the data goes) and after this we'll decide where we should go.

Thanks

Il giorno martedì 3 giugno 2014 09:55:21 UTC-7, Jörg Prante ha scritto:
>
>  If you have indexed the data in Solr, you should consider a tool that 
> can traverse the Lucene index and reconstruct the documents. This is not a 
> straightforward process, as you know already, because analyzed fields look 
> different than the original input. The reconstruction may not recover the 
> original input, but could be used for input into Elasticsearch, when 
> transformed to JSON. It heavily depends on the Solr analyzers you used. 
>
> You know that Elasticsearch index is sharded, so it is obvious you have to 
> reindex the documents in order to take advantage of ES sharding.
>
> What time intervals do you mean to be expected at ES startup? When 
> shutting down ES, you should use the _shutdown endpoint for a clean 
> shutdown. A clean shutdown writes checksums to disk for fast startup. When 
> starting with valid checksums, ES is available within a few seconds and 
> turns to state "green". Otherwise it performs indices recovery. After all 
> shards respond after invalid checksums, and this duration is due to the 
> shard sizes and disk I/O speed, an ES cluster starts usually within 30 
> seconds to 1 minute. It can not do much faster after unclean shutdowns 
> because of the index recovery. The recovery, like index/search depends on 
> the overall power of your ES cluster. There are tunables to increase 
> recovery speed, by suppressing search/index performance at the same time.
>
> Jörg
>
> Am 02.06.14 21:33, schrieb Diego Marchi:
>  
> Hello all, 
>
>  I'm testing the ES environment to see if a migration from Solr could 
> bring benefits to our system. We are considering a complete renovation of 
> our service, taking it from Java to Python plus a lot of new enhancements. 
>
>  Currently we use Solr for indexing purposes. We store webpages from 
> customers and index them using solar. Within a solr document we have a 
> dozen of fields to keep track of the data, the data itself is indexed in 
> Solr in a *content *field which is set (in the schema.xml) to be 
> indexed="true" stored="false". In fact, I can do a text search on it but I 
> cannot retrieve the whole field (obviously..)
>
>  The actual content is saved on our server and it is a massive 22TB of 
> data. You'll understand we cannot reindex the whole thing just for testing 
> purposes. We're considering to use a subset of it but also this is time 
> consuming.
>
>  I was looking if there was any way to transfer the indexed but unstored 
> *content *field directly from solr to elastic search.
>
>  On another topic, when I shut down and turn on again the ES engine, I 
> noticed that the documents are not all available at once, but they take 
> time to load.
> Is that an expected behavior or is there a way (configuration option..) to 
> have all the documents available right away? I'm thinking, for instance, if 
> I have to update the engine or add some more options or for whatever reason 
> I need to turn down the engine and turn it on again, do I need to wait for 
> all the documents to be loaded in the system?
>  With Solr I see all of them available immediately after the search 
> engine has been launched...
>
>  Thank you,
> Diego
>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email t

Re: Migration from Solr to ElasticSearch

If you have indexed the data in Solr, you should consider a tool that can
traverse the Lucene index and reconstruct the documents. This is not a
straightforward process, as you know already, because analyzed fields look
different than the original input. The reconstruction may not recover the
original input, but could be used for input into Elasticsearch, when
transformed to JSON. It heavily depends on the Solr analyzers you used.

You know that Elasticsearch index is sharded, so it is obvious you have to
reindex the documents in order to take advantage of ES sharding.

What time intervals do you mean to be expected at ES startup? When shutting
down ES, you should use the _shutdown endpoint for a clean shutdown. A
clean shutdown writes checksums to disk for fast startup. When starting
with valid checksums, ES is available within a few seconds and turns to
state "green". Otherwise it performs indices recovery. After all shards
respond after invalid checksums, and this duration is due to the shard
sizes and disk I/O speed, an ES cluster starts usually within 30 seconds to
1 minute. It can not do much faster after unclean shutdowns because of the
index recovery. The recovery, like index/search depends on the overall
power of your ES cluster. There are tunables to increase recovery speed, by
suppressing search/index performance at the same time.

Jörg

Am 02.06.14 21:33, schrieb Diego Marchi:

Hello all,

I'm testing the ES environment to see if a migration from Solr could bring
benefits to our system. We are considering a complete renovation of our
service, taking it from Java to Python plus a lot of new enhancements.

Currently we use Solr for indexing purposes. We store webpages from
customers and index them using solar. Within a solr document we have a
dozen of fields to keep track of the data, the data itself is indexed in
Solr in a *content *field which is set (in the schema.xml) to be
indexed="true" stored="false". In fact, I can do a text search on it but I
cannot retrieve the whole field (obviously..)

The actual content is saved on our server and it is a massive 22TB of
data. You'll understand we cannot reindex the whole thing just for testing
purposes. We're considering to use a subset of it but also this is time
consuming.

I was looking if there was any way to transfer the indexed but unstored
*content *field directly from solr to elastic search.

On another topic, when I shut down and turn on again the ES engine, I
noticed that the documents are not all available at once, but they take
time to load.
Is that an expected behavior or is there a way (configuration option..) to
have all the documents available right away? I'm thinking, for instance, if
I have to update the engine or add some more options or for whatever reason
I need to turn down the engine and turn it on again, do I need to wait for
all the documents to be loaded in the system?
With Solr I see all of them available immediately after the search engine
has been launched...

Thank you,
Diego
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8c23e11d-74fd-48c0-98b0-4d75514a6a33%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGZuAfVV6nr74EpT3DpBH6jMfryoKefLR8YTbd13HEG0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation bug? Or user error?


By the way this test fails with elastic 1.2 also.

How do I go about uploading an index with aggregation request json, etc?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation bug? Or user error?

I have managed to produce a unit test that exposes this (albeit different 
to the data above).
The index is quite small - and the data fictional - so theres no problem 
sending you the index.

Here is a result I get - and we can see the sub-aggregations have higher 
counts than the parent:
{
"sales_quotas": {
"doc_count": 6,
"shipmentDate": {
"buckets": [
{
"key": "Overdue",
"to": 1.3989024E12,
"to_as_string": "2014-05-01",
"doc_count": 2,
"nothingAllocated": {
"doc_count": 6,
"ME": {
"doc_count": 0
},
"NOT_ME": {
"doc_count": 6
}
}
}
]
}
}
}








On Tuesday, 6 May 2014 10:34:53 UTC+1, mooky wrote:
>
> I am using elastic 1.1.1.
> The index isn't huge (600m) - but it contains financially sensitive 
> data... will be too problematic legally to allow it offsite. I can try 
> anonymise the data - see if it can be reproduced that way - might learn 
> something about what is causing it.
>
>
>
>
>
> On Friday, 2 May 2014 14:34:21 UTC+1, Adrien Grand wrote:
>>
>> What version of Elasticsearch are you using? If it is small enough, I 
>> would also be interested if you could share your index so that I can try to 
>> reproduce the issue locally.
>>
>>
>> On Fri, May 2, 2014 at 12:07 PM, mooky  wrote:
>>
>>>  
>>> I havent been able to figure out what is required to recreate it.
>>> I am doing a number of identical aggregations (just different values 
>>> intentMarketCode 
>>> and intentDate
>>> Three aggregations give correct numbers - one doesnt I havent 
>>> figured why
>>>  
>>>
>>> On Wednesday, 30 April 2014 14:13:00 UTC+1, Adrien Grand wrote:
>>>
 This looks wrong indeed. By any chance, would you have a curl 
 recreation of this issue?


 On Tue, Apr 29, 2014 at 7:35 PM, mooky  wrote:

> It looks like a bug to me - but if its user error, then obviously I 
> can fix it a lot quicker :)
>  
>
> On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
>
>>  I am seeing some very odd aggregation results - where the sum of 
>> the sub-aggregations is more than the parent bucket.
>>
>> Results:
>> "CSSX" : {
>>   "doc_count" : *24*,
>>   "intentDate" : {
>> "buckets" : [ {
>>   "key" : "Overdue",
>>   "to" : 1.3981248E12,
>>   "to_as_string" : "2014-04-22",
>>   "doc_count" : *1*,
>>   "ME" : {
>> "doc_count" : *0*
>>   },
>>   "NOT_ME" : {
>> "doc_count" : *24*
>>   }
>> }, {
>>   "key" : "May",
>>   "from" : 1.3981248E12,
>>   "from_as_string" : "2014-04-22",
>>   "to" : 1.4006304E12,
>>   "to_as_string" : "2014-05-21",
>>   "doc_count" : *23*,
>>   "ME" : {
>> "doc_count" : 0
>>   },
>>   "NOT_ME" : {
>> "doc_count" : *24*
>>   }
>> }, {
>>   "key" : "June",
>>   "from" : 1.4006304E12,
>>   "from_as_string" : "2014-05-21",
>>   "to" : 1.4033088E12,
>>   "to_as_string" : "2014-06-21",
>>   "doc_count" : *0*,
>>   "ME" : {
>> "doc_count" : *0*
>>   },
>>   "NOT_ME" : {
>> "doc_count" : *24*
>>   }
>> } ]
>>   }
>> },
>>
>>
>> I wouldn't have thought that to be possible at all.
>> Here is the request that generated the dodgy results.
>>
>>
>> "CSSX" : {
>>   "filter" : {
>> "and" : {
>>   "filters" : [ {
>> "type" : {
>>   "value" : "inventory"
>> }
>>   }, {
>> "term" : {
>>   "isAllocated" : false
>> }
>>   }, {
>> "term" : {
>>   "intentMarketCode" : "CSSX"
>> }
>>   }, {
>> "terms" : {
>>   "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873"
>> , "0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91F
>> BA", "33A946DC2B0E494EB371993D345F52E4", "
>> 6471AA50DFCF4192B8DD1C2E72A032C7", "9FB2FFDC0FF0797FE04014AC6F0616B6"
>> , "9FB2FFDC0FF1797FE04014AC6F0616B6", "9FB2FFDC0FF2797FE04014AC6F0616
>> B6", "9FB2FFDC0FF3797FE04014AC6F0616B6", "
>> 9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6"
>> , "AFE0FED33F06AFB6E04015AC5

Re: Document found with _search but not with GET

2014-06-03 Thread David Pilato

Yeah. We just released 1.2.1 which fix that issue.

See http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 3 juin 2014 à 16:31:56, Neil Andrassy (neil.andra...@thefilter.com) a écrit:

Looks like this might be a bug in 1.2 -
https://github.com/elasticsearch/elasticsearch/pull/6393

On Monday, 26 May 2014 08:47:41 UTC+1, Pir Abdul Rasool Qureshi wrote:
Hi

We have a single machine elastic search server (8 shards but all hosted at the
same machine). Index contains 7 million documents. We do not specify any custom
routing when indexing the documents. We are using Elastic search version 1.2.

The problem is that we are unable to retrieve many of our documents using GET
, . However using search?_id: we are able to retrieve all of those documents.

We are also successful in retrieving a document by specifying routing parameter
(with different values (1,2,3,...) ) with GET.

With previous version, i.e. Elastic Search 1.0.3, we did not have that problem.

Any suggestions for resolution?
Thanks in advance
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5f64de11-d918-4120-a899-0a5daf44df4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.538df986.5ff87e05.28b%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Migration from Solr to ElasticSearch

2014-06-03 Thread Diego Marchi

Hi, thanks for the answer!

Since they both share Lucene as common underlying engine, this could be a 
starting point.. but are we sure that both the engines store and structure 
the information in the same way? In this case the porting should be pretty 
easy.. 

Do you have an e-guide or a handbook you could suggest me, on how the data 
in ES and solr is structured?

Thank you
Diego

Il giorno lunedì 2 giugno 2014 20:54:44 UTC-7, Otis Gospodnetic ha scritto:
>
> Hi,
>
> You could migrate from Solr to ES without reindexing because at the end of 
> the day it is Lucene that writes data to index.
> You'd want to make sure your ES mappings match your Solr schema.
> You'd want to create the matching number of shards and replicas you had in 
> Solr(Cloud?).
> You'd manually copy Lucene indexes from Solr to ES and pray.
> I'm sure I'm skipping over about a dozen details you can trip over, though.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Monday, June 2, 2014 3:33:38 PM UTC-4, Diego Marchi wrote:
>>
>> Hello all,
>>
>> I'm testing the ES environment to see if a migration from Solr could 
>> bring benefits to our system. We are considering a complete renovation of 
>> our service, taking it from Java to Python plus a lot of new enhancements. 
>>
>> Currently we use Solr for indexing purposes. We store webpages from 
>> customers and index them using solar. Within a solr document we have a 
>> dozen of fields to keep track of the data, the data itself is indexed in 
>> Solr in a *content *field which is set (in the schema.xml) to be 
>> indexed="true" stored="false". In fact, I can do a text search on it but I 
>> cannot retrieve the whole field (obviously..)
>>
>> The actual content is saved on our server and it is a massive 22TB of 
>> data. You'll understand we cannot reindex the whole thing just for testing 
>> purposes. We're considering to use a subset of it but also this is time 
>> consuming.
>>
>> I was looking if there was any way to transfer the indexed but unstored 
>> *content *field directly from solr to elastic search.
>>
>> On another topic, when I shut down and turn on again the ES engine, I 
>> noticed that the documents are not all available at once, but they take 
>> time to load.
>> Is that an expected behavior or is there a way (configuration option..) 
>> to have all the documents available right away? I'm thinking, for instance, 
>> if I have to update the engine or add some more options or for whatever 
>> reason I need to turn down the engine and turn it on again, do I need to 
>> wait for all the documents to be loaded in the system?
>> With Solr I see all of them available immediately after the search engine 
>> has been launched...
>>
>> Thank you,
>> Diego
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9c504ea4-3c37-4816-990f-800c3c2a0959%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: encoding is longer than the max length 32766

2014-06-03 Thread Karel Minařík

This is actually a change in Lucene -- previously, the long term was 
silently dropped, now it raises an exception, see Lucene 
ticket https://issues.apache.org/jira/browse/LUCENE-5710

You might want to add a `length` filter to your analyzer 
(http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-length-tokenfilter.html#analysis-length-tokenfilter).

All in all, it hints at some strange data, because such "immense" term 
shouldn't probably be in the index in the first place.

Karel

On Thursday, May 29, 2014 10:47:37 PM UTC+2, Jeff Dupont wrote:
>
> We’re running into a peculiar issue when updating indexes with content for 
> the document.
>
>
> "document contains at least one immense term in (whose utf8 encoding is 
> longer than the max length 32766), all of which were skipped. please 
> correct the analyzer to not produce such terms”
>
>
> I’m hoping that there’s a simple fix or setting that can resolve this.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a91895cb-437a-4642-8734-4445bb420125%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Index template requires settings object even if its value is empty

2014-06-03 Thread Brian

I am not sure if this is a problem or if it's OK.

Working with the ELK stack I have switched direction, and instead of
locking down the Elasticsearch mappings I am now using its automatic
mapping functions. And by adding the following JSON template definition to
the /*path.to.config*/templates/automap.json file I can get numeric fields
automatically correctly mapped even though logstash always emits their
values as strings ("45.6" instead of 45.6). Very nice!

{
"automap" : {
"template" : "*",
*"settings" : { },*
"mappings" : {
"_default_" : {
"numeric_detection" : true,
"properties" : {
"message" : {"type" : "string"},
"host" : {"type" : "string"},
"@version" : {"type" : "string"}
}
}
}
}
}

When I removed the *"settings":{}* entirely, it was as if the template did
not exist; the numeric detection was not enabled and all string values were
seen as strings even if they contained numbers. Because all of the settings
are being controlled within elasticsearch.yml and not the template (e.g.
number of shards, number of replicas, and so on), eliminating the settings
from the template is desired, even if I have to leave it in but set its
value to the empty JSON object.

If this is the way it's supposed to work, that's OK. But I couldn't find
anything in the documentation about it, and just wanted to get a
verification either way.

Thanks!

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch/Lucene Delete space reuse? recovery?

2014-06-03 Thread smonasco

I'm starting a project to index log files.  I don't particularly want to 
wait until the log files roll over.  There will be files from 100's of apps 
running across 100's of machines (not all apps intersect with all machines, 
but you get the drift).  Some roll over very fast; some may take days.

The problem comes that if I am constantly reindexing the same document 
(same id) am I loosing all old space (store and or index) or is 
Elasticsearch/Lucene smart enough to say here's a new version we'll 
overwrite the old store/index entries and point to this one where they are 
the same and add new ones.

Certainly, there is a more sophisticated model that treats every line as a 
unique document/row such that this doesn't become an issue, but I'm not 
ready to spend that kind of dev and hardware at this issue.  (Our 
elasticsearch solution is wrapped in a system that becomes really heavy 
handed when indexing such small pieces.)

--Shannon Monasco

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9d9d38f7-ba4f-470c-9864-5b9af8abc773%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Elasticsearch Simple Action Plugin

Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of
Elasticsearch is the plugin mechanism. Here, custom code can be hooked into
Elasticsearch, without having to ask for inclusion into the core code.
Nevertheless, plugin code can be published on Github and easily included
into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many
plugins, some of them are very advanced, and finding a starting point for a
personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting
point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard
search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query)
is logged

The plugin code comes with a junit test. It is available at

https://github.com/jprante/elasticsearch-simple-action-plugin

In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: avg aggregation on string values

2014-06-03 Thread Subhadip Bagui

Hi,

please suggest...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cef8d6e2-95f7-4040-9683-df188f1717cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Red status unassigned shards help

2014-06-03 Thread Jason Weber

Mark, appreciate the response I will look into both!


On Fri, May 30, 2014 at 5:47 PM, Mark Walkom 
wrote:

> You can set the replicas for an index using the API (or kopf).
>
> As for your upgrade concerns, see
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 31 May 2014 00:15, Jason Weber  wrote:
>
>> Thanks Mark and pawan,
>>
>> Here is my output from netstat:
>>
>> tcp6   0  0 :::9200 :::*
>> LISTEN  1155/java
>>
>> Mark are you talking about upgrading to the lastest 0.9 or to 1.x.x?
>> Still waiting on a good method to go to the lastest 1.x in ES with out
>> messing up a bunch of stuff. Still in development but dont want to loose my
>> data.
>>
>> I think you are right about the replica set, I read about a setting I
>> need to change in elasticsearch.yml, I will see if I can find that doc.
>> Also will install kopf. Thanks again for the help!
>>
>>
>> On Friday, May 30, 2014 12:18:47 AM UTC-4, Mark Walkom wrote:
>>>
>>> It could also be the elasticsearch integrated output in ES, which adds
>>> the LS instance as a client node to the cluster.
>>> And you probably don't want to kill that.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 30 May 2014 14:11, Pawan Sharma  wrote:
>>>
 In the node another instances of elasticsearch is started, so the
 solution is first you have to find the PID ok another instances of es by


 *netstat -lnp | grep 920*
 and kill the PID if there is another es is started in 9201  port

 Thanks


 On Fri, May 30, 2014 at 4:03 AM, Mark Walkom >>> > wrote:

> Install a visual monitoring plugin like kopf and ElasticHQ, you will
> be able to see which shards are unassigned.
> However I think you may have replicas set, which, given you only have
> one one, will always result in a yellow state as the cluster cannot assign
> replicas to another node.
>
> You should also upgrade ES to a newer version if you can :)
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 29 May 2014 23:45, Jason Weber  wrote:
>
>> I rebooted several times and I believe its collecting the correct
>> data now. I still show 520 unassigned shards, but its collecting all my
>> logs now. Is this something I can use the redirect command for to assign 
>> it
>> to a new index?
>>
>> Jason
>>
>> On Tuesday, May 27, 2014 11:39:49 AM UTC-4, Jason Weber wrote:
>>>
>>> Could someone walk me through getting my cluster up and running.
>>> Came in from long weekend and my cluster was red status, I am showing a 
>>> lot
>>> of unassigned shards.
>>>
>>> jmweber@MIDLOG01:/var/log/logstash$ curl localhost:9200/_cluster/
>>> health?pretty
>>> {
>>>   "cluster_name" : "midlogcluster",
>>>   "status" : "red",
>>>   "timed_out" : false,
>>>   "number_of_nodes" : 2,
>>>   "number_of_data_nodes" : 1,
>>>   "active_primary_shards" : 512,
>>>   "active_shards" : 512,
>>>   "relocating_shards" : 0,
>>>   "initializing_shards" : 0,
>>>   "unassigned_shards" : 520
>>> }
>>>
>>>
>>> I am running ES 0.90.11
>>>
>>> LS and ES are on a single server, I only have 1 node, although it
>>> shows 2, I get yellow status normally, it works fine with that. But I am
>>> only collecting like 43 events per minute vs my usual 50K.
>>>
>>> I have seen several write ups but I seem to get a lot of no handler
>>> found for uri statements when I try to run them.
>>>
>>> Thanks,
>>> Jason
>>>
>>  --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/elasticsearch/1307dd8d-411e-4690-a6d1-8e27ce26ecec%
>> 40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit https://grou

Re: Document found with _search but not with GET

2014-06-03 Thread Neil Andrassy

Looks like this might be a bug in 1.2 - 
https://github.com/elasticsearch/elasticsearch/pull/6393

On Monday, 26 May 2014 08:47:41 UTC+1, Pir Abdul Rasool Qureshi wrote:
>
> Hi
>
> We have a single machine elastic search server (8 shards but all hosted at 
> the same machine). Index contains 7 million documents. We do not specify 
> any custom routing when indexing the documents. We are using Elastic search 
> version 1.2.  
>
> The problem is that we are unable to retrieve many of our documents using GET 
> , . However using search?_id: we are able to retrieve all 
> of those documents.
>
> We are also successful in retrieving a document by specifying routing 
> parameter (with different values (1,2,3,...) ) with GET.
>
> With previous version, i.e. Elastic Search 1.0.3, we did not have that 
> problem.
>
> Any suggestions for resolution?
>  
> Thanks in advance
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5f64de11-d918-4120-a899-0a5daf44df4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sort descending with JavaScript API

2014-06-03 Thread el

Great, this works. Thanks a lot.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c171f577-2d60-42de-82e0-bec45e2c4921%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hadoop] Setting Document ID in Map Reduce Mapper

2014-06-03 Thread Daniel Tardón

Hi all,

I'm newbie with ES and i'm trying to set manually each document ID. I've 
seen in the documentation the *es.mapping.id* propperty and I'm trying to 
set it in the conf part of the driver class the same way i set the index 
and type of documents:

conf.set("es.resource", "logs/{event}");
> conf.set("es.mapping.id", "id"); 


In the Mapper class I put in the MapWritable object a new key value pair 
for each map:

MapWritable doc = new MapWritable();
> String id = node+"|"+timestamp; //node and timestamp are two String values 
> that I have.
> doc.put(new Text("id"), new Text(id));


And as a result I can't write in ES and get exceptions with this message: 
JsonParseException[Unexpected character ('"' (code 34))

If I comment the es.mapping.id line and allow ES to set the documents ID 
everything works fine. 

What could I do?

Thanks in advance  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ae11fa62-582e-4c67-8819-cd8616243e8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch error log with Popcorn Webmaker

2014-06-03 Thread arshpreet singh

Hi when I save video using popcorn and again try to edit saved video
the same video does not come back and new video started.
At the specific time when unable to recall saved video elasticsearch
shows the following error log.
http://pastebin.com/REypwqfF

-- 

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

ਜਿੰਦਗੀ ਬਹੁਤ ਲੰਬੀ ਹੁੰਦੀ ਹੈ ਅਤੇ ਦੁਨੀਆ ਬਹੁਤ ਛੋਟੀ |

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAstK2GyoADKUAwSJsDD3CFYAt3q1ydkDO4sXLFxMn7jbQ7swA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unexpected behavior from nested -> filter -> nested aggregation

2014-06-03 Thread Michael Sander

I'm jumping into the world of nested filter aggregations. Did you ever 
figure this out?

On Tuesday, January 21, 2014 3:15:11 PM UTC-5, John Freeman wrote:
>
> Bumping because I posted this on the weekend to few views. Should I file a 
> bug report?
>
> On Monday, January 20, 2014 3:55:54 PM UTC-5, John Freeman wrote:
>>
>> First, let me say I'm very excited about the new aggregations. Great work!
>>
>> I've got a type with two layers of nesting:
>>
>> script:
>>   calls: [
>> name: string
>> params: [
>>   name: string
>>   value: string
>> ]
>>   ]
>>
>> I want to run an aggregation over the parameter values for calls to a 
>> specific function. Here's the skeleton of what I tried:
>>
>> 'aggs': {'b': {
>>   'nested': {'path': 'calls'},
>>   'aggs': {'c': {
>> 'filter': {'term': {'calls.name': 'particular_func'}},
>> 'aggs': {'d': {
>> 'nested': {'path': 'calls.params'},
>> 'aggs': ...
>>
>> The structure is three aggregations: a nested wrapping a filter wrapping 
>> a nested. Checking the doc counts on these, I see that the outer two work 
>> as expected: the doc count for the outer nested is the number of nested 
>> "calls" documents, the doc count for the filter is the number of those 
>> nested "calls" docs that pass the filter. But it appears that the inner 
>> nested resets the buckets: it returns the number of inner nested "params" 
>> documents across all "calls" docs, regardless of the filter.
>>
>> Is there a way to do what I want?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/50159e46-17b9-4ad3-9911-20cbcfe7eae4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

recovery took long

2014-06-03 Thread jaguarx


Hi,
We just upgrade ES from 0.90 to 1.0.1, and met some strange issue. 
The cluster status is green, but /_cat/recovery shows one ongoing task 
on index1. When trying to flush index1, we got error saying 
"...recoverying, flush not allowed...". And the recovery took almost one 
day, but still didn't complete.
Is this a known issue for 1.0.1? If it's, which version contains 
the bug fix?


Thanks!
Jaguar

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/538DC6D5.3090003%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

ChangeLogLevel

2014-06-03 Thread Anantha Govindarajan

Hi,

How to dynamically change the log level for specified node ? The following 
link is pretty helpful to change the log-level for the entire cluster.

https://github.com/elasticsearch/elasticsearch/issues/2517


When changing log level to entire cluster it also affects client nodes too 
(NodeClient). 

How to change log-level dynamically without affecting client nodes ? 


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6560db05-64ec-4cac-a553-6ba0177647ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Queries, filters and match_all

2014-06-03 Thread Arkadiy Zabazhanov

Hello. Help me please, I'm confused. As far as I remember, there was the
only way to pass filters to search query - via filtered query. But
currently there is a top-level filter part of the query. However, top-level
filter affects query only and doesn't affect i.e. facets. But filtered
query filter affects both of the query and facets facilities. Also, I
remember there was a time I need to add match_all query to filtered query
section
if query was empty and filters only was present. Otherwise returned empty
set of documents. Since I'm trying to create high-level Ruby library could
you please answer following questions:

1) Which way is preferred now and in future: filtered top-level query or
top-level filter with top-level query?
2) How do you plan to resolve such an API inconsistency when filtered query
filter affects outside statements and top-level filter doesn't affect some
parts of request?
3) Why do I remember about match_all feature and when did requests started
to return all the documents with empty query section in filtered query? I'm
checking it right now on 1.2.0 and I don't need to use match_all, or
constant_score it just returns all the docs for me.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1325caa9-cd4b-492e-a443-f4b63f99c104%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch with Webmaker

2014-06-03 Thread arshpreet singh

Hi I have no idea some days ago about Elasticsearch but it came with
Mozilla Webmaker suit. I want to install Webmaker on localhost and I
am able to install all parts of webmaker sucessfully but Elasticsearch
is causing some problems due to not configured properly so my question
is:
Is there any specific configuration of Elasticsearch for Webmaker?
If not
Where should I look for or what should I do. Really trapped :(

-- 

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

ਜਿੰਦਗੀ ਬਹੁਤ ਲੰਬੀ ਹੁੰਦੀ ਹੈ ਅਤੇ ਦੁਨੀਆ ਬਹੁਤ ਛੋਟੀ |

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAstK2Em7qHC4Y7f9UAfEFRYgXmxBbTqGmV8RdG2c49f8iOG_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Synonym filter behavior for single word / multi words

2014-06-03 Thread Bernhardt Scherer


>
> Hey Binh,
>
> thanks for your reply!
>
> I tried the following:
>
> 1.
> POST localhost:9200/index_v2/
>
> {
> "settings": {
> "analysis": {
> "filter": {
> "synonym": {
> "type": "synonym",
> "synonyms": [
> "Schraubenzieher, Schraubendreher",
> "Inbus, Innensechskant, Imbus, Innen-6-Kant",
> "Innensechskantschlüssel, Inbusschlüssel",
> "Bauhelm, Schutzhelm"
> ],
> "ignore_case": true,
> "expand": true
> }
> },
> "analyzer": {
> "synonym_analyzer": {
> "type":"custom",
> "tokenizer": "whitespace",
> "filter": [
> "synonym",
> "lowercase",
> "asciifolding"
> ]
> }
> }
> }
> }
> }
>
> 2.
> POST localhost:9200/index_v2/_analyze?analyzer=synonym_analyzer
> 'Inbus'
>
> Output:
> {
> "tokens": [
> {
> "token": "'inbus'",
> "start_offset": 0,
> "end_offset": 7,
> "type": "word",
> "position": 1
> }
> ]
> }
>
> 3.
> POST localhost:9200/index_v2/_analyze?analyzer=synonym_analyzer
> 'Der Inbus ist'
>
> Output:
> {
> "tokens": [
> {
> "token": "'der",
> "start_offset": 0,
> "end_offset": 4,
> "type": "word",
> "position": 1
> },
> {
> "token": "inbus",
> "start_offset": 5,
> "end_offset": 10,
> "type": "SYNONYM",
> "position": 2
> },
> {
> "token": "innensechskant",
> "start_offset": 5,
> "end_offset": 10,
> "type": "SYNONYM",
> "position": 2
> },
> {
> "token": "imbus",
> "start_offset": 5,
> "end_offset": 10,
> "type": "SYNONYM",
> "position": 2
> },
> {
> "token": "innen-6-kant",
> "start_offset": 5,
> "end_offset": 10,
> "type": "SYNONYM",
> "position": 2
> },
> {
> "token": "ist'",
> "start_offset": 11,
> "end_offset": 15,
> "type": "word",
> "position": 3
> }
> ]
> }
>
> I have changed to 1.0.1 but the behavior was the same on 1.2..
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/18c6aa41-12b7-4301-8661-53162454f158%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Synonym filter behavior for single word / multi words

2014-06-03 Thread Bernhardt Scherer

Hey Binh,

thanks for your reply!

I tried the following:

1.
POST localhost:9200/index_v2/

{
"settings": {
"analysis": {
"filter": {
"synonym": {
"type": "synonym",
"synonyms": [
"Schraubenzieher, Schraubendreher",
"Inbus, Innensechskant, Imbus, Innen-6-Kant",
"Innensechskantschlüssel, Inbusschlüssel",
"Bauhelm, Schutzhelm"
],
"ignore_case": true,
"expand": true
}
},
"analyzer": {
"synonym_analyzer": {
"type":"custom",
"tokenizer": "whitespace",
"filter": [
"synonym",
"lowercase",
"asciifolding"
]
}
}
}
}
}

2.
POST localhost:9200/index_v2/_analyze?analyzer=synonym_analyzer
'Inbus'

Output:
{
"tokens": [
{
"token": "'inbus'",
"start_offset": 0,
"end_offset": 7,
"type": "word",
"position": 1
}
]
}

3.
POST localhost:9200/index_v2/_analyze?analyzer=synonym_analyzer
'Inbus'

Output:
{
"tokens": [
{
"token": "'der",
"start_offset": 0,
"end_offset": 4,
"type": "word",
"position": 1
},
{
"token": "inbus",
"start_offset": 5,
"end_offset": 10,
"type": "SYNONYM",
"position": 2
},
{
"token": "innensechskant",
"start_offset": 5,
"end_offset": 10,
"type": "SYNONYM",
"position": 2
},
{
"token": "imbus",
"start_offset": 5,
"end_offset": 10,
"type": "SYNONYM",
"position": 2
},
{
"token": "innen-6-kant",
"start_offset": 5,
"end_offset": 10,
"type": "SYNONYM",
"position": 2
},
{
"token": "ist'",
"start_offset": 11,
"end_offset": 15,
"type": "word",
"position": 3
}
]
}

I have changed to 1.0.1 but the behavior was the same on 1.2..
 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f57e83e1-8af1-4102-8083-b23532932ac1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation vs Search/Filter discrepancy - caching issue?

Update elastic to 1.2 - still seeing the same issue...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2805708d-57dd-4977-a17c-2c27d9ee98d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cluster gets stuck after full re-index

Am I reading that right, you're basically at 100% heap usage? If that is
the case then it'd be GC that's killing you.

Did you add more nodes when you moved to AWS or do you have the same number?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 3 June 2014 20:27, Florian Munz  wrote:

> Other than the jmap -heap I didn't manage to look more specifically into
> it:
>
> https://gist.github.com/theflow/b983d512ea344545f7f6#file-jmap
>
> The same process runs fine on much smaller machines in our staging
> environment, without the live traffic, of course.
>
> Anything particular I should run that would give more insights?
>
>
> Cheers,
> Florian
>
>
> On Tuesday, June 3, 2014 12:21:32 PM UTC+2, Mark Walkom wrote:
>>
>> How does your heap look during all this?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 3 June 2014 20:14, Florian Munz  wrote:
>>
>>> Hello,
>>>
>>> we recently moved our ES cluster from dedicated hardware to AWS
>>> instances, they have less memory available, but use SSDs for the ES data
>>> directory. We kept JVM (1.7.0_17) and ES (0.90.9) version exactly the same.
>>> On the new hardware, after running a full re-index (creating a new index,
>>> pointing an alias to the new and one alias to the old index, sending
>>> realtime updates to both aliases and running a script to fill up the new
>>> index) our cluster gets stuck.
>>>
>>> 10 minutes after the re-index finishes and we move both aliases to the
>>> new index, ES stops answering any search or index queries, no errors in the
>>> logs apart from it not answering queries anymore:
>>>
>>> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
>>> rejected execution (queue capacity 1000) on org.elasticsearch.action.
>>> search.type.TransportSearchTypeAction$BaseAsyncAction$4@172018e5
>>>
>>> CPU load is low, it doesn't look like it's doing anything expensive. A
>>> request to hot_threads times out. I've put the output from jstack and jmap
>>> here:
>>>
>>> https://gist.github.com/theflow/b983d512ea344545f7f6
>>>
>>> We tried upgrading to 0.90.13, since the changelog mentioned a problem
>>> with infinite loops, but same behavior. We're planning to upgrade to a more
>>> recent version of ES soon, but it'll take a bit to fully test that.
>>>
>>>
>>> Any ideas what could be causing this?
>>>
>>>
>>> thanks,
>>> Florian
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/04b3d0a2-a47e-47c6-8411-eb619c3c54bc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YTb8m1qRZ5iuAf3eq6v-2FkSpemmw8d2UhVJep8zt0BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cluster gets stuck after full re-index

2014-06-03 Thread Florian Munz

Other than the jmap -heap I didn't manage to look more specifically into it:

https://gist.github.com/theflow/b983d512ea344545f7f6#file-jmap

The same process runs fine on much smaller machines in our staging 
environment, without the live traffic, of course.

Anything particular I should run that would give more insights?


Cheers,
Florian


On Tuesday, June 3, 2014 12:21:32 PM UTC+2, Mark Walkom wrote:
>
> How does your heap look during all this?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 3 June 2014 20:14, Florian Munz > wrote:
>
>> Hello,
>>
>> we recently moved our ES cluster from dedicated hardware to AWS 
>> instances, they have less memory available, but use SSDs for the ES data 
>> directory. We kept JVM (1.7.0_17) and ES (0.90.9) version exactly the same. 
>> On the new hardware, after running a full re-index (creating a new index, 
>> pointing an alias to the new and one alias to the old index, sending 
>> realtime updates to both aliases and running a script to fill up the new 
>> index) our cluster gets stuck.
>>
>> 10 minutes after the re-index finishes and we move both aliases to the 
>> new index, ES stops answering any search or index queries, no errors in the 
>> logs apart from it not answering queries anymore:
>>
>> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: 
>> rejected execution (queue capacity 1000) on 
>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@172018e5
>>
>> CPU load is low, it doesn't look like it's doing anything expensive. A 
>> request to hot_threads times out. I've put the output from jstack and jmap 
>> here:
>>
>> https://gist.github.com/theflow/b983d512ea344545f7f6
>>
>> We tried upgrading to 0.90.13, since the changelog mentioned a problem 
>> with infinite loops, but same behavior. We're planning to upgrade to a more 
>> recent version of ES soon, but it'll take a bit to fully test that.
>>
>>
>> Any ideas what could be causing this?
>>
>>
>> thanks,
>> Florian
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/04b3d0a2-a47e-47c6-8411-eb619c3c54bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cluster gets stuck after full re-index

How does your heap look during all this?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 3 June 2014 20:14, Florian Munz  wrote:

> Hello,
>
> we recently moved our ES cluster from dedicated hardware to AWS instances,
> they have less memory available, but use SSDs for the ES data directory. We
> kept JVM (1.7.0_17) and ES (0.90.9) version exactly the same. On the new
> hardware, after running a full re-index (creating a new index, pointing an
> alias to the new and one alias to the old index, sending realtime updates
> to both aliases and running a script to fill up the new index) our cluster
> gets stuck.
>
> 10 minutes after the re-index finishes and we move both aliases to the new
> index, ES stops answering any search or index queries, no errors in the
> logs apart from it not answering queries anymore:
>
> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
> rejected execution (queue capacity 1000) on
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@172018e5
>
> CPU load is low, it doesn't look like it's doing anything expensive. A
> request to hot_threads times out. I've put the output from jstack and jmap
> here:
>
> https://gist.github.com/theflow/b983d512ea344545f7f6
>
> We tried upgrading to 0.90.13, since the changelog mentioned a problem
> with infinite loops, but same behavior. We're planning to upgrade to a more
> recent version of ES soon, but it'll take a bit to fully test that.
>
>
> Any ideas what could be causing this?
>
>
> thanks,
> Florian
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624Z-RX_ggzJzjLaP2%2B7Dt7a8aotvcwF7kQ98mpDS1cvZkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Cluster gets stuck after full re-index

2014-06-03 Thread Florian Munz

Hello,

we recently moved our ES cluster from dedicated hardware to AWS instances,
they have less memory available, but use SSDs for the ES data directory. We
kept JVM (1.7.0_17) and ES (0.90.9) version exactly the same. On the new
hardware, after running a full re-index (creating a new index, pointing an
alias to the new and one alias to the old index, sending realtime updates
to both aliases and running a script to fill up the new index) our cluster
gets stuck.

10 minutes after the re-index finishes and we move both aliases to the new
index, ES stops answering any search or index queries, no errors in the
logs apart from it not answering queries anymore:

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@172018e5

CPU load is low, it doesn't look like it's doing anything expensive. A
request to hot_threads times out. I've put the output from jstack and jmap
here:

https://gist.github.com/theflow/b983d512ea344545f7f6

We tried upgrading to 0.90.13, since the changelog mentioned a problem with
infinite loops, but same behavior. We're planning to upgrade to a more
recent version of ES soon, but it'll take a bit to fully test that.

Any ideas what could be causing this?

thanks,
Florian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregation vs Search/Filter discrepancy - caching issue?

I have a test that indexes some data, does an aggregation + search to 
verify the results.
Then my test changes 1 piece of data and re-indexes (everything).
refresh() is called.

Then I do an aggregation + search again to verify my results.
The search results are correct (ie updated to reflect the changed data).
The aggregation results are not correct - they are the same as the first 
aggregation results - implying the data has not changed.
The aggregation results go wrong at the result of a filter aggregation. The 
results of the filter aggregation give a count of 2 - the results of the 
search (equiv query or *same* filter) give 1.

Suspecting a caching issue (e.g. filter caching) I then made sure I flushed 
the caches for that index. But sadly, the problem still persists.

I am kinda  running out of ideas. Can anyone suggest what to check next?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/61140ef1-441f-477d-8129-d30588b6838b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Identify word as dominant word in search

2014-06-03 Thread Itamar Syn-Hershko

Depending on your corpus, this should happen automatically. That's what
TF/IDF is about.

What you can do further is use NLP methods to tag those items in search and
indexing. Look up POS tagging and entity extraction.

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 


On Tue, Jun 3, 2014 at 12:22 PM, Rotem Haber  wrote:

> hi,
> Is there a search in elasticsearch that support the behavior that when a
> user enter a string to search, the ES recognize words as important words in
> search.
> for example: the user enter the string NEXUS COVER FOR EVERY DAY USE SILK
> SOFT BLUE, and I want that the brand(NEXUS) and color(BLUE) will be more
> dominant in search, and I have a list of all the colors and all the brand
> that exist.
>
> is it possible? and if yes, how do I implement that?
>
> thank you!
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2948b28d-6c1e-490d-bdbb-80df5d7b0ebd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWggX9KP_fd75Qbfgk4uph9VNMbXUyaQQ3obMresdVyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch Server Not Utilizing Full Memory

2014-06-03 Thread Ajay Singh

 

I have recently deployed ElasticSearch at our local servers as well as on a 
client server.

 

As we know ElasticSearch uses 1GB memory (RAM) by default.

 

I increased this memory by setting the environment variable ES_HEAP_SIZE to 
4g on local server (where RAM is 8GB). On local server it immediately takes 
the given memory and starts using it.

 

However when we follow the same step on Client Server and set ES_HEAP_SIZE 
to 12g (RAM is 24GB), ElasticSearch seems to be using only 1GB at beginning 
which slowly increases. As a result on client servers we are getting very 
poor performance and even many service hits get timeout.

 

We have verified that JVM is actually getting the allocated memory size, 
but somehow this memory is not being utilized by ElasticSearch on Client 
servers.

 

Our local server is Windows 7 Professional SP1, while client server has got 
Windows 2012 R2 Standard edition. 

 

Please help me with this issue. I have 64bit version of jdk1.7.0_55 on 
Windows 2012 R2.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d29181a7-74a8-4f20-83bd-b0e858b92e92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to debug aggregation problems?