Can ES scale to 30TB / day, and still be usable?

[Nelson Jeppesen]

Can ES scale to 30TB / day, and still be usable?

This is a typical logstash/elasticsearch/kibana setup. I have a small 
environment logging 20GB / day that seems to work fine. At 30TB, very 
little will be able to cached into ram, can ES still be usable at that 
point?

Also, what's is the best way to pick the proper index creation rate (per 
day, per hour?). Is there a guideline for max. index size?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/865e67f5-bcd4-4247-9f39-813424d6747c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: The Max Number of Filelds ES Can Support

[xiehaiwei]

Hi, 

  Actually, search need go over all fields,  every field has special 
definition, the number of fields is too big to remember, 
I need  search all fields for any keywords, return matched docs and field 
name to request.

 Why so many fields, cause the data come from tables of database. 
Client want a system to search any data conveniently.

 There are other problems like, search results from two tables need do 
a join-like operation.

 These requirements exceed  ES capcacity, I am not sure.

Thanks.

 

> A search is independent from the number of fields as long as you do not 
> search over this number of fields.
>
> You can create as much fields as your memory und resources will let you do 
> so.
>
> Jörg 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/93c6af1d-0f3b-4243-b79c-b0e682e4520c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problems upgrading an existing field to a multi-field

[Travis Bullock]

I have the same problem with lowercase_terms. Is there a fix? The term 
suggester lowercases them by default (not desirable), and the completion 
suggester doesn't.

On Tuesday, July 8, 2014 4:14:22 PM UTC-5, Ryan Tanner wrote:
>
> Side question:
>
> If I try to set lowercase_terms to true, I get a 400 back saying 
> "suggester[term] doesn't support [lowercase_terms]" which seems to 
> contradict the documentation.
>
>   "suggest" : {
> "text" : "my query string",
> "person_name" : {
>   "term" : {
> "field" : "name.suggestion",
> "size" : 1,
> "sort" : "score",
> "lowercase_terms" : true
>   }
> }
>   }
>
> Fails against both 1.1.1 and 1.2.1.
>
> On Tuesday, July 8, 2014 12:53:57 PM UTC-6, Ryan Tanner wrote:
>>
>> 1.1.1 in production but I tested this with 1.2.1 locally and had the same 
>> problem.
>>
>> On Tuesday, July 8, 2014 12:53:14 PM UTC-6, kimchy wrote:
>>>
>>> Which version of ES are using? I believe we fixed a bug around this 
>>> several versions ago.
>>>
>>> On Jul 8, 2014, at 20:31, Ryan Tanner  wrote:
>>>
>>> *bump*
>>>
>>> Anyone?
>>>
>>> On Monday, July 7, 2014 5:15:06 PM UTC-6, Ryan Tanner wrote:

 I'm having trouble upgrading an existing field to a multi-field.  I've 
 done this before with no issues on other fields.

 I think the issue here is that the original mapping specifically 
 defines an analyzer:

   "mappings" : {
 "person" : {
   "properties" : {
 "domain_titles" : {
   "type" : "string",
   "analyzer" : "stop",
   "include_in_all" : true
 }
   }
 }
   }

 The other fields that have been upgraded do not have an analyzer in the 
 original mapping.

 This is the upgrade I'm attempting:

 {
   "settings" : {
 "index.analysis.filter.shingle_filter.type" : "shingle",
 "index.analysis.filter.shingle_filter.min_shingle_size" : 2,
 "index.analysis.filter.shingle_filter.max_shingle_size" : 5,
 "index.analysis.analyzer.shingle_analyzer.type" : "custom",
 "index.analysis.analyzer.shingle_analyzer.tokenizer" : "standard",
 "index.analysis.analyzer.shingle_analyzer.filter" : [ "lowercase",
 "shingle_filter" ]
   },
   "mappings" : {
 "person" : {
   "properties" : {
 "domain_titles" : {
   "type" : "string",
   "fields" : {
 "suggestions" : {
   "type" : "string",
   "index" : "analyzed",
   "include_in_all" : false,
   "analyzer" : "nicknameAnalyzer"
 }
   }
 }
   }
 }
   }
 }

 Is there any reason why this sort of upgrade should fail?  This is the 
 error message I get:


 {"error":"MergeMappingException[Merge failed with failures {[mapper 
 [domain_titles] has different index_analyzer]}]","status":400}


 Thanks for the help.

>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email toelasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/3ffb1a0c-c1eb-4c2c-90f3-7d579aab7a41%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c1757b54-fb8c-4438-a4b4-70d500de1690%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Get the last entry from each of multiple types

[Jingzhao Ou]

Hi, David, 

Thanks a lot for your prompt help. I got both approaches working, which is 
very exciting. I prefer the top_hits aggregation approach. The msearch 
approach is not accepting normal JSON payloads, which makes things a bit 
harder for processing in Javascript. 

My query payload is shown below. Works like a champ! 

Best regards,
Jingzhao

{
"size": 0,
"query": {
"bool": {
"should": [ 
{ "term" : { "_type": "0001" } },
{ "term" : { "_type": "0002" } }
]
}
},
"sort": {
"id": "desc"
},
"aggs": {
"top-tags": {
"terms": {
"field": "_type"
},
"aggs": {
"top_tag_hits": {
"top_hits": { "size" : 1 }
}
}
}
}
}


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a9ab08a2-cfda-4a3a-a3e9-013527505c09%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New version of Kibana in the works?

[Doug Nelson]

Thanks for the update Rashid.

It would be really great to get a look at the early bits.  We are just 
starting to use es and so backwards compatibility would not be an issue.

Thanks

Doug

On Wednesday, September 17, 2014 5:37:59 PM UTC-5, Rashid Khan wrote:

> Unfortunately I can’t give you an ETA other than soon ;-)
>
> Initially backwards compatibility will not be available as there have been 
> a large number of core changes. We’re looking at providing a compatibility 
> layer to ease the transition, but there are some challenges there.
>
>
> On Thu, Aug 14, 2014 at 10:22 AM, Antonio Augusto Santos  > wrote:
>
>> Thanks a lot for the update Rashid.
>> I know it might be too much but... any dates we can mark on out calendar? 
>> ;)
>>
>> On Thursday, August 14, 2014 1:04:24 PM UTC-3, Rashid Khan wrote:
>>>
>>> Yes there is a new version in the works. 
>>>
>>> Elasticsearch has grown considerably in the last 18 months. With 
>>> Elasticsearch 1.0, and the introduction of aggregations, we quickly 
>>> realized we needed a better platform for building more powerful interfaces. 
>>> So for the last few months we've been head down, working to polish Kibana 4 
>>> to the point where we can collect feedback and contributions.
>>>
>>> We've left lots of pull requests open for a couple reasons: For the 
>>> community to pull into their own fork if needed, and for our own 
>>> inspiration. When someone submits code it says "I really want this!". Even 
>>> if the code doesn't get merged, we take it very seriously. Once we push out 
>>> the first preview of Kibana 4 we'll be cleaning up lots of pull requests 
>>> for features we're implemented.
>>>
>>> As for the features you're looking for: User separation and event 
>>> correlation are things we're looking to do in Elasticsearch. Multi-value 
>>> metric aggregations are a feature I'm looking forward to in Kibana. It 
>>> will, of course, be open source.
>>>
>>> On Tuesday, August 12, 2014 9:44:55 AM UTC-7, Antonio Augusto Santos 
>>> wrote:

 This one is for the devs, and Rashid in special: there is any new 
 version of Kibana in the works?
 I'm asking this because I'm about to start a project in my company for 
 log management, and there are some requisites to it (user separation, 
 event 
 correlation, histogram to compare two values, and so on).

 So, any changes of these functionalities landing on Kibana 4.0? ;)

>>>-- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/elasticsearch/I7um1mX4GSk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/0a3bb3d3-c2dc-418b-bce0-5d376065fb7f%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/363514af-1c1a-4ad8-99f9-f700995a302e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Disable writes over http

[joergpra...@gmail.com]

You can filter out HTTP PUT (and DELETE) and get a pretty good approach to
not accidentally remove or overwrite anything (due to REST semantics)

Jörg

On Fri, Sep 19, 2014 at 5:41 PM, Marie Jacob  wrote:

> I'm sure that this has been asked before on the forum, but I couldn't find
> an answer specifically for this one:
>
> Is there any way at all to disable writes over http for elasticsearch?
> It's very easy for people to accidentally create indexes that they didn't
> mean to create.
> If there is no way, I think this would be a GREAT option to add (something
> like http.writes_enabled?) ... it seems to be the most basic access control
> that would be useful for many use cases.
>
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5c472d3f-6fe6-40b7-afcf-366430a953d6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHO-26RspwnTYAaFAUA6T12EhjGG39JJtsKTWsT%2BA3a6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Facing error and warning messages with ELK stack

[shriyansh jain]

which configuration's you will be interested to look at.?

-Shriyansh

On Friday, September 19, 2014 2:17:16 PM UTC-7, Mark Walkom wrote:
>
> It looks like networking issues, lots of connection reset/closed/timeout.
>
> It might help if you can out your configs into a pastebin too.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
> On 20 September 2014 04:33, shriyansh jain  > wrote:
>
>> I have a setup of ELK stack with redis as a broker and a cluster of 2 
>> Elasticsearch Nodes. The stack was running good and suddenly I started 
>> facing the following error and warning messages at different levels of the 
>> stack.
>>
>> http://pastebin.com/eG7p0PCc
>>
>> If I restart the redis broker, Logstash and the Elasticsearch node 
>> showing error message, the stack comes back, with Redis-broker performing 
>> well, able to allocate memory but I have been seeing the same error 
>> messages for E.S and Logstash after the restart. Because of which the 
>> redis-broker start buffering the logs in memory and eventually goes out of 
>> memory as E.S and logstash stop indexing the data.
>> I am trying out figure out what might be the possible cause of this, any 
>> help will be really appreciated. Please let me know if there is any 
>> confusion understanding the situation.
>>
>> Thank you,
>> Shriyansh
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/99ac902f-c183-4700-9d32-9d23ea0d7c3c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ca66b25e-77ab-4339-820e-74e25f5e9e7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Facing error and warning messages with ELK stack

[shriyansh jain]

Which configuration you will be interested to loo at.?

-Shriyansh

On Friday, September 19, 2014 2:17:16 PM UTC-7, Mark Walkom wrote:
>
> It looks like networking issues, lots of connection reset/closed/timeout.
>
> It might help if you can out your configs into a pastebin too.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
> On 20 September 2014 04:33, shriyansh jain  > wrote:
>
>> I have a setup of ELK stack with redis as a broker and a cluster of 2 
>> Elasticsearch Nodes. The stack was running good and suddenly I started 
>> facing the following error and warning messages at different levels of the 
>> stack.
>>
>> http://pastebin.com/eG7p0PCc
>>
>> If I restart the redis broker, Logstash and the Elasticsearch node 
>> showing error message, the stack comes back, with Redis-broker performing 
>> well, able to allocate memory but I have been seeing the same error 
>> messages for E.S and Logstash after the restart. Because of which the 
>> redis-broker start buffering the logs in memory and eventually goes out of 
>> memory as E.S and logstash stop indexing the data.
>> I am trying out figure out what might be the possible cause of this, any 
>> help will be really appreciated. Please let me know if there is any 
>> confusion understanding the situation.
>>
>> Thank you,
>> Shriyansh
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/99ac902f-c183-4700-9d32-9d23ea0d7c3c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/624cd024-cc69-4eba-8bf9-d05b21f8d41a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: NPE in SignificanceHeuristicStreams.read while deserializing response (SearchResponse)

[Felipe Hummel]

More information. All 5 ES nodes are on 1.3.2 (checked with curl 
localhost:9200/) with java 1.7.0_65. Client machine is also on 1.3.2 
and 1.7.0_65


On Friday, September 19, 2014 6:21:06 PM UTC-3, Felipe Hummel wrote:
>
> I missed a part of the error message:
>
> [WARN] 2014-09-19 20:29:13.176 o.e.t.netty - [Sigyn] Message not fully 
>> read (response) for [61] handler 
>> org.elasticsearch.action.TransportActionNodeProxy$1@2e6201d0, error 
>> [false], resetting
>
>
> On Friday, September 19, 2014 5:58:15 PM UTC-3, Felipe Hummel wrote:
>>
>> Hey guys, I’m getting NullPointerException while using a 
>> *significant_terms* aggregation. It happens in this line: 
>>
>>
>> org.elasticsearch.search.aggregations.bucket.significant.heuristics.SignificanceHeuristicStreams.read(SignificanceHeuristicStreams.java:38)
>>
>> The error is in the deserialization: *Failed to deserialize response of 
>> type [org.elasticsearch.action.search.SearchResponse]*
>> I’m using the Java API. I just printed the request and manually did it 
>> through the REST API and everything went fine. It happens only when using 
>> the Java API.
>>
>> I'm using ES 1.3.2.
>>
>> *The printed search request:*
>>
>>> {
>>>   "from" : 0,
>>>   "size" : 6,
>>>   "timeout" : 3,
>>>   "query" : {
>>> "filtered" : {
>>>   "query" : {
>>> "query_string" : {
>>>   "query" : "ayrton senna",
>>>   "fields" : [ "title^2.0", "description" ],
>>>   "default_operator" : "and"
>>> }
>>>   },
>>>   "filter" : {
>>> "bool" : {
>>>   "must" : [ {
>>> "range" : {
>>>   "created_at" : {
>>> "from" : null,
>>> "to" : "2014-09-19T20:28:30.000Z",
>>> "include_lower" : true,
>>> "include_upper" : true
>>>   },
>>>   "_cache" : true
>>> }
>>>   }, {
>>> "range" : {
>>>   "published_at" : {
>>> "from" : null,
>>> "to" : "2014-09-19T20:28:30.000Z",
>>> "include_lower" : true,
>>> "include_upper" : true
>>>   },
>>>   "_cache" : true
>>> }
>>>   }, {
>>> "range" : {
>>>   "published_at" : {
>>> "from" : "2014-08-20T20:28:30.000Z",
>>> "to" : "2014-09-19T20:28:30.000Z",
>>> "include_lower" : true,
>>> "include_upper" : true
>>>   },
>>>   "_cache" : false
>>> }
>>>   } ]
>>> }
>>>   }
>>> }
>>>   },
>>>   "fields" : [ ],
>>>   "aggregations" : {
>>> "topics" : {
>>>   "significant_terms" : {
>>> "field" : "topic_ids",
>>> "size" : 20
>>>   }
>>> }
>>>   }
>>> }
>>
>>
>> *The complete error stacktrace:*
>>
>> [ERROR] 2014-09-19 20:29:13.177 c.b.s.SearchServlet - 
>> org.elasticsearch.transport.TransportSerializationException: Failed to 
>> deserialize response of type 
>> [org.elasticsearch.action.search.SearchResponse]
>> org.elasticsearch.transport.TransportSerializationException: Failed to 
>> deserialize response of type 
>> [org.elasticsearch.action.search.SearchResponse]
>> at 
>> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:152)
>> at 
>> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
>> at 
>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> at 
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at 
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> at 
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at 
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>> at 
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>> at 
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>> at 
>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> at 
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at 
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at 
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at 
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

Re: Configuring kibana/elasticsearc for SSL

[Mark Walkom]

There is a few good results here
https://www.google.com.au/?gws_rd=ssl#q=kibana+https+apache

Check out
http://blog.stevenmeyer.co.uk/2014/02/securing-kibana-and-elasticsearch-with-https-ssl.html
for example

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 20 September 2014 02:36,  wrote:

> Has anyone configured kibana/elasticsearch to use HTTPS? I'm new to it and
> was wondering if there are any goods tuts out there? I'm using apache as my
> webserver and have ssl enabled. If I try to connect to ES using https I get
> a message in Kibana that says it was unable to connect to ES at
> x.x.x.x:9200.
>
> Looking for some guidance. Any help is appreciate :)
>
> Larry
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a681f4e3-65f8-4f5d-82e6-8358d8c36a58%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624a8WLkHMqDZFFXA%2BKfvQVTTnynsbfAK_wzACzR-kCTv-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: NPE in SignificanceHeuristicStreams.read while deserializing response (SearchResponse)

[Felipe Hummel]

I missed a part of the error message:

[WARN] 2014-09-19 20:29:13.176 o.e.t.netty - [Sigyn] Message not fully read 
> (response) for [61] handler 
> org.elasticsearch.action.TransportActionNodeProxy$1@2e6201d0, error 
> [false], resetting


On Friday, September 19, 2014 5:58:15 PM UTC-3, Felipe Hummel wrote:
>
> Hey guys, I’m getting NullPointerException while using a 
> *significant_terms* aggregation. It happens in this line: 
>
>
> org.elasticsearch.search.aggregations.bucket.significant.heuristics.SignificanceHeuristicStreams.read(SignificanceHeuristicStreams.java:38)
>
> The error is in the deserialization: *Failed to deserialize response of 
> type [org.elasticsearch.action.search.SearchResponse]*
> I’m using the Java API. I just printed the request and manually did it 
> through the REST API and everything went fine. It happens only when using 
> the Java API.
>
> I'm using ES 1.3.2.
>
> *The printed search request:*
>
>> {
>>   "from" : 0,
>>   "size" : 6,
>>   "timeout" : 3,
>>   "query" : {
>> "filtered" : {
>>   "query" : {
>> "query_string" : {
>>   "query" : "ayrton senna",
>>   "fields" : [ "title^2.0", "description" ],
>>   "default_operator" : "and"
>> }
>>   },
>>   "filter" : {
>> "bool" : {
>>   "must" : [ {
>> "range" : {
>>   "created_at" : {
>> "from" : null,
>> "to" : "2014-09-19T20:28:30.000Z",
>> "include_lower" : true,
>> "include_upper" : true
>>   },
>>   "_cache" : true
>> }
>>   }, {
>> "range" : {
>>   "published_at" : {
>> "from" : null,
>> "to" : "2014-09-19T20:28:30.000Z",
>> "include_lower" : true,
>> "include_upper" : true
>>   },
>>   "_cache" : true
>> }
>>   }, {
>> "range" : {
>>   "published_at" : {
>> "from" : "2014-08-20T20:28:30.000Z",
>> "to" : "2014-09-19T20:28:30.000Z",
>> "include_lower" : true,
>> "include_upper" : true
>>   },
>>   "_cache" : false
>> }
>>   } ]
>> }
>>   }
>> }
>>   },
>>   "fields" : [ ],
>>   "aggregations" : {
>> "topics" : {
>>   "significant_terms" : {
>> "field" : "topic_ids",
>> "size" : 20
>>   }
>> }
>>   }
>> }
>
>
> *The complete error stacktrace:*
>
> [ERROR] 2014-09-19 20:29:13.177 c.b.s.SearchServlet - 
> org.elasticsearch.transport.TransportSerializationException: Failed to 
> deserialize response of type 
> [org.elasticsearch.action.search.SearchResponse]
> org.elasticsearch.transport.TransportSerializationException: Failed to 
> deserialize response of type 
> [org.elasticsearch.action.search.SearchResponse]
> at 
> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:152)
> at 
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
> at 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at 
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at 
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at 
> org.elasticse

Re: Facing error and warning messages with ELK stack

[Mark Walkom]

It looks like networking issues, lots of connection reset/closed/timeout.

It might help if you can out your configs into a pastebin too.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 20 September 2014 04:33, shriyansh jain  wrote:

> I have a setup of ELK stack with redis as a broker and a cluster of 2
> Elasticsearch Nodes. The stack was running good and suddenly I started
> facing the following error and warning messages at different levels of the
> stack.
>
> http://pastebin.com/eG7p0PCc
>
> If I restart the redis broker, Logstash and the Elasticsearch node showing
> error message, the stack comes back, with Redis-broker performing well,
> able to allocate memory but I have been seeing the same error messages for
> E.S and Logstash after the restart. Because of which the redis-broker start
> buffering the logs in memory and eventually goes out of memory as E.S and
> logstash stop indexing the data.
> I am trying out figure out what might be the possible cause of this, any
> help will be really appreciated. Please let me know if there is any
> confusion understanding the situation.
>
> Thank you,
> Shriyansh
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/99ac902f-c183-4700-9d32-9d23ea0d7c3c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YD7tqLU2%2BuufpvhO%2BgzSZA%3DLyMFods1o5g2JxHzr2%2B-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Disable writes over http

[Mark Walkom]

You can disable http entirely, that's it
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html#_disable_http

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 20 September 2014 01:41, Marie Jacob  wrote:

> I'm sure that this has been asked before on the forum, but I couldn't find
> an answer specifically for this one:
>
> Is there any way at all to disable writes over http for elasticsearch?
> It's very easy for people to accidentally create indexes that they didn't
> mean to create.
> If there is no way, I think this would be a GREAT option to add (something
> like http.writes_enabled?) ... it seems to be the most basic access control
> that would be useful for many use cases.
>
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5c472d3f-6fe6-40b7-afcf-366430a953d6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YJFe%2B8Ff_V1bzxKK%3DOE5eTBZOxsHeK%3DoodKWEBXk9ehw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

NPE in SignificanceHeuristicStreams.read while deserializing response (SearchResponse)

[Felipe Hummel]

 

Hey guys, I’m getting NullPointerException while using a *significant_terms* 
aggregation. It happens in this line: 

org.elasticsearch.search.aggregations.bucket.significant.heuristics.SignificanceHeuristicStreams.read(SignificanceHeuristicStreams.java:38)

The error is in the deserialization: *Failed to deserialize response of 
type [org.elasticsearch.action.search.SearchResponse]*
I’m using the Java API. I just printed the request and manually did it 
through the REST API and everything went fine. It happens only when using 
the Java API.

I'm using ES 1.3.2.

*The printed search request:*

> {
>   "from" : 0,
>   "size" : 6,
>   "timeout" : 3,
>   "query" : {
> "filtered" : {
>   "query" : {
> "query_string" : {
>   "query" : "ayrton senna",
>   "fields" : [ "title^2.0", "description" ],
>   "default_operator" : "and"
> }
>   },
>   "filter" : {
> "bool" : {
>   "must" : [ {
> "range" : {
>   "created_at" : {
> "from" : null,
> "to" : "2014-09-19T20:28:30.000Z",
> "include_lower" : true,
> "include_upper" : true
>   },
>   "_cache" : true
> }
>   }, {
> "range" : {
>   "published_at" : {
> "from" : null,
> "to" : "2014-09-19T20:28:30.000Z",
> "include_lower" : true,
> "include_upper" : true
>   },
>   "_cache" : true
> }
>   }, {
> "range" : {
>   "published_at" : {
> "from" : "2014-08-20T20:28:30.000Z",
> "to" : "2014-09-19T20:28:30.000Z",
> "include_lower" : true,
> "include_upper" : true
>   },
>   "_cache" : false
> }
>   } ]
> }
>   }
> }
>   },
>   "fields" : [ ],
>   "aggregations" : {
> "topics" : {
>   "significant_terms" : {
> "field" : "topic_ids",
> "size" : 20
>   }
> }
>   }
> }


*The complete error stacktrace:*

[ERROR] 2014-09-19 20:29:13.177 c.b.s.SearchServlet - 
org.elasticsearch.transport.TransportSerializationException: Failed to 
deserialize response of type 
[org.elasticsearch.action.search.SearchResponse]
org.elasticsearch.transport.TransportSerializationException: Failed to 
deserialize response of type 
[org.elasticsearch.action.search.SearchResponse]
at 
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:152)
at 
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
at 
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at 
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at 
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at 
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at 
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at 
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at 
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at 
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at 
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at 
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at 
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused 

Re: Get the last entry from each of multiple types

[David Pilato]

May be multi search could help in that case?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-multi-search.html#search-multi-search

Or may be a terms aggregation on _type field (you'll need to index it) and a 
top hits sub aggregation: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#search-aggregations-metrics-top-hits-aggregation

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 22:29:07, Jingzhao Ou (jingzhao...@gmail.com) a écrit:

Hi, all, 

To get the last entry from two different types, I am doing

GET localhost:9200/index/type1
{
    size: 1,
    sort: { id: 'desc' },
}

GET localhost:9200/index/type2
{
    size: 1,
    sort: { id: 'desc' },
}

For more efficient queries over multiple types, I want to combine the two 
queries into one by doing

GET localhost:9200/index/type1,type2
{
    size: 2,
    sort: { id: 'desc' },
}

However, this only gives me the last two samples from either type1 or type 2. 
If the last two samples both belong to type1, the combined query only returns 
two type1 samples, which is different from the two separate GET queries. 

Can any one give me some hints how to get this done? I appreciate your help! 

Best regards,
Jingzhao

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/64c5323f-9a5c-4c79-9c4e-8e8cad50607b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541c93fa.41b71efb.23d3%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Get the last entry from each of multiple types

[Jingzhao Ou]

Hi, all, 

To get the last entry from two different types, I am doing

GET localhost:9200/index/type1
{
size: 1,
sort: { id: 'desc' },
}

GET localhost:9200/index/type2
{
size: 1,
sort: { id: 'desc' },
}

For more efficient queries over multiple types, I want to combine the two 
queries into one by doing

GET localhost:9200/index/type1,type2
{
size: 2,
sort: { id: 'desc' },
}

However, this only gives me the last two samples from either type1 or type 
2. If the last two samples both belong to type1, the combined query only 
returns two type1 samples, which is different from the two separate GET 
queries. 

Can any one give me some hints how to get this done? I appreciate your 
help! 

Best regards,
Jingzhao

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/64c5323f-9a5c-4c79-9c4e-8e8cad50607b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom Collector using a plugin

[Ivan Brusic]

You basically want to create your own aggregation, which are basically
collectors at the Lucene level. Look at existing plugins which provide
custom aggregation.

Basically, elasticsearch uses a scatter-gather/map-reduce model for
distributed collections.

-- 
Ivan
On Sep 18, 2014 12:56 AM, "tim glabisch"  wrote:

> Hello,
>
> i am just looking for an entry point for a custom (lucene) collector.
> is it possible to use a custom collector at all?
>
> what classes do i have to implement to run the collector in a distributed
> way?
>
> thanks a lot,
> tim
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/91c16446-0e5e-4f20-99a9-83a49a46b0bd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCYx%2BFxN93GJxdd1x_8g-Ftk5qd2sZPRjRbe%3D5%3DWe%3DC9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Facing error and warning messages with ELK stack

[shriyansh jain]

I have a setup of ELK stack with redis as a broker and a cluster of 2 
Elasticsearch Nodes. The stack was running good and suddenly I started 
facing the following error and warning messages at different levels of the 
stack.

http://pastebin.com/eG7p0PCc

If I restart the redis broker, Logstash and the Elasticsearch node showing 
error message, the stack comes back, with Redis-broker performing well, 
able to allocate memory but I have been seeing the same error messages for 
E.S and Logstash after the restart. Because of which the redis-broker start 
buffering the logs in memory and eventually goes out of memory as E.S and 
logstash stop indexing the data.
I am trying out figure out what might be the possible cause of this, any 
help will be really appreciated. Please let me know if there is any 
confusion understanding the situation.

Thank you,
Shriyansh

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/99ac902f-c183-4700-9d32-9d23ea0d7c3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Range Aggregations: response bucket order?

[James Addison]

   

I would have thought that range aggregations return the bucket list in the 
order the range agg is listed, but I'm not seeing that result. (ES 1.3.2) 
Is there a way to enforce original ordering that I'm missing?
  My range aggregation:

"day_of_week_range": {
"range": {
"field": "sessions.events.days_of_week",
"ranges": [
{"from":0, "to":5, "key": "Weekdays"},
{"from":5, "to":7, "key": "Weekends"},
{"from":6, "to":6, "key": "Sunday"},
{"from":0, "to":1, "key": "Monday"},
{"from":1, "to":2, "key": "Tuesday"},
{"from":2, "to":3, "key": "Wednesday"},
{"from":3, "to":4, "key": "Thursday"},
{"from":4, "to":5, "key": "Friday"},
{"from":5, "to":6, "key": "Saturday"}
]
}
}

And the response (Note that Weekdays is 2nd in the response, which is why 
I'm posting):

"day_of_week_range": {
 "buckets": [
{
   "key": "Monday",
   "from": 0,
   "from_as_string": "0.0",
   "to": 1,
   "to_as_string": "1.0",
   "doc_count": 11
},
{
   "key": "Weekdays",
   "from": 0,
   "from_as_string": "0.0",
   "to": 5,
   "to_as_string": "5.0",
   "doc_count": 40
},
{
   "key": "Tuesday",
   "from": 1,
   "from_as_string": "1.0",
   "to": 2,
   "to_as_string": "2.0",
   "doc_count": 11
},
{
   "key": "Wednesday",
   "from": 2,
   "from_as_string": "2.0",
   "to": 3,
   "to_as_string": "3.0",
   "doc_count": 10
},
{
   "key": "Thursday",
   "from": 3,
   "from_as_string": "3.0",
   "to": 4,
   "to_as_string": "4.0",
   "doc_count": 17
},
{
   "key": "Friday",
   "from": 4,
   "from_as_string": "4.0",
   "to": 5,
   "to_as_string": "5.0",
   "doc_count": 15
},
{
   "key": "Saturday",
   "from": 5,
   "from_as_string": "5.0",
   "to": 6,
   "to_as_string": "6.0",
   "doc_count": 28
},
{
   "key": "Weekends",
   "from": 5,
   "from_as_string": "5.0",
   "to": 7,
   "to_as_string": "7.0",
   "doc_count": 37
},
{
   "key": "Sunday",
   "from": 6,
   "from_as_string": "6.0",
   "to": 6,
   "to_as_string": "6.0",
   "doc_count": 0
}
 ]
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a245439-0fd8-4dd3-888b-43ec882c0e16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Snapshots don't get compressed

[Igor Motov]

There were two reasons for not enabling compression on data files. First of 
all, the way "chunking" in snapshot/restore API was implemented didn't 
allow simple implementation of compression on data files. Moreover, the 
data files are already compressed to a certain degree. In my tests I was 
getting about 20% compressions rates on index data with recent versions of 
Elasticsearch (we have a limitation that we can compress only one file at a 
time). So, difficulties with implementation together with limit benefits 
made this feature not very compelling. After recent refactoring of the 
storage code, it's now much easier to add this feature in if it makes 
sense. However, I am really curious how you've got such great compression 
rates. Which version of Elasticsearch were these indices created with? Did 
you upgrade from older version of elasticsearch recently and most of your 
data files are created with older versions of Lucene?

On Thursday, September 18, 2014 10:53:58 AM UTC-4, Domonkos Tomcsanyi wrote:
>
>  thank you again, I'll keep an eye on this issue.
>
> Domonkos
>
> 2014.09.18. 16:38 keltezéssel, David Pilato írta:
>  
> I don't know. I think this could happen in the future but unsure though.
> May be Igor could answer this?
>
>  Here is a related doc PR: 
> https://github.com/elasticsearch/elasticsearch/pull/7654 
> 
>
>  
>  -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos (tomcsan...@modit.hu) 
> a écrit:
>
>  Thank you for the answer, but may I know why? What is the reason behind 
> this? 
>
> thanks, 
> Domonkos 
>
> 2014.09.18. 16:15 keltezéssel, David Pilato írta: 
> > Only metadata are compressed. 
> > 
> > -- 
> > David ;-) 
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs 
> > 
> >> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos"  
>  a écrit : 
> >> 
> >> Hi Everyone, 
> >> 
> >> I just started experimenting with the cool snapshot feature of ES 
> (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository 
> on a mounted NFS storage, using only the default options (compression 
> turned on). I checked it using curl: 
> >> 
> >> user@myserver:~# curl -XGET '
> http://IP_ADDRESS:9200/_snapshot/logBack?pretty' 
> >> { 
> >> "logBack" : { 
> >> "type" : "fs", 
> >> "settings" : { 
> >> "compress" : "true", 
> >> "location" : "/es_snapshots" 
> >> } 
> >> } 
> >> } 
> >> 
> >> So, after that I used curator to create a snapshot of some older 
> indices. The process finished after some minutes, so I decided to have a 
> look at the files it created. It turned out that the snapshot's files take 
> up exactly as much space as the indices did originally while they were in 
> the cluster, so no compression happened at all. This is kind of a problem 
> for me, because I assumed that compression will greatly reduce the size of 
> the indices I put in a snapshot. So is there anything I'm doing wrong? 
> >> 
> >> Thank you, 
> >> Domonkos 
> >> 
> >> -- 
> >> You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> >> To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearch+unsubscr...@googlegroups.com. 
> >> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu
> .
> >> For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group. 
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com. 
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu
> .
> For more options, visit https://groups.google.com/d/optout. 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.c

How elasticsearch.yml are coordinated between cluster nodes?

[Konstantin Erman]

I would expect this question to be popular, but still cannot google the 
answer. 

If I have multiple ES nodes in the cluster, each having its own 
configuration file (elasticsearch.yml) - what happens if some settings in 
those files go out of sync? For instance, index creation config? Which 
settings are going to take precedence? My gut feeling is that it would be 
settings on the current master node, but I would like a confirmation.

Konstantin
 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f763829e-ab50-47d3-95cb-e0ef36975396%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Configuring kibana/elasticsearc for SSL

[larrychuzzy]

Has anyone configured kibana/elasticsearch to use HTTPS? I'm new to it and 
was wondering if there are any goods tuts out there? I'm using apache as my 
webserver and have ssl enabled. If I try to connect to ES using https I get 
a message in Kibana that says it was unable to connect to ES at 
x.x.x.x:9200.

Looking for some guidance. Any help is appreciate :)

Larry

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a681f4e3-65f8-4f5d-82e6-8358d8c36a58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Disable writes over http

[Marie Jacob]

I'm sure that this has been asked before on the forum, but I couldn't find 
an answer specifically for this one:

Is there any way at all to disable writes over http for elasticsearch? It's 
very easy for people to accidentally create indexes that they didn't mean 
to create. 
If there is no way, I think this would be a GREAT option to add (something 
like http.writes_enabled?) ... it seems to be the most basic access control 
that would be useful for many use cases.





-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5c472d3f-6fe6-40b7-afcf-366430a953d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch 1.4.0 release data?

[Tom Miller]

I'm in the same boat as Dan. Desperate for child aggregation!

Looks like the label has changed 
too: https://github.com/elasticsearch/elasticsearch/labels/v1.4.0.Beta1

Tom.


On Wednesday, September 10, 2014 6:02:27 PM UTC+1, Ivan Brusic wrote:
>
> I think this release might be their biggest one since 1.0. Lots of big 
> changes including a change in the consensus algorithm. It might take time, 
> but that is only a guess.
>
> -- 
> Ivan
>
> On Wed, Sep 10, 2014 at 2:57 AM, joerg...@gmail.com  <
> joerg...@gmail.com > wrote:
>
>> I use the Github issue tracker to watch the progress of the fabulous ES 
>> dev team
>>
>> https://github.com/elasticsearch/elasticsearch/labels/v1.4.0
>>
>> Today: 20 issues left, 4 blockers. Looks like it will still take some 
>> days.
>>
>> Jörg
>>
>>
>> On Wed, Sep 10, 2014 at 11:39 AM, Dan Tuffery > > wrote:
>>
>>> Is there are release date scheduled for ES 1.4.0? I need the child 
>>> aggregation for the project I'm working on at the moment.
>>>
>>> https://github.com/elasticsearch/elasticsearch/pull/6936
>>>
>>> Dan
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/0238c4fd-a702-4fca-8bcc-3dab6d71bc6f%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP%2Bq64F5FVAfjym9SvO6RM5dHOzuJMe7L8xFL4ekut%3Dg%40mail.gmail.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/43241323-6d39-4549-836a-a629694a2523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Regexp group result

[Michaël Guiral]

Hi,

Is it possible to get regexp group result into regexp request in 
eleasticsearch?

For example, if i make this request with a regexp group, I don't have the 
group value under request response. So can I have it or not ? 
{
  "query":{
"regexp":{
  "path": "*Prods/([^/]+)*",
  "flags": "all"
}
  },
  from:0,
  size:10
}

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/73b09e26-4053-4a42-9898-4eafa0a213f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with word-separators in bool search with standard tokenizer

[Ankush Jhalani]

On other hand, If I use a single query_string instead of bool of terms it 
works. Does ES/lucene determines not to use the word-separators by looking 
at the definition of the fields. 

On Friday, September 19, 2014 11:05:59 AM UTC-4, Ankush Jhalani wrote:
>
> In our search we have configured text with 2 analyzers, english and 
> standard so we can match phrases on the standard-analyzer. We break the 
> keywords by space, and create a bool query for each word. 
>
> This is working fine for all cases except where the query has standard 
> word-separators like & (ampersand), ; (semi-colon), etc.  As 
> word-separators are stripped in index by analyzer, searching for them 
> returns 0 results. Gist. 
> https://gist.github.com/ajhalani/3def3ea7caec5cd58490
>
> I don't want to use a whitespace analyzer because we do actually want to 
> ignore word separators. I was thinking about hacky workarounds like 
> removing all standalone non-alphanumeric characters, or moving them in 
> "should" instead of default "must" (in case we do have analyzers in future 
> that are whitespace). 
>
> Thanks in advance.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4b205133-eecd-490a-a028-9a53a3230973%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Problem with word-separators in bool search with standard tokenizer

[Ankush Jhalani]

In our search we have configured text with 2 analyzers, english and 
standard so we can match phrases on the standard-analyzer. We break the 
keywords by space, and create a bool query for each word. 

This is working fine for all cases except where the query has standard 
word-separators like & (ampersand), ; (semi-colon), etc.  As 
word-separators are stripped in index by analyzer, searching for them 
returns 0 results. Gist. 
https://gist.github.com/ajhalani/3def3ea7caec5cd58490

I don't want to use a whitespace analyzer because we do actually want to 
ignore word separators. I was thinking about hacky workarounds like 
removing all standalone non-alphanumeric characters, or moving them in 
"should" instead of default "must" (in case we do have analyzers in future 
that are whitespace). 

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f2abbc24-52d5-4567-afa3-66610956ce0b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with zipcode

[Hari Rajaonarifetra]

 

> After populating index with this mapping, I just use 
> http://localhost:9200/_plugin/head/ for testing query.
>
>
But when I try to find a city with zipcode:03000 (for example), there is no 
result. I check the index with the head navigator and I see that all 
zipcode beginning  with "0" is cut and indexed without "0" (03000 => 3000).

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8ee8a161-11e4-4394-a3c2-5ce0f768894a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Calculating rolling average using aggregations

[Michaël Gallego]

+1 for this feature!

What I need is pretty similar: calculate rolling sum, so for each day, I 
need to sum the previous 30 days (on each point). Oracle and Postgre make 
this very easy with aggregation function (and they can take advantage of 
very interesting optimization for sum, as each point is actually the sum of 
previous day + sum of elements in the given day). Actually, the simplest 
way I've found is asking for more data (so if I want rolling sum for 30 
days, I ask for 60 days, and do the sum myself). But this is quite 
inefficient.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b72eea67-d8cb-4ac3-ac4f-4e984183731b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with zipcode

[Hari Rajaonarifetra]

There was something wrong in what I did ? 

Thanks a lot,

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a650a6b1-4454-49ec-8dd2-28e8fa59ec81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with zipcode

[David Pilato]

Could you GIST a full SENSE script which helps to reproduce your issue?



-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 14:06:04, Hari Rajaonarifetra (rhar...@gmail.com) a 
écrit:

Thanks for your answer. 

GET /ville/_mapping gives this : 

{ "my_project": { "mappings": { "ville": { "_meta": { "model": 
"MyProject\ReferenceBundle\Entity\Ville" }, "properties": { "ville_cp": { 
"type": "string", "index": "not_analyzed", "store": true }, "ville_id": { 
"type": "long" }, "ville_nom": { "type": "string", "store": true, 
"index_analyzer": "cities_index_analyzer", "search_analyzer": 
"cities_search_analyzer" } } } } } }
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4ff68968-2d01-41cb-96be-deb51ebc4119%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541c1cb9.643c9869.1f5f%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with zipcode

[Hari Rajaonarifetra]

>
> Thanks for your answer. 
>
>
GET /ville/_mapping gives this : 


   1. {
  "my_project":
  {
  "mappings":
  {
  "ville":
  {
  "_meta":
  {
  "model": "MyProject\ReferenceBundle\Entity\Ville"
  },
  "properties":
  {
  "ville_cp":
  {
  "type": "string",
  "index": "not_analyzed",
  "store": true
  },
  "ville_id":
  {
  "type": "long"
  },
  "ville_nom":
  {
  "type": "string",
  "store": true,
  "index_analyzer": "cities_index_analyzer",
  "search_analyzer": "cities_search_analyzer"
  }
  }
  }
  }
  }
   }
   

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4ff68968-2d01-41cb-96be-deb51ebc4119%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: The Max Number of Filelds ES Can Support

[joergpra...@gmail.com]

A search is independent from the number of fields as long as you do not
search over this number of fields.

You can create as much fields as your memory und resources will let you do
so.

Jörg

On Fri, Sep 19, 2014 at 9:53 AM,  wrote:

> HI,
>
>  In my case , there are about five thousand fields, Can ES support
> this?  How do the number of fields affect the speed of searching?
>
> Thanks.
> terrs
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a30b4502-9a50-4f84-bee1-c5519e47c3bd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQDjNumnwABzXYUyJ1dVuTynPcut2k_SUszmRe-WuMPw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index design for very dynamic Form/Fields

[Michael Chen]

Let me make the question more clear. The challenge we have now, is how to
index a EAV[1] model database.

Let's take Google Form as an example. Every user can create a form. They
can choose from various field types including text, number, choice etc.
They construct one form like this:

Form 1: a survery
  - field_1: type=text
  - field_2: type=number
  - field_3: type=choice

And people submit data entry into this form with data like:

{
  field_1: "hello",
  field_2: 20,
  field_3: ["red"]
}

And you can imagine that all this data entries saved into one single mongo
collection "entries".

Well, the second user might create another form like this:

Form 2: a questionare
  - field_1: type=number
  - field_2: type=text
  - field_3: type=number
  - field_4: type=text

the data submission might like this:

{
  field_1: 100,
  field_2: "hello questionare",
  field_3: 20,
  field_4: "this is my answer"
}

Indexing the second data entry while we have the first one in ES will throw
NumberFormatException because field_2 was guessed by ES it should be
number. Then the transforming all value into string make sense but...

Any thoughts?

[1]EAV: Entity–attribute–value model,
http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model

On Fri, Sep 19, 2014 at 6:01 PM, Michael Chen  wrote:

> We cannot guarantee that field_1 is always address. In Form 1, field_1
> might be address while in another form it might be string or number
> whatever. Thinking about designing the storage for Google Forms and it's
> data entries.
>
> Re "you could force each field to be a String and do the transformation
> at a client level."
>
> Forcing means serialize all data into a string right? In the example JSON
> mentioned in previous email, it will transformed to something like
>
> { field_1: "\{country: \"US\", province: \"CA\", city: \"New York\",
> address: \"Street Address\"\}"  }
>
> Then not able to do the aggregation.
>
> On Fri, Sep 19, 2014 at 5:50 PM, David Pilato  wrote:
>
>> I don't get it.
>>
>> If field_1.country is a String why you can not aggregate on it?
>>
>> --
>> *David Pilato* | Technical Advocate | *elasticsearch.com
>> *
>> david.pil...@elasticsearch.com
>> @dadoonet  | @elasticsearchfr
>>  | @scrutmydocs
>> 
>> 
>>
>>
>>
>> Le 19 septembre 2014 à 08:27:19, Michael Chen (mechil...@gmail.com) a
>> écrit:
>>
>> Thanks David. Based on the system behavior, having all type as string is
>> fine for queries. But for the aggregation level it might be trouble. For
>> example a type of address is a complex JSON object:
>>
>> { field_1: { country: "US", province: "CA", city: "New York", address:
>> "Street Address"} }
>>
>> If we transform this type into any form of string, and trying to
>> aggregate based on country/state, it will be VERY hard even not possible.
>>
>> On Fri, Sep 19, 2014 at 2:15 PM, David Pilato  wrote:
>>
>>>  You could have one type per form although the cluster state will be
>>> very big.
>>>  But you should test that option.
>>>
>>>  Or if you don't really search for numbers as numbers (I mean with Range
>>> queries/filters), you could force each field to be a String and do the
>>> transformation at a client level.
>>>
>>>  My 2 cents
>>>
>>>
>>>  --
>>> * David Pilato* | Technical Advocate | *elasticsearch.com
>>> *
>>>  david.pil...@elasticsearch.com
>>>  @dadoonet  | @elasticsearchfr
>>>  | @scrutmydocs
>>> 
>>>  
>>>
>>>
>>>
>>> Le 19 septembre 2014 à 05:31:38, Michael Chen (mechil...@gmail.com) a
>>> écrit:
>>>
>>>  Hi,
>>>
>>> We have a system very much like Google Forms, which allow users to
>>> design their own forms with various fields (single line text, paragraph,
>>> number, address etc, you can imagine that.) Without any doubt, it's running
>>> on top of MongoDB. Now it has 120K forms with nearly 10 million entries.
>>>
>>> Recently we found a performance bottleneck on the query. After we have
>>> done every possible performance tuning on mongoDB side, we decide to index
>>> the form entries into elastic search. And there is a trouble:
>>>
>>> Given there is a Form A, has field_1 as string type, field_2 as number,
>>> the data entry might look like: { field_1: "hello", field_2: 100}
>>>
>>> Form B could be field_1 as number, field_2 as number, the date entry
>>> will look like { field_1: 100, field_2: "hello form"}
>>>
>>> We have successfully create an index "entries" in ES, and can index the
>>> first entry successfully. But the second one failed for an obvious reason:
>>> type mismatch.
>>>
>>> I am not sure how to deal with this problem. I definitely don't want to
>>> create 120K indices for every single form. And I am not sure it's doable to
>>> write custom tran

Limit query results using elasticsearchtemplate

[Vipin Sudhakar]

How can we limit the query results using elasticsearchtemplate spring data 
elastic search integration.

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8d38f9a7-4297-4b30-a4e0-ba25c23efcb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index design for very dynamic Form/Fields

[Michael Chen]

We cannot guarantee that field_1 is always address. In Form 1, field_1
might be address while in another form it might be string or number
whatever. Thinking about designing the storage for Google Forms and it's
data entries.

Re "you could force each field to be a String and do the transformation at
a client level."

Forcing means serialize all data into a string right? In the example JSON
mentioned in previous email, it will transformed to something like

{ field_1: "\{country: \"US\", province: \"CA\", city: \"New York\",
address: \"Street Address\"\}"  }

Then not able to do the aggregation.

On Fri, Sep 19, 2014 at 5:50 PM, David Pilato  wrote:

> I don't get it.
>
> If field_1.country is a String why you can not aggregate on it?
>
> --
> *David Pilato* | Technical Advocate | *elasticsearch.com
> *
> david.pil...@elasticsearch.com
> @dadoonet  | @elasticsearchfr
>  | @scrutmydocs
> 
> 
>
>
>
> Le 19 septembre 2014 à 08:27:19, Michael Chen (mechil...@gmail.com) a
> écrit:
>
> Thanks David. Based on the system behavior, having all type as string is
> fine for queries. But for the aggregation level it might be trouble. For
> example a type of address is a complex JSON object:
>
> { field_1: { country: "US", province: "CA", city: "New York", address:
> "Street Address"} }
>
> If we transform this type into any form of string, and trying to aggregate
> based on country/state, it will be VERY hard even not possible.
>
> On Fri, Sep 19, 2014 at 2:15 PM, David Pilato  wrote:
>
>>  You could have one type per form although the cluster state will be
>> very big.
>>  But you should test that option.
>>
>>  Or if you don't really search for numbers as numbers (I mean with Range
>> queries/filters), you could force each field to be a String and do the
>> transformation at a client level.
>>
>>  My 2 cents
>>
>>
>>  --
>> * David Pilato* | Technical Advocate | *elasticsearch.com
>> *
>>  david.pil...@elasticsearch.com
>>  @dadoonet  | @elasticsearchfr
>>  | @scrutmydocs
>> 
>>  
>>
>>
>>
>> Le 19 septembre 2014 à 05:31:38, Michael Chen (mechil...@gmail.com) a
>> écrit:
>>
>>  Hi,
>>
>> We have a system very much like Google Forms, which allow users to design
>> their own forms with various fields (single line text, paragraph, number,
>> address etc, you can imagine that.) Without any doubt, it's running on top
>> of MongoDB. Now it has 120K forms with nearly 10 million entries.
>>
>> Recently we found a performance bottleneck on the query. After we have
>> done every possible performance tuning on mongoDB side, we decide to index
>> the form entries into elastic search. And there is a trouble:
>>
>> Given there is a Form A, has field_1 as string type, field_2 as number,
>> the data entry might look like: { field_1: "hello", field_2: 100}
>>
>> Form B could be field_1 as number, field_2 as number, the date entry will
>> look like { field_1: 100, field_2: "hello form"}
>>
>> We have successfully create an index "entries" in ES, and can index the
>> first entry successfully. But the second one failed for an obvious reason:
>> type mismatch.
>>
>> I am not sure how to deal with this problem. I definitely don't want to
>> create 120K indices for every single form. And I am not sure it's doable to
>> write custom transform script to change the index type identical across all
>> entries.
>>
>> Any suggestion? Much appreciate any response.
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/4a6d47d7-ae0e-44f5-bd3a-756ea94e3899%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>   --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>>  To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/etPan.541bc9fd.66334873.18d1%40MacBook-Air-de-David.local
>> .
>>
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Michael Chen
> 
> Blog: htt

Re: How to d exist or no tin elasticsearch

[Vipin Sudhakar]

Thanks a lot.. :)


On Friday, September 19, 2014 3:18:27 PM UTC+5:30, David Pilato wrote:
>
> And you meant in Java: 
> http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/query-dsl-filters.html#exists-filter
>
> -- 
> *David Pilato* | Technical Advocate | *elasticsearch.com 
> *
> david@elasticsearch.com 
> @dadoonet  | @elasticsearchfr 
>  | @scrutmydocs 
> 
> 
>
>
>
> Le 19 septembre 2014 à 11:47:15, David Pilato (da...@pilato.fr 
> ) a écrit:
>
>  
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-exists-filter.html#query-dsl-exists-filter
>  
>  -- 
> * David Pilato* | Technical Advocate | *elasticsearch.com 
> *
>  david@elasticsearch.com 
>  @dadoonet  | @elasticsearchfr 
>  | @scrutmydocs 
> 
>   
>
>  
>   
> Le 19 septembre 2014 à 11:42:43, Vipin Sudhakar (sumith...@gmail.com 
> ) a écrit:
>
>  We are migrating from mongo to ES 
>
> We have got  mongo query as below;
>
>  query.addCriteria(Criteria.where(FIELD_NAME).exists(true))
>
> What is the equivalent query in Elastic Search usnig QueryBuilder?
>
> Thanks.
>  --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/1ce800f3-4192-4d5c-a08a-3766699a3b0c%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c6395319-80a9-49ad-bf22-f552e62b6730%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index design for very dynamic Form/Fields

[David Pilato]

I don't get it.

If field_1.country is a String why you can not aggregate on it?

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 08:27:19, Michael Chen (mechil...@gmail.com) a écrit:

Thanks David. Based on the system behavior, having all type as string is fine 
for queries. But for the aggregation level it might be trouble. For example a 
type of address is a complex JSON object: 

{ field_1: { country: "US", province: "CA", city: "New York", address: "Street 
Address"} }

If we transform this type into any form of string, and trying to aggregate 
based on country/state, it will be VERY hard even not possible. 

On Fri, Sep 19, 2014 at 2:15 PM, David Pilato  wrote:
You could have one type per form although the cluster state will be very big.
But you should test that option.

Or if you don't really search for numbers as numbers (I mean with Range 
queries/filters), you could force each field to be a String and do the 
transformation at a client level.

My 2 cents


-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 05:31:38, Michael Chen (mechil...@gmail.com) a écrit:

Hi,

We have a system very much like Google Forms, which allow users to design their 
own forms with various fields (single line text, paragraph, number, address 
etc, you can imagine that.) Without any doubt, it's running on top of MongoDB. 
Now it has 120K forms with nearly 10 million entries. 

Recently we found a performance bottleneck on the query. After we have done 
every possible performance tuning on mongoDB side, we decide to index the form 
entries into elastic search. And there is a trouble:

Given there is a Form A, has field_1 as string type, field_2 as number, the 
data entry might look like: { field_1: "hello", field_2: 100}

Form B could be field_1 as number, field_2 as number, the date entry will look 
like { field_1: 100, field_2: "hello form"}

We have successfully create an index "entries" in ES, and can index the first 
entry successfully. But the second one failed for an obvious reason: type 
mismatch. 

I am not sure how to deal with this problem. I definitely don't want to create 
120K indices for every single form. And I am not sure it's doable to write 
custom transform script to change the index type identical across all entries. 

Any suggestion? Much appreciate any response. 
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a6d47d7-ae0e-44f5-bd3a-756ea94e3899%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541bc9fd.66334873.18d1%40MacBook-Air-de-David.local.

For more options, visit https://groups.google.com/d/optout.



--
Michael Chen

Blog: http://michael.nona.name
GTalk/Twitter/Facebook/Yahoo/Skype:  mechiland
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAzF%2BWj0jFb642x_hgko%2Bs9mfM3MCNqeS5Jr7TxtWq5XXAPvXg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541bfc4e.1f16e9e8.18d1%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: How to d exist or no tin elasticsearch

[David Pilato]

And you meant in Java: 
http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/query-dsl-filters.html#exists-filter

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 11:47:15, David Pilato (da...@pilato.fr) a écrit:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-exists-filter.html#query-dsl-exists-filter

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 11:42:43, Vipin Sudhakar (sumitharju...@gmail.com) a 
écrit:

We are migrating from mongo to ES

We have got  mongo query as below;

query.addCriteria(Criteria.where(FIELD_NAME).exists(true))

What is the equivalent query in Elastic Search usnig QueryBuilder?

Thanks.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1ce800f3-4192-4d5c-a08a-3766699a3b0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541bfbe1.12200854.18d1%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: How to d exist or no tin elasticsearch

[David Pilato]

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-exists-filter.html#query-dsl-exists-filter

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 11:42:43, Vipin Sudhakar (sumitharju...@gmail.com) a 
écrit:

We are migrating from mongo to ES

We have got  mongo query as below;

query.addCriteria(Criteria.where(FIELD_NAME).exists(true))

What is the equivalent query in Elastic Search usnig QueryBuilder?

Thanks.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1ce800f3-4192-4d5c-a08a-3766699a3b0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541bfba1.7545e146.18d1%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: New version of Kibana in the works?

[Julien Palard]

Hi Rashid !

On Thu, Sep 18, 2014 at 12:37 AM, Rashid Khan
 wrote:
> Unfortunately I can’t give you an ETA other than soon ;-)

Will you communicate on Kibana 4 in the meantime, like a kind of
changelog overview ? And will the source be available ? Before release
? After release ? Now ? Never ?

TL;DR: We want to contribute, but we're in the dark )o:

-- 
Julien Palard - 
CTO - meltygroup
Mobile : +33 6 21 19 49 10
http://www.meltygroup.com

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOO%3DpgBOLEu11HWXfnNzbPxjkYDJ588A78ACrTNYmBd20i6Lcw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to d exist or no tin elasticsearch

[Vipin Sudhakar]

We are migrating from mongo to ES

We have got  mongo query as below;

query.addCriteria(Criteria.where(FIELD_NAME).exists(true))

What is the equivalent query in Elastic Search usnig QueryBuilder?

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1ce800f3-4192-4d5c-a08a-3766699a3b0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem with zipcode

[David Pilato]

What gives GET /yourindex/_mapping

Your mapping definition does look like a mapping here

-- 
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs



Le 19 septembre 2014 à 10:44:31, Hari Rajaonarifetra (rhar...@gmail.com) a 
écrit:

Hi,

I have a big problem with zipcode.

See below my config :

ville:
                    mappings:
                        ville_nom:
                            index_analyzer : cities_index_analyzer
                            search_analyzer : cities_search_analyzer
                            type : string
                        ville_cp:
                            type: string
                            index: not_analyzed

But when I run "populate", elasticsearch cut the 0 on the begining of ville_cp 
(03000 => 3000).

Someone can help me?

Thanks,
Hari
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0348b69f-5418-4c9d-9f9e-e3ce2425a0fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.541bf471.2eb141f2.18d1%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Problem with zipcode

[Hari Rajaonarifetra]

Hi,

I have a big problem with zipcode.

See below my config :

ville:
mappings:
ville_nom:
index_analyzer : cities_index_analyzer
search_analyzer : cities_search_analyzer
type : string
ville_cp:
type: string
index: not_analyzed

But when I run "populate", elasticsearch cut the 0 on the begining of 
ville_cp (03000 => 3000).

Someone can help me?

Thanks,
Hari

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0348b69f-5418-4c9d-9f9e-e3ce2425a0fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch performance improvment

[Hsu Orozco]

Hi Mark,

I use command GET http://xxx.xxx.xxx.xxx:9200/_nodes to get the one node 
information 

direct_max_in_bytes=68518871040
heap_init_in_bytes=68719476736
heap_max_in_bytes=68518871040
non_heap_init_in_bytes=24313856
non_heap_max_in_bytes=136314880

total_in_bytes=135366918144

Java version=1.7.0_51
ES version=1.1.1

PS: I just think it's good to separate ES(indexer) and Mysql(data)? I mean 
ES only saves the indices and data stores in the Mysql or other powerful 
database.
If so, that will increase the query performance (aggregation query)

Thanks a lot



Mark Walkom於 2014年9月18日星期四UTC+8下午3時58分58秒寫道：
>
> What version of ES and java are you running?
> What is your heap size?
> What sort of queries are you running?
>
> Replicas and extra nodes may/will help but a bit more info would clear 
> things up :)
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
> On 18 September 2014 17:54, Hsu Orozco > 
> wrote:
>
>> Hi all,
>>
>> I have 4 nodes, memory size respe respectively 64GB, 64GB, 64GB and 128 
>> GB 
>> for handing 8,577,271,728 Documents and 68 TB data
>>
>> I open that API for kibana and a webUI query, but some issues happen to 
>> slow-query or out of memory.
>>
>> Here is my briefly configuration:
>>
>> shards = 176
>> replica = 0
>>
>> my question:
>>
>> 1. Should I set replica from 0 to 1 ?? As I know, replica means 
>> distributing data into different nodes, and that will help data recovery.
>> But does it helps to increase data searching while ES is doing the query??
>>
>> 2. Adding more machines is one of the solution, but is there any other 
>> approaches that I can try to make a better ES configuration?
>>
>> Thank
>>
>> Best, Orozco 
>>
>>
>>
>>
>>  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6ac1ac33-d3fd-409f-80a9-fd13abd80ca5%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a6840a06-e5a5-4e7c-a43f-01f418d4156a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The Max Number of Filelds ES Can Support

[xiehaiwei]

HI,

 In my case , there are about five thousand fields, Can ES support 
this?  How do the number of fields affect the speed of searching?

Thanks.
terrs


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a30b4502-9a50-4f84-bee1-c5519e47c3bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.