Elasticsearch and Smile encoded JSON

2014-05-28 Thread Drew Kutcharian
Hey Guys, I wanted to get some clarification on how Elasticsearch handles/uses Smile binary JSON. Mainly: 1. Does ES convert JSON to Smile before saving into Lucene? 2. Does ES use Smile as the wire protocol for the Java Client? 3. If I wanted to have everything in Smile format (What's stored

Re: Elasticsearch and Smile encoded JSON

2014-05-29 Thread Drew Kutcharian
, standardization in RFC 7049) https://github.com/elasticsearch/elasticsearch/pull/5509 Jörg On Thu, May 29, 2014 at 7:07 AM, Drew Kutcharian d...@venarc.com wrote: Hey Guys, I wanted to get some clarification on how Elasticsearch handles/uses Smile binary JSON. Mainly: 1. Does ES convert

How to map a dynamic map of key values?

2014-06-10 Thread Drew Kutcharian
Hey Guys, How can I map an arbitrary map of key/values in ES? My JSON looks like the following, where name and age are static but attributes is dynamic: { name: john, age: 25, attributes : { key1: value1, key2: value2, key3: value3, ... } } Things to consider: 1. Not

Re: How to map a dynamic map of key values?

2014-06-11 Thread Drew Kutcharian
Thanks. I was also looking at dynamic mapping templates (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates) too. Would that approach work if I were to flatten my JSON so it'd be more like: { name: john, age: 25,

Clarification on has_child filter memory requirements

2014-06-18 Thread Drew Kutcharian
Based on the official docs (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html): {quote} memory considerations With the current implementation, all _parent field values and all _id field values of parent documents are loaded into memory (heap)

Re: Clarification on has_child filter memory requirements

2014-06-20 Thread Drew Kutcharian
in the nodes stats). Hope that helps. --Alex On Thu, Jun 19, 2014 at 6:03 AM, Drew Kutcharian d...@venarc.com wrote: Based on the official docs (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html): {quote} memory considerations

Re: Clarification on has_child filter memory requirements

2014-06-23 Thread Drew Kutcharian
...@traveljury.com wrote: I've updated the docs on memory usage with parent-child. Hopefully more understandable: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html?1#_memory_considerations_8 On 21 June 2014 07:32, Drew Kutcharian d

Corss-index parent/child relationship

2014-06-24 Thread Drew Kutcharian
Hi! Does ES support cross-index parent/child relationship? More specifically, can I have all the parents in one index (say users) and the children (say events) in a multiple time series style (managed by curator) indices? If so, how is this done? If not, what's the alternative? Thanks, Drew

Re: Corss-index parent/child relationship

2014-06-25 Thread Drew Kutcharian
bump ;) On Jun 24, 2014, at 6:48 PM, Drew Kutcharian d...@venarc.com wrote: Hi! Does ES support cross-index parent/child relationship? More specifically, can I have all the parents in one index (say users) and the children (say events) in a multiple time series style (managed

Re: has_child performance - alternative implementation

2014-06-25 Thread Drew Kutcharian
I'm interested to find this out too. On Jun 25, 2014, at 4:45 AM, Moran B moran.beni...@gmail.com wrote: In the documentation it says that ALL parent IDs must be resident in memory, the question is why. Why can't the has_child run the query per shard, load into memory all of the parent

Multi-tenancy strategy: 1 index with 1 shard and 1 replica per client

2014-06-26 Thread Drew Kutcharian
Hey Guys, I'm working on an analytics dashboard project where we collect events into Elasticsearch for clients. Each client could have millions of events per month. We are thinking of using one index with one shard and one replica per client. Looking at Logstash, it seems like Logstash creates

Re: Multi-tenancy strategy: 1 index with 1 shard and 1 replica per client

2014-06-26 Thread Drew Kutcharian
, 2014, at 8:29 PM, Drew Kutcharian d...@venarc.com wrote: Hey Guys, I'm working on an analytics dashboard project where we collect events into Elasticsearch for clients. Each client could have millions of events per month. We are thinking of using one index with one shard and one replica per

Re: Multi-tenancy strategy: 1 index with 1 shard and 1 replica per client

2014-06-26 Thread Drew Kutcharian
and use aliases and routing as this would be a much more efficient way of doing things. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 27 June 2014 11:21, Drew Kutcharian d...@venarc.com wrote: Hi

Re: Corss-index parent/child relationship

2014-06-26 Thread Drew Kutcharian
Thanks Matt, that feature is exactly what we need. One thing I couldn't figure out was that I would be able to pass a routing key so only relevant shards would be queried, right? On Jun 26, 2014, at 8:14 AM, Matt Weber matt.we...@gmail.com wrote: See PR #3278. Hopefully it will get merged

Any issues using 2 shards for an index?

2014-07-08 Thread Drew Kutcharian
Hi All, We are thinking of using two shards per index + 1 replica to keep the number of shards low for some indices. Are there any gotchas with using 2 shards per index besides that at most we can scale the writes to this index to two machines? Thanks, Drew -- You received this message

Re: Any issues using 2 shards for an index?

2014-07-08 Thread Drew Kutcharian
Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 9 July 2014 12:16, Drew Kutcharian d...@venarc.com wrote: Hi All, We are thinking of using two shards per index + 1 replica to keep the number of shards low for some indices. Are there any gotchas with using 2 shards

Re: Any issues using 2 shards for an index?

2014-07-08 Thread Drew Kutcharian
manageable, but you may cross a point where managing that becomes more hassle than it's worth. Something to keep in mind. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 9 July 2014 12:24, Drew Kutcharian d

How to ignore dynamic fields not matching any dynamic_templates?

2014-08-07 Thread Drew Kutcharian
Hey Guys, In my mapping, I set 'dynamic' to 'true' and I have a few 'dynamic_templates' that based on the name of the fields map them to different types. How can I ignore and not index fields that don't match any of the templates? I tried adding an ignoredFields dynamic template like below to

Problem with combined nested bool filters (nested key/value matching)

2014-08-20 Thread Drew Kutcharian
Hey Guys, Seems like there is an issue with a combined bool filter with nested docs. I have the following mapping: { mappings: { test: { properties: { name: { type: string, index: not_analyzed },

Re: Problem with combined nested bool filters (nested key/value matching)

2014-08-21 Thread Drew Kutcharian
I was able to get this to work. The problem was that I had to put the bool filter before the nested filter so now what I have is bool - nested - bool. On Aug 20, 2014, at 9:44 PM, Drew Kutcharian d...@venarc.com wrote: Hey Guys, Seems like there is an issue with a combined bool filter

Re: Open closed index automatically

2015-01-26 Thread Drew Kutcharian
is not. My 2 cents -- David Pilato | Technical Advocate | Elasticsearch.com http://elasticsearch.com/ @dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs Le 26 janv. 2015 à 21:41, Drew Kutcharian d

Open closed index automatically

2015-01-26 Thread Drew Kutcharian
Hi, Is there a way to tell ES to open closed indices automatically on first access (search or indexing)? I can do it on the application side by looking for IndexClosedException, but this seems like a natural thing that ES should be able to do, after all ES creates new indices automatically. -

Better understanding Lucene/Shard overheads

2015-01-22 Thread Drew Kutcharian
Hi, I just came across this blog post: http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html Seems like there has been a lot of work done on Lucene to reduce its memory requirements and even more

Re: Better understanding Lucene/Shard overheads

2015-01-23 Thread Drew Kutcharian
://blog.mikemccandless.com/ On Thu, Jan 22, 2015 at 4:07 PM, Drew Kutcharian d...@venarc.com mailto:d...@venarc.com wrote: Hi, I just came across this blog post: http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html http://blog.mikemccandless.com/2010/07/lucenes-ram-usage

murmur3 field type and doc_values

2015-03-25 Thread Drew Kutcharian
Hi Guys, Would it make sense to set doc_values: enabled on a murmur3 type field since it’s being used for aggregations? - Drew -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it,

Re: murmur3 field type and doc_values

2015-03-25 Thread Drew Kutcharian
can see what it is but how does that relate to ES? On 26 March 2015 at 10:28, Drew Kutcharian d...@venarc.com mailto:d...@venarc.com wrote: Hi Guys, Would it make sense to set doc_values: enabled on a murmur3 type field since it’s being used for aggregations? - Drew -- You