Re: ES - 1.4.2 - Uploading index data seems low and slow

2015-01-14 Thread Bhumir Jhaveri
ay, January 14, 2015 at 3:13:55 PM UTC-8, Mark Walkom wrote: > > FYI once you get to 32GB heap you lose some efficiency, try to keep heap > under 32GB, so 31GB or less. > > Are you using the bulk API? > > On 15 January 2015 at 10:03, Bhumir Jhaveri > wrote: > >> I

ES - 1.4.2 - Uploading index data seems low and slow

2015-01-14 Thread Bhumir Jhaveri
I have just migrated to ES 1.4.2 - I have 5 data nodes and 1 master node - all these ES instances are having 32 gigs of heap (machine has 120 gigs) - 2tb of storage each - 16 cpus I am trying to upload 3 indexes simultaneously - each of them are of size 12 gigs - i am trying to upload this via

Re: How to count all occurences in ES via search api call?

2015-01-14 Thread Bhumir Jhaveri
st the entire document. By having > nested (or parent child), now you have documents for each tag that you can > aggregate against. > > On Wednesday, January 14, 2015 at 11:35:07 AM UTC-8, Bhumir Jhaveri wrote: >> >> Hey, >> >> I have data stored in ES(1.4.2) as

How to count all occurences in ES via search api call?

2015-01-14 Thread Bhumir Jhaveri
Hey, I have data stored in ES(1.4.2) as 1st document : { "col1":"123","col2":"tag1,tag2,tag4" } 2nd... { "col1":"333","col2":"tag1,tag4,tag5" } 3rd... { "col1":"111","col2":"tag1,tag1,tag5,tag5" } now when I am searching it via making search api call - Search is for tag1 - it returns me the cou

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-08 Thread Bhumir Jhaveri
Did anyone noticed this? On Wednesday, January 7, 2015 4:29:58 PM UTC-8, Bhumir Jhaveri wrote: > > Here is what I did - > I had some data - I bumped up the ES version and then restarted the ES - > > It started giving following warnings - > > [2015-01-

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
uary 2015 at 05:55, Bhumir Jhaveri > wrote: > >> Also one more question - lets say intially I have one node architecture - >> i.e. everything on one single node and additional mount is having all the >> ES data (index, documents etc) which is like of 300 gigs - now lets s

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
emove a node, you can either stop the service and let > things recover, or use the shutdown API call to do it in a more ordered > fashion. > > On 8 January 2015 at 08:37, Bhumir Jhaveri > wrote: > >> what happens when you take off one of the node which is already having >

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
nuary 7, 2015 11:46:57 AM UTC-8, Mark Walkom wrote: > > Yes it auto distributes existing, and new, shards. > > On 8 January 2015 at 05:55, Bhumir Jhaveri > wrote: > >> Also one more question - lets say intially I have one node architecture - >> i.e. everything on one

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
anuary 2015 at 07:10, Bhumir Jhaveri > wrote: > >> Oh nice - so some indexes which I have created month ago which has - lets >> say 5 shards on one node - after adding node - 2 of these existing shards >> might move to newly added data node - this is great! >> >&g

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
whats the volume of data etc but in general who should be most powerful in terms of hardware capacity? On Wednesday, January 7, 2015 11:46:57 AM UTC-8, Mark Walkom wrote: > > Yes it auto distributes existing, and new, shards. > > On 8 January 2015 at 05:55, Bhumir Jhaveri > wrote

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
- > *David Pilato* | *Technical Advocate* | *Elasticsearch.com > <http://Elasticsearch.com>* > @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr > <https://twitter.com/elasticsearchfr> | @scrutmydocs > <https://twitter.com/scrutmydocs> > > > &g

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
icsearch.com > <http://Elasticsearch.com>* > @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr > <https://twitter.com/elasticsearchfr> | @scrutmydocs > <https://twitter.com/scrutmydocs> > > > > Le 7 janv. 2015 à 18:50, Bhumir Jhaveri > &

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
arch.com > <http://Elasticsearch.com>* > @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr > <https://twitter.com/elasticsearchfr> | @scrutmydocs > <https://twitter.com/scrutmydocs> > > > > Le 7 janv. 2015 à 18:41, Bhumir Jhaveri > &

Upgrade from ES 1.2.x to ES 1.4 or 1.3?

2015-01-07 Thread Bhumir Jhaveri
I am planning to upgrade from ES 1.2.2 to ES 1.4.2 or 1.3.9 or whichever is the most latest (stable) in 1.3 series - so should I go for 1.4 or should stay with 1.3? the reason why I am not thinking about absolute upgrade to 1.4 is just so that if certain things like shards allocations to differe

Re: Performance/Slowness issues with ES

2015-01-02 Thread Bhumir Jhaveri
wrote: > > Marvel is good for looking at metrics. I find the _cat API very helpful > for spot checking. _cat/nodes?v and _cat/indices?v are two I use pretty > frequently. We have a pretty tight statsd/graphite integration as well. > > On Fri, Jan 2, 2015 at 12:38 PM, Bhumir Jhaveri &g

Re: Performance/Slowness issues with ES

2015-01-02 Thread Bhumir Jhaveri
ers > I try to pay attention to what percent of the heap is being used for what > purposes. How does disk I/O look? What are your CPUs doing when those > queries come in? > > > On Fri, Jan 2, 2015 at 11:19 AM, Bhumir Jhaveri > wrote: > >> Also I am planning to c

Re: Performance/Slowness issues with ES

2015-01-02 Thread Bhumir Jhaveri
this to something. On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote: > > Hey guys, > > I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is > the situation - I am currently running everything on single node - which I > know is

Re: Performance/Slowness issues with ES

2015-01-02 Thread Bhumir Jhaveri
rie. > > > Op woensdag 31 december 2014 21:25:37 UTC+1 schreef Bhumir Jhaveri: >> >> Hey guys, >> >> I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here >> is the situation - I am currently running everything on single node - which >

Re: Performance/Slowness issues with ES

2015-01-02 Thread Bhumir Jhaveri
ere in the immediate term > and you will definitely see performance improvements. > > Given you're currently only using one node, you can also disable replicas, > that will eliminate those unassigned shards and you can easily add them > later. > > On 1 January 2015 at 07:

Performance/Slowness issues with ES

2014-12-31 Thread Bhumir Jhaveri
Hey guys, I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is the situation - I am currently running everything on single node - which I know is biggest problem n bottleneck. VM on which I am running is - 24gb ram- 8proc - 2TB storage space Here is snapshot of what I have