Re: Elasticsearch Version Upgrade

2015-04-22 Thread Norberto Meijome
Yup thanks , that's what I thought. On 22/04/2015 2:49 pm, David Pilato da...@pilato.fr wrote: Only post 1.0 -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 22 avr. 2015 à 01:14, Norberto Meijome num...@gmail.com a écrit : David, is this the case with older

Re: Elasticsearch Version Upgrade

2015-04-21 Thread Norberto Meijome
David, is this the case with older versions (both client and server on 0.90.x versions using java client), and across the 0.90 to 1.x boundary, or only post 1.x? On 22/04/2015 12:03 am, David Pilato da...@pilato.fr wrote: This should work in both ways. The client knows what is the node

Re: Index Size and Replica Impact

2015-04-20 Thread Norberto Meijome
Replica = 3 means 4 copies of your data ( for each shard, 1 master and 3 replicas) On 21/04/2015 7:54 am, TB txind...@gmail.com wrote: I have my indexes size @ 6 GB currently with replica set @ 1. I have 3 node cluster, in order to utilize the cluster , my understanding that i would have set

Re: Getting XML into ES efficiently

2015-04-04 Thread Norberto Meijome
Hi, My gut feel is don't add this to the ES setup itself. Horses for courses - have your script (Python +1) running somewhere taking care of the processing, dealing with issues on the ftp side , etc. Let ES do its thing...specially if the XML parsing will take so much memory and you need external

Re: CAT API

2015-04-04 Thread Norberto Meijome
Replica =1 means you have 1 replica of the master shards - I.e. you have 2 copies of the data in total. On 02/04/2015 2:06 pm, Nishad Karekar nishad.kare...@acxiom.com wrote: I am unable to understand the results from the CAT API curl

Re: Why does creating a repository fail?

2015-03-18 Thread Norberto Meijome
yes, that's the difference between a (network service which exposes ) block storage like iSCSI and a network file system like NFS ( or glusterFS or Lustre... ). I don't see why on a local device (iSCSI) you'd have any issue with numeric uid not matching the 'name' of the user - unless, of course,

Re: logstash failed to send ping to elasticsearch

2015-03-11 Thread Norberto Meijome
TCP/9200 is for REST interface...Zen ping should be on 9300 ... I suspect you set a config wrong... On 11/03/2015 4:33 pm, Monika Bhadauria mon...@codeignition.co wrote: Hi guys, I have my Elasticsearch on one server and logstash on another. I am getting the following error in my logstash,

Re: EC2 cluster storage question

2015-02-25 Thread Norberto Meijome
? Tagging nodes probably won't help in this instance. Basically if you want to shut everything down you need to go through recovery, and depending on how long that takes it may not be worth the cost. This is something you need to test. On 25 February 2015 at 18:14, Norberto Meijome num...@gmail.com

Re: ES 0.9 on EC2 - Processor load maximizes on 100% of 1 core on multi core processor.

2015-02-21 Thread Norberto Meijome
, it just blows fully to 395% (~4*100%). On Friday, February 20, 2015 at 1:20:16 PM UTC+1, Norberto Meijome wrote: Hi Maik, Have you tried changing bulk size? May also be worth seeing if separating masters to their own nodes makes a difference... On 20/02/2015 8:22 pm, Maik Broxterman broxt

Re: ES 0.9 on EC2 - Processor load maximizes on 100% of 1 core on multi core processor.

2015-02-21 Thread Norberto Meijome
BTW, are you reducing / disabling the refresh rate while bulk indexing ? On 22/02/2015 10:08 am, Norberto Meijome num...@gmail.com wrote: OK, so what you have is resource contention between searches and indexing... On 22/02/2015 12:44 am, Maik Broxterman broxter...@gmail.com wrote: Ok, I

Re: ES 0.9 on EC2 - Processor load maximizes on 100% of 1 core on multi core processor.

2015-02-20 Thread Norberto Meijome
Hi Maik, Have you tried changing bulk size? May also be worth seeing if separating masters to their own nodes makes a difference... On 20/02/2015 8:22 pm, Maik Broxterman broxter...@gmail.com wrote: Hello, We are currently in the process of moving from an ES 0.9 cluster to an ES 1.4 cluster.

Re: Discovery on EC2 - unicast, separate VPCs, public IPs

2015-02-08 Thread Norberto Meijome
Sure...the interesting point in the OP is the fact both servers are in different VPCs - not sure if it should be possible to resolve across vpcs ... On 08/02/2015 7:04 pm, Ivan G igonzalezvalie...@gmail.com wrote: DNS queries inside vpc are resolved to the internal IP by aws servers. One

Re: Discovery on EC2 - unicast, separate VPCs, public IPs

2015-02-07 Thread Norberto Meijome
Are you referring to the master server by fqdn or IP? If fqdn, don't forget about ec2's split horizon (though I don't think it should be resolvable across 2 separate VPCs... Can you open a socket from host 1 to host 2 manually (with nc or telnet) on TCP/9300? On 08/02/2015 9:43 am, Eugen

Re: Is re-election/assignment of the master node possible?

2014-11-27 Thread Norberto Meijome
The load issue affecting master detection / election shouldn't happen if you have dedicated masters... At least it is with 0.90.x ( with my limited knowledge of ES implementation details, there seems to be a lock or priority issue when serving large # of requests (http / thrift) , affecting

Re: If I use EC2 Discovery Plugin do I necessarily give internet access to my instances?

2014-11-20 Thread Norberto Meijome
Yes..but this might not be an option if your instance is in a private subnet...it also means handling all your IPS like this ( though in theory you don't need internal IPs, security group id/name would do as well...) - there r limits to how many rules you can add to a secgroup At the same

Re: If I use EC2 Discovery Plugin do I necessarily give internet access to my instances?

2014-11-19 Thread Norberto Meijome
Hi David, Indeed, the plugin makes AWS API calls ( ec2 describe instances) in order to find candidates to cluster with. Unfortunately, if memory serves me right, those are to external IPs... Hint - tinyproxy with whitelist on your nat gw , and proper env configuration so that the client side

Re: ES filling up the 'old' GC pool

2014-11-18 Thread Norberto Meijome
FWIW, we saw many long running GC events using the default GC manager - changing to G1 solved most of the problems ( at the expense of slightly higher CPU all the time) After that you can take the longer road to debugging memory allocation for your use case :-) On 18/11/2014 6:21 am, Wilfred

Re: Cluster discovery on Amazon EC2 problem - need urgent help

2014-10-17 Thread Norberto Meijome
I am pretty sure you can open the ports for the sec group the elb belongs to , regardless of the az. (Az, not region). Unless you r using network acls. Anyway, not really ES... pm me if u want to continue the AWS discussion :-) On 16/10/2014 3:37 pm, Zoran Jeremic zoran.jere...@gmail.com wrote:

Re: running on EC2 S3 vs EBS

2014-10-13 Thread Norberto Meijome
Or, if your use case allows for it, have a very well oiled rebuild process (data included). On 14/10/2014 8:36 am, Itamar Syn-Hershko ita...@code972.com wrote: Yes, you don't want to use anything other than local storage for Elasticsearch. Not EBS and definitely not S3. You can use the

Re: Cluster discovery on Amazon EC2 problem - need urgent help

2014-10-12 Thread Norberto Meijome
, Norberto Meijome wrote: Zoran, good to hear it is working now. It should work pretty well with ec2 auto discovery - unicast is a good starting point but unless you are statically assigning them via cloud formation (or manually?), it may not be worth the trouble (and it stops you from dynamically

Re: Cluster discovery on Amazon EC2 problem - need urgent help

2014-10-10 Thread Norberto Meijome
Zoran, good to hear it is working now. It should work pretty well with ec2 auto discovery - unicast is a good starting point but unless you are statically assigning them via cloud formation (or manually?), it may not be worth the trouble (and it stops you from dynamically scaling your cluster) -

Re: alerts from Kibana/ES

2014-05-27 Thread Norberto Meijome
Hi, not sure tbh Kibana is a js interface so I don't think it makes sense to alert from it. You could monitor the results stored in ES with nagios/zabbix/ your monitoring of choice and parse the json result and alert based on that. We've used logstash's statsd module to send data we are interested

Re: 2 clusters versus 1 big cluster?

2014-03-21 Thread Norberto Meijome
@mauri, thank you for such interesting analysis. On 21/03/2014 1:01 PM, Mauri ma...@proactive-edge.com.au wrote: Hi Brad I agree with what Mark and Zachary have said and will expand on these. Firstly, shard and index level operations in ElasticSearch are peer-to-peer. Single-shard

Re: EC2 Discovery

2014-03-21 Thread Norberto Meijome
Don't try ec2 discovery until you have tested that: - you can connect from one machine to another on port 9300 ( nc as client and server, basic networking/ firewalling) - run a simple aws ec2 describe instances call with the API key you plan to use, and you can see the machines you need there.

Re: Elasticsearch 1.0.0 is now GA

2014-02-19 Thread Norberto Meijome
Agreed is bad form to force reinstall.but surely you would have your yml in a code/cfg repository? On 18/02/2014 9:14 AM, Tony Su tonysu...@gmail.com wrote: What?! Removing and re-installing the ES package either removes the original or over-writes the existing elasticsearch.yml The is

Re: ec2 discovery

2014-01-23 Thread Norberto Meijome
As I understand it, the ec2 plugin simply does ec2 API calls to list instances, filtered as per your config. It plays no actual part in the connectivity or clustering part - just discovery. So yes it makes sense what you saw. On 23/01/2014 7:43 PM, barak barak.ya...@gmail.com wrote: Hi, I've

Re: Node will not shut down

2014-01-21 Thread Norberto Meijome
Brad, What version of ES? What was ES doing on the problem node? (Hot threads call/ log file/ strace). Any related OS info( was it io bound? ) If it was really hung, I am not sure why the shutdown would work after moving the shards off it ( I.e. cluster was green...) ...it sounds to me like it was

Re: Cluster state yellow

2014-01-16 Thread Norberto Meijome
Gotcha, my bad. On 17/01/2014 12:43 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch

Re: Strategy for keeping Elasticsearch updated with MySQL

2014-01-10 Thread Norberto Meijome
+1 having a queue and consumers between your source of truth and ES is a great approach. You cab decouple and independently scale ( and stop when needed as DP said) the different components, minimising impact to your users. On 09/01/2014 7:35 AM, David Pilato da...@pilato.fr wrote: I would do 1/