Re: Setting up elasticsearch to scale: shards per index

2014-02-11 Thread Matt Fulton
Hey thanks for clarifying! I actually ended up setting it up as 1x master-only, 2x master-eligible data-nodes, realizing that I would need 3 eligible masters while putting it all together. On the heap problems, could you be more specific about what you are referring to, or maybe point me towards

Re: Setting up elasticsearch to scale: shards per index

2014-02-11 Thread joergpra...@gmail.com
Three master nodes are enough, for as many data nodes as you wish to add. You can search this mailing list for discussions where kimchy explained the dedicated master nodes, and how it fits for split-brain situations For example https://groups.google.com/forum/#!topic/elasticsearch/dxjpMd4vNXQ

Setting up elasticsearch to scale: shards per index

2014-02-10 Thread Matt
Hey all! Background: I am using elasticsearch with logstash to do some log analysis. My use-case is write-heavy, and I have configured ES accordingly. After experimenting with different setups, I am considering the following implementation: *separate log processing from ES cluster* 1x

Re: Setting up elasticsearch to scale: shards per index

2014-02-10 Thread Nik Everett
If you have an index per (some time period) then you can create the new indexes with more shards when you have more hardware and leave the old ones with their old number. You can also allocate more shards then noses in preparation for getting more nodes. I believe those are the standard tactics

Re: Setting up elasticsearch to scale: shards per index

2014-02-10 Thread joergpra...@gmail.com
Do not put single-node cluster (no replicas) in production. Never use single nodes except for development and demo. Always use replica and at least 3 nodes with minimum master nodes = 2 to avoid splitbrain in production. Having 17G RAM on a master-only server is more than enough of a beefy server

Re: Setting up elasticsearch to scale: shards per index

2014-02-10 Thread Matt
Thanks for the responses! After reading up on the split brain problem, I am moving to a three-node cluster with one master-only (on logstash server), one master/data, and one data-only server -- You received this message because you are subscribed to the Google Groups elasticsearch group. To

Re: Setting up elasticsearch to scale: shards per index

2014-02-10 Thread joergpra...@gmail.com
That is a misconception. To avoid split-brain, it is a good idea to set up master-only (data-less) nodes so they can not face heap problems. But that is not all, more important is to have at least three master eligible nodes, in case you get network disruptions. Additionally, you should set up at