Please explain the flow of data?

2014-03-21 Thread Josh Harrison
I'm trying to build a basic understanding of how indexing and searching works, hopefully someone can either point me to good resources or explain! I'm trying to figure out what having multiple coordinator nodes as defined in the elasticsearch.yml would do, and what having multiple search load

Re: Please explain the flow of data?

2014-03-21 Thread Mark Walkom
A couple of things; 1. You should have n/2+1 masters in your cluster, where n = number of nodes. This helps prevent split brain situations and is best practise. 2. Your master nodes can store data, this way you don't need to add more nodes to fulfil the above. Your indexing scenario

Re: Please explain the flow of data?

2014-03-21 Thread Josh Harrison
Awesome, ok, thank you. Is the logic behind not allowing storage on master nodes to both: Take advantage of a system with limited storage resources and Have a dedicated results aggregator/search handler? I can imagine if I had a particularly badly written gnarly search, trying to deal with the

Re: Please explain the flow of data?

2014-03-21 Thread Mark Walkom
Yes you can leverage a master to be a search node in that way. We have a 15 node cluster with 3 masters, I'm thinking I'll add another 2 when we add a few more data nodes in the next few weeks. Essentially you want an uneven number of masters to ensure a quorum is reached. But when you start