Re: Cluster state yellow after upgrade to 1.0
Hi Mo, The m indicates that all nodes *may* become master. The * means that node is the current master. There is nothing wrong with that. Your cluster is yellow because not all shards are started. If things don't sort out themselves within a couple of minutes (usually much quicker), something is going on. Cheers, Boaz On Monday, February 17, 2014 8:45:58 PM UTC+1, Mo wrote: I also see some shards in INITIALIZING [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/shards/test2' test2 2 p STARTED 37382168 9.8gb 10.80.140.59 test2 2 r INITIALIZING 10.80.140.62 test2 0 p STARTED 37382444 9.8gb 10.80.140.59 test2 0 r STARTED 37382444 9.8gb 10.80.140.62 test2 3 p STARTED 37388544 9.8gb 10.80.140.59 test2 3 r STARTED 37388544 9.8gb 10.80.140.62 test2 1 p STARTED 37389978 9.8gb 10.80.140.59 test2 1 r STARTED 37389978 9.8gb 10.80.140.62 test2 4 p STARTED 37393666 9.8gb 10.80.140.59 test2 4 r INITIALIZING 10.80.140.62 On Mon, Feb 17, 2014 at 11:44 AM, Mohit Anchlia mohita...@gmail.comjavascript: wrote: I just upgraded from .90 to 1.0 and my cluster state is showing yellow: Not sure why one node doesn't show up as m [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/nodes' 17 5 1.47 d m 10.80.140.62 22 4 0.00 d m 10.80.140.58 27 5 0.55 d * 10.80.140.59 21 4 0.06 d m 10.80.140.13 [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/indices' yellow test2 5 1 186936800 0 79gb 49.4gb green test1 5 1 142163219 0 76.9gb 38.4gb [root@ip-10-80-140-59 ec2-user]# -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95c8d401-ddc3-4922-b2ab-47453ad879a2%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow after upgrade to 1.0
I also see some shards in INITIALIZING [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/shards/test2' test2 2 p STARTED 37382168 9.8gb 10.80.140.59 test2 2 r INITIALIZING 10.80.140.62 test2 0 p STARTED 37382444 9.8gb 10.80.140.59 test2 0 r STARTED 37382444 9.8gb 10.80.140.62 test2 3 p STARTED 37388544 9.8gb 10.80.140.59 test2 3 r STARTED 37388544 9.8gb 10.80.140.62 test2 1 p STARTED 37389978 9.8gb 10.80.140.59 test2 1 r STARTED 37389978 9.8gb 10.80.140.62 test2 4 p STARTED 37393666 9.8gb 10.80.140.59 test2 4 r INITIALIZING 10.80.140.62 On Mon, Feb 17, 2014 at 11:44 AM, Mohit Anchlia mohitanch...@gmail.comwrote: I just upgraded from .90 to 1.0 and my cluster state is showing yellow: Not sure why one node doesn't show up as m [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/nodes' 17 5 1.47 d m 10.80.140.62 22 4 0.00 d m 10.80.140.58 27 5 0.55 d * 10.80.140.59 21 4 0.06 d m 10.80.140.13 [root@ip-10-80-140-59 ec2-user]# curl 'localhost:9200/_cat/indices' yellow test2 5 1 186936800 0 79gb 49.4gb green test1 5 1 142163219 0 76.9gb 38.4gb [root@ip-10-80-140-59 ec2-user]# -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWq6%2B6yhNHan1rMhSE2vvqTxEPbVr%2B%2BB0Y3haprAO_C%2BxQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Hey, not sure, if I made my point clear here: The minimum_master_nodes is an important setting, if a master node election is about to happen (this is not checked for each write of a document). The write_consistency is a check, when a document is about to be stored. Also note, that this check happens, before the data is written to the primary shard. (This is not checked during a master election). Hope it makes more sense now. ---Alex On Thu, Feb 6, 2014 at 6:35 PM, Mohit Anchlia mohitanch...@gmail.comwrote: So the checks are made in that order? 1) enough master nodes are available 2) Within that cluster decide how many replicas would acknowledge success before returning back to the client In other words just by having write_consistency set to quorum it doesn't gurantee protection against split brain. On Thu, Feb 6, 2014 at 1:30 AM, Alexander Reelsen a...@spinscale.dewrote: Hey, these two settings to not have a lot in common, wondering in what context you see them the same. minimum_master_nodes setting is only about the count of available nodes, which are eligible to become a master in order to actually have an existing master in a cluster. write_consistency qourum is actually a check when a document is indexed (and not about the nodes in a cluster) in order to decide if the document should be written at all (before it is actually indexed) See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#master-election and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-consistency Does that make sense, is this more clear now? --Alex On Mon, Feb 3, 2014 at 11:19 PM, Mohit Anchlia mohitanch...@gmail.comwrote: How does minimum_master_nodes differ from action.write_consistency? Would setting write_consistency to quorum help when minimum_master is not set appropriately? On Thu, Jan 16, 2014 at 6:08 AM, Norberto Meijome num...@gmail.comwrote: Gotcha, my bad. On 17/01/2014 12:43 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFgi5Wj%2Bz0P%2Bq3AJj55bTjaNhjGAzj_DHPmVfq6iRpBQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4JfuGA4wAmTCjMdo2T%2B4cuY96EV1EN3Grc-pDMt%3D3xX%3DA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrGJHi_R_WXquKD-GmYv_fojs23h%2B-rBs-NZxNSznQwmQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-4K9f6dDScGMS75nZU-LYBvdQ5opi7k-nVBi6w%3DwstyA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWp6QV%2B8ySSY7PYLBiz2DEE%2B-xRvU0SnUiORJdRc1r85Fg%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-wn_YzhkiPS1F1aM_hxAioHNtmO_ndQyUOPA-_S52KFQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Hey, these two settings to not have a lot in common, wondering in what context you see them the same. minimum_master_nodes setting is only about the count of available nodes, which are eligible to become a master in order to actually have an existing master in a cluster. write_consistency qourum is actually a check when a document is indexed (and not about the nodes in a cluster) in order to decide if the document should be written at all (before it is actually indexed) See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#master-election and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-consistency Does that make sense, is this more clear now? --Alex On Mon, Feb 3, 2014 at 11:19 PM, Mohit Anchlia mohitanch...@gmail.comwrote: How does minimum_master_nodes differ from action.write_consistency? Would setting write_consistency to quorum help when minimum_master is not set appropriately? On Thu, Jan 16, 2014 at 6:08 AM, Norberto Meijome num...@gmail.comwrote: Gotcha, my bad. On 17/01/2014 12:43 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFgi5Wj%2Bz0P%2Bq3AJj55bTjaNhjGAzj_DHPmVfq6iRpBQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4JfuGA4wAmTCjMdo2T%2B4cuY96EV1EN3Grc-pDMt%3D3xX%3DA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrGJHi_R_WXquKD-GmYv_fojs23h%2B-rBs-NZxNSznQwmQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-4K9f6dDScGMS75nZU-LYBvdQ5opi7k-nVBi6w%3DwstyA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
So the checks are made in that order? 1) enough master nodes are available 2) Within that cluster decide how many replicas would acknowledge success before returning back to the client In other words just by having write_consistency set to quorum it doesn't gurantee protection against split brain. On Thu, Feb 6, 2014 at 1:30 AM, Alexander Reelsen a...@spinscale.de wrote: Hey, these two settings to not have a lot in common, wondering in what context you see them the same. minimum_master_nodes setting is only about the count of available nodes, which are eligible to become a master in order to actually have an existing master in a cluster. write_consistency qourum is actually a check when a document is indexed (and not about the nodes in a cluster) in order to decide if the document should be written at all (before it is actually indexed) See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#master-election and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-consistency Does that make sense, is this more clear now? --Alex On Mon, Feb 3, 2014 at 11:19 PM, Mohit Anchlia mohitanch...@gmail.comwrote: How does minimum_master_nodes differ from action.write_consistency? Would setting write_consistency to quorum help when minimum_master is not set appropriately? On Thu, Jan 16, 2014 at 6:08 AM, Norberto Meijome num...@gmail.comwrote: Gotcha, my bad. On 17/01/2014 12:43 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFgi5Wj%2Bz0P%2Bq3AJj55bTjaNhjGAzj_DHPmVfq6iRpBQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4JfuGA4wAmTCjMdo2T%2B4cuY96EV1EN3Grc-pDMt%3D3xX%3DA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrGJHi_R_WXquKD-GmYv_fojs23h%2B-rBs-NZxNSznQwmQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-4K9f6dDScGMS75nZU-LYBvdQ5opi7k-nVBi6w%3DwstyA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWp6QV%2B8ySSY7PYLBiz2DEE%2B-xRvU0SnUiORJdRc1r85Fg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFgi5Wj%2Bz0P%2Bq3AJj55bTjaNhjGAzj_DHPmVfq6iRpBQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Gotcha, my bad. On 17/01/2014 12:43 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: minimum_master_nodes is a dynamic cluster setting, that means, it can be set via cluster update API. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFgi5Wj%2Bz0P%2Bq3AJj55bTjaNhjGAzj_DHPmVfq6iRpBQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4JfuGA4wAmTCjMdo2T%2B4cuY96EV1EN3Grc-pDMt%3D3xX%3DA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Unicast is a nightmare for large ES deployments, with provisioning and failures all the time. I'm used to DHCP/TFTP/PXE in my DC thanks to RedHat so why should I waste time setting up hostnames or count hosts for ES? Jörg On Wed, Jan 15, 2014 at 5:17 PM, InquiringMind brian.from...@gmail.comwrote: One more reason to prefer unicast over multicast! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGdS1h-f85Tk-h8SBY92eqfXEvrGV6E0sqPNwkbXrfaxA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
IMHO I think ES can still be smart enough to calculate that formula dynamically since it knows when servers are being added to the cluster, correct? As for the node crash it's still a crash and if user wants to decomission the node then the better way would be to explicitly run a decomission command to indicate that nodes is not part of cluster anymore? Is there any problem with this logic? On Wed, Jan 15, 2014 at 9:15 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: Unicast is a nightmare for large ES deployments, with provisioning and failures all the time. I'm used to DHCP/TFTP/PXE in my DC thanks to RedHat so why should I waste time setting up hostnames or count hosts for ES? Jörg On Wed, Jan 15, 2014 at 5:17 PM, InquiringMind brian.from...@gmail.comwrote: One more reason to prefer unicast over multicast! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGdS1h-f85Tk-h8SBY92eqfXEvrGV6E0sqPNwkbXrfaxA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWoCEg%3DL-wDNXR1mFFScDZT%2BTxUsAoyBZ31OHoLtEyo6uA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Jörg I avoided multicast and preferred unicast based on many discussions in the newsgroups and other sites. In particular, the ElasticSearch Preflight Checklisthttp://asquera.de/opensource/2012/11/25/elasticsearch-pre-flight-checklist/. Within this checklist, the sections entitled DISCOVERY and AVOIDING SPLIT-BRAINS were the recommendationsI that I followed. This write-up didn't say that mutlicast was bad, but neither have I read before that unicast was a nightmare. Whenever you get a chance, if you could describe the provisioning issues and failures, that would help to shed some more light on this subject. And it does seem that somebody needs to waste time counting hosts. You stated (and I agree) that minimum_master_nodes must be set to N/2 + 1, and N must be specified by the system admin... via counting? Or is there something else I am not seeing? Brian On Wednesday, January 15, 2014 12:15:27 PM UTC-5, Jörg Prante wrote: Unicast is a nightmare for large ES deployments, with provisioning and failures all the time. I'm used to DHCP/TFTP/PXE in my DC thanks to RedHat so why should I waste time setting up hostnames or count hosts for ES? Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4353c8da-b73e-4175-a083-a80ea3d65f69%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
The preflight checklist is misleading in several statements: - you can not accidentally join clusters because of multicast. It happens when using the default cluster name - a lot of chatter with no use is just pure ignorance of network technology. Multicast was designed for zero config, and I love ES for zero config. And there is no chatter if you set up a private network for ES hosts. That is simple with a router/switch and DHCP. Whole corporation and datacenter networks can not live without link-local config for example. - you don't have to include the whole cluster is misleading - each data node must always be able to connect to one of the eligible master nodes, or the cluster may not work correctly - the avoiding split brain section is a very bold and wrong headline. It is promising that split-brains can be avoided by a N/2+1 setting. Unfortunately that is nonsense, the truth is, it lowers the risk significantly. But there can still be obscure network splits where two halves of a split cluster may overlap, so the condition of N/2+1 can become true for both halves. Why do I not count hosts/nodes? There is a difference between the number of data nodes and master eligible nodes in a cluster. I have three master eligible nodes and set minimum_master_nodes to 3, and that's it. After that I can start or stop additional data nodes as much as I like. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG8Yi8KuG7%2BYa3HQEAj%2BDhPUvzoqd6aR-Gpd4iQxAF%3D%2Bw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Thank you very much, Jörg. Your explanation is clear and concise and greatly improves my understanding! Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18afbcfe-fcd1-47e9-91cb-a617dc089abf%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
IHO, I think there is no perfect solution for any of the complex network issues, however if we know of a config that reduces the risk signficantly then I think we should adopt that as a default config. On Wed, Jan 15, 2014 at 11:17 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: The preflight checklist is misleading in several statements: - you can not accidentally join clusters because of multicast. It happens when using the default cluster name - a lot of chatter with no use is just pure ignorance of network technology. Multicast was designed for zero config, and I love ES for zero config. And there is no chatter if you set up a private network for ES hosts. That is simple with a router/switch and DHCP. Whole corporation and datacenter networks can not live without link-local config for example. - you don't have to include the whole cluster is misleading - each data node must always be able to connect to one of the eligible master nodes, or the cluster may not work correctly - the avoiding split brain section is a very bold and wrong headline. It is promising that split-brains can be avoided by a N/2+1 setting. Unfortunately that is nonsense, the truth is, it lowers the risk significantly. But there can still be obscure network splits where two halves of a split cluster may overlap, so the condition of N/2+1 can become true for both halves. Why do I not count hosts/nodes? There is a difference between the number of data nodes and master eligible nodes in a cluster. I have three master eligible nodes and set minimum_master_nodes to 3, and that's it. After that I can start or stop additional data nodes as much as I like. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG8Yi8KuG7%2BYa3HQEAj%2BDhPUvzoqd6aR-Gpd4iQxAF%3D%2Bw%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWoibQk_0o313aXEwYkbYrQiVnmgrp4eNnOMTurmCS6PPw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Using quorum consensus (another name for the 'minimum_master_node' approach) as default is not possible, since the quorum count is only known by the admin. There are perfect solutions for consensus but they are not easy to implement, see Byzantine fault tolerance http://en.wikipedia.org/wiki/Byzantine_fault_tolerance or Paxos http://research.google.com/archive/paxos_made_live.html or a more promising approach started lately, RAFT http://raftconsensus.github.io/ Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH4FFgKVeJY37aRuurAt7To5jLa%3D0Rp3b--AcGE%3DhEqoA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Default value for minimum master node is 1. No formula here. It was just an advice. Sorry if I was unclear. :-) -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 14 janv. 2014 à 20:58, Mohit Anchlia mohitanch...@gmail.com a écrit : I didn't quite understand. What I am trying to find is that when default formula of N / 2 + 1 I will always have more master nodes on one side correct? And is my understanding incorrect that the other side that has less master nodes than N /2 + 1 would simply not accept any requests? The only point of increasing this value would be for performance reasons? On Mon, Jan 13, 2014 at 3:58 PM, Mark Walkom ma...@campaignmonitor.com wrote: That depends on however many nodes lose contact, but it's a possibility that if you lose 2 of your nodes at once they will just stop responding and the other 3 will be ok. However you maintain cluster integrity. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 January 2014 10:55, Mohit Anchlia mohitanch...@gmail.com wrote: How would the nodes on the other side of the network become master with the formula (total num of nodes / 2 + 1) one side will always become inaccessible? On Mon, Jan 13, 2014 at 3:50 PM, Mark Walkom ma...@campaignmonitor.com wrote: The logs shows that a node was not contactable and so was removed from the cluster. That could be a load issue on the node, network issues, or something else. Check the logs on the node that disconnected and see what you find. If you leave the defaults, you risk the chance of a split-brain situation, where nodes lose contact with each other and then decide to become a master. This can cause you to end up with multiple masters in multiple clusters, which is not good for data integrity. Setting the min masters means that if a node cannot contact N masters, it will refuse connections. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 January 2014 10:46, Mohit Anchlia mohitanch...@gmail.com wrote: I am trying to understand why I should change it? This setting looks like the right setting in most cases. Did you see anything out of ordinary in that log message I pasted? On Mon, Jan 13, 2014 at 3:40 PM, David Pilato da...@pilato.fr wrote: No it's not the default setting. min master node = total num of nodes / 2 + 1 You have 5 nodes, 5 / 2 + 1 = 3 -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 14 janvier 2014 at 00:06:34, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Is the default setting 3? How do I go about selecting right configuration setting for this property? On Mon, Jan 13, 2014 at 3:01 PM, David Pilato da...@pilato.fr wrote: So I think you should first set minimum master node to 3. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery Probably you need to close your isolated node and restart it. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 23:58:33, Mohit Anchlia (mohitanch...@gmail.com) a écrit: I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- David Pilato | Technical Advocate |
Re: Cluster state yellow
What's the problem we face when ES node starts up? Can you please elaborate? On Tue, Jan 14, 2014 at 12:54 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: To be fair, the default minimum_master_node value is -1 (disabled). Problem is, you do not know N when an ES node starts up and uses discovery - it can grow by time ;) Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoED55JCMB65gb-13pNxinqR0F1hDuVvh_dtjLNuyWB1iA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWoxQkYnB0cn-rvJG6d6f3q56L8W%3D%2B56akhWZHcNydkw%2Bw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
*Wouldn't the value increase as you add more nodes?* Indeed, it most certainly will. And with unicast discovery, each node will need to be told about the new node. Which is the perfect time to tell it about the newly calculated minimum number of masters. Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54e31afb-ea7f-4854-b970-f534505775cd%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
But today elasticsearch doesn't do this automatically? On Tue, Jan 14, 2014 at 2:17 PM, InquiringMind brian.from...@gmail.comwrote: *Wouldn't the value increase as you add more nodes?* Indeed, it most certainly will. And with unicast discovery, each node will need to be told about the new node. Which is the perfect time to tell it about the newly calculated minimum number of masters. Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54e31afb-ea7f-4854-b970-f534505775cd%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWq_iK7dFfooQt7HLgZ77WjhZUMV%3DDTtKOSEcU-fWK0-xw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Nope. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 January 2014 09:48, Mohit Anchlia mohitanch...@gmail.com wrote: But today elasticsearch doesn't do this automatically? On Tue, Jan 14, 2014 at 2:17 PM, InquiringMind brian.from...@gmail.comwrote: *Wouldn't the value increase as you add more nodes?* Indeed, it most certainly will. And with unicast discovery, each node will need to be told about the new node. Which is the perfect time to tell it about the newly calculated minimum number of masters. Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54e31afb-ea7f-4854-b970-f534505775cd%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWq_iK7dFfooQt7HLgZ77WjhZUMV%3DDTtKOSEcU-fWK0-xw%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624a-47F_w7e0u2jaURfiPiHGL-p5M8-UD1qOe0bMqFKvuA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Don't forget gateway.expected_nodes :) Wouldn't the value increase as you add more nodes? It will, which is precisely why the value is not computed automatically. The value can decrease/increase over time, but the cluster does not know if this is because it is on purpose or because of failures. The suggested (n/2)+1 formula breaks down if n is constantly changing. With 10 nodes, it is suggested to use the value of 6 (10/2+1) for the minimum number of nodes. But if 5 nodes crash, then the suggested value is now 3 (5/2+1), but clearly 3 is not the value we want, we want 6. Cheers, Ivan On Tue, Jan 14, 2014 at 2:58 PM, Mark Walkom ma...@campaignmonitor.comwrote: Nope. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 15 January 2014 09:48, Mohit Anchlia mohitanch...@gmail.com wrote: But today elasticsearch doesn't do this automatically? On Tue, Jan 14, 2014 at 2:17 PM, InquiringMind brian.from...@gmail.comwrote: *Wouldn't the value increase as you add more nodes?* Indeed, it most certainly will. And with unicast discovery, each node will need to be told about the new node. Which is the perfect time to tell it about the newly calculated minimum number of masters. Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54e31afb-ea7f-4854-b970-f534505775cd%40googlegroups.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWq_iK7dFfooQt7HLgZ77WjhZUMV%3DDTtKOSEcU-fWK0-xw%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624a-47F_w7e0u2jaURfiPiHGL-p5M8-UD1qOe0bMqFKvuA%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMrYAFBv%2BnRCMRYbev0bEaTPHswuUyEBsFvHS8XzKKvA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
*But today elasticsearch doesn't do this automatically?* Short answer: No. Long answer: Nooo. *:-)* For unicast discovery, you need to tell each node the full list of nodes in the cluster. As far as I can tell, that list may include the local node so it's easier to configure the same list everywhere. But it's not automatic. For multicast discovery, it's automatic. But I've read enough about issues with multicast to want to stay with unicast. Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2fc5288-9da9-412b-afc6-9e7ba923f7b6%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
It's not a a matter of unicast vs. multicast or failure vs. non-failure. It's only the cluster admin who knows what the maximum number of nodes is. For ES it is impossible to find that out during runtime, and therefore, minimum_master_nodes is a user defined setting. Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGWAo6A6LK48%2BY0_z2nysrjW_jTud%2Bu1UEtJSJA-aUUA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Cluster state yellow
Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d46968.5577f8e1.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300 ]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d46968.5577f8e1.376c%40MacBook-Air-de-David.local . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrsU-k9u6hir2CWoobhnaOTD_3Bjk3Xf_ZjMDSBkB0DeQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cluster state yellow
Is the default setting 3? How do I go about selecting right configuration setting for this property? On Mon, Jan 13, 2014 at 3:01 PM, David Pilato da...@pilato.fr wrote: So I think you should first set minimum master node to 3. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery Probably you need to close your isolated node and restart it. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 23:58:33, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300 ]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d46968.5577f8e1.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit
Re: Cluster state yellow
No it's not the default setting. min master node = total num of nodes / 2 + 1 You have 5 nodes, 5 / 2 + 1 = 3 -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 14 janvier 2014 at 00:06:34, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Is the default setting 3? How do I go about selecting right configuration setting for this property? On Mon, Jan 13, 2014 at 3:01 PM, David Pilato da...@pilato.fr wrote: So I think you should first set minimum master node to 3. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery Probably you need to close your isolated node and restart it. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 23:58:33, Mohit Anchlia (mohitanch...@gmail.com) a écrit: I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d46968.5577f8e1.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrsU-k9u6hir2CWoobhnaOTD_3Bjk3Xf_ZjMDSBkB0DeQ%40mail.gmail.com. For more options, visit
Re: Cluster state yellow
I am trying to understand why I should change it? This setting looks like the right setting in most cases. Did you see anything out of ordinary in that log message I pasted? On Mon, Jan 13, 2014 at 3:40 PM, David Pilato da...@pilato.fr wrote: No it's not the default setting. min master node = total num of nodes / 2 + 1 You have 5 nodes, 5 / 2 + 1 = 3 -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 14 janvier 2014 at 00:06:34, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Is the default setting 3? How do I go about selecting right configuration setting for this property? On Mon, Jan 13, 2014 at 3:01 PM, David Pilato da...@pilato.fr wrote: So I think you should first set minimum master node to 3. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery Probably you need to close your isolated node and restart it. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 23:58:33, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300 ]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52d45be9.7c3dbd3d.376c%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpLn_AgMKcTwQuGD-CHHe6ptg%3DRxaNGLPxFn3kroed%3DVQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group
Re: Cluster state yellow
The logs shows that a node was not contactable and so was removed from the cluster. That could be a load issue on the node, network issues, or something else. Check the logs on the node that disconnected and see what you find. If you leave the defaults, you risk the chance of a split-brain situation, where nodes lose contact with each other and then decide to become a master. This can cause you to end up with multiple masters in multiple clusters, which is not good for data integrity. Setting the min masters means that if a node cannot contact N masters, it will refuse connections. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 January 2014 10:46, Mohit Anchlia mohitanch...@gmail.com wrote: I am trying to understand why I should change it? This setting looks like the right setting in most cases. Did you see anything out of ordinary in that log message I pasted? On Mon, Jan 13, 2014 at 3:40 PM, David Pilato da...@pilato.fr wrote: No it's not the default setting. min master node = total num of nodes / 2 + 1 You have 5 nodes, 5 / 2 + 1 = 3 -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 14 janvier 2014 at 00:06:34, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Is the default setting 3? How do I go about selecting right configuration setting for this property? On Mon, Jan 13, 2014 at 3:01 PM, David Pilato da...@pilato.fr wrote: So I think you should first set minimum master node to 3. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html#_discovery Probably you need to close your isolated node and restart it. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 23:58:33, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: I have 5 nodes. I am using all the default settings. On Mon, Jan 13, 2014 at 2:32 PM, David Pilato da...@pilato.fr wrote: It sounds like a split brain issue or something. How many nodes do you have? Did you set minimum master node? -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:48:32, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Thanks! It looks like one of the nodes even though it was up still it went out of cluster. Is there no retry? [2014-01-02 17:55:01,148][INFO ][cluster.service ] [10.80.140.14] master {new [10.80.140.14][267foffDRhO6zzXlHg3YKA][inet[/10.80.140.14:9300]]{group=group1}, previous [10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2}}, removed {[10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300]]{group=group2},}, reason: zen-disco-master_failed ([10.80.140.59][YDHawZWnRUKukXDIIT5MJQ][inet[/10.80.140.59:9300 ]]{group=group2}) [2014-01-02 17:55:02,158][INFO ][cluster.service ] [10.80.140.14] removed {[10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2},}, reason: zen-disco-node_failed([10.80.140.13][6cf39fcASKSn6dNNsSXguA][inet[/10.80.140.13:9300]]{group=group2}), reason failed to ping, tried [3] times, each with maximum [30s] timeout On Mon, Jan 13, 2014 at 1:34 PM, David Pilato da...@pilato.fr wrote: Yellow means that some replicas are not allocated. If you have only one node and have index set with one replica (default), yellow is the expected state. -- *David Pilato* | *Technical Advocate* | *Elasticsearch.com* @dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr Le 13 janvier 2014 at 22:30:23, Mohit Anchlia (mohitanch...@gmail.com//mohitanch...@gmail.com) a écrit: Cluster state is showing yellow. What's the best way to find why it's in yellow state? It has been up and stable so far except when I looked today it shows state turned into yellow. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpSSEVTpYA-DB8Hgfyo9WdR4bVRoKtQ9c_PC%2BFZ1aQiKg%40mail.gmail.com . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion