[Linux-HA] Linux High Availibility/Stonith in VMs
Hi, I was wondering how does fencing/STONITH work in a VM environment? Obviously the nodes running inside a VM should run in separate machines but can we STONITH a node that is running inside a VM? Also what is the best practice in regards to VMs and HA? Thank you ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
Some are IBM servers mentioning VMK (optional) and others are Fujitsu mentioning iRMC. Not sure what can I do in the IBM case From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 10:36 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes I don't know anything about your setup or office situation, so I can't speak to your specific issues. What I can say is that almost all actual servers have remote management; Fujitsu servers have IPMI, HP servers have iLO, Dell servers have DRAC, IBM servers have RSA and so on. Most generic servers have at least basic IPMI. So it's very likely that you will already have what you need to implement fencing. If you don't, then something like the APC AP7900 is usually about $500 canadian and will work as a fence device. I don't know what country you are in, so I can't speak more on availability, applicability or cost. You might seriously want to consider hiring someone to help you with this setup. Clustering is not a straight-forward topic. The cost to hire a consultant might save you enough time and headache that it's cheaper in the end than trying to do it yourself. On 12/02/2012 03:31 PM, Hermes Flying wrote: I am an application developer and so far I didn't care about HW specs (not my responsibility). Now it seems I must care and so since I am not sure what HW is being recommended for servers I will verify tomorrow and let you know. My concern is that by going to pacemaker (although seems to be exactly what I need) is that we would need to request for new HW on migrate to new deployment. I don't even know if these devices are expensive. Did you ever had an issue with this? If I have problem with this, should I be looking into different solution? Any suggestion for alternative? *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:19 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes It must be an external device. If, for example, the kernel crashes hard, or if you get a spinlock, the system may not respond to anything or a service may never stop, blocking a reboot. You can not trust that the system is accessible or functioning in any way. Fencing *must* be an external device. Period. What kind of servers do you have? On 12/02/2012 03:15 PM, Hermes Flying wrote: If I have a requirement not to include external HW, is there any other way? I mean, I am not -by far- a linux expert, but how come it doesn't do a restart or halt? *From:* Digimer li...@alteeve.ca mailto:li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com mailto:flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:01 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 PM, Hermes Flying wrote: Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? Yes, there must be external hardware (out-of-band management counts as external, despite physically being on the server's mainboard). The most common fence device is IPMI, iLO, RSA, iDRAC and the like. Alternatives are switched PDUs, like APC's AP7900. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Does stonith always succeed?
Hi, My understanding is that HA using Pacemaker is based on fencing/STONITH . So my question is: Is STONITH guaranteed to ALWAYS succeed? Are there cases when it fails? When? E.g. in specific deployments (example geo-separation)? Thanks ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Does stonith always succeed?
But this assumes that the servers are co-located, right? How is geo-separated nodes supported? From: Digimer li...@alteeve.ca To: General Linux-HA mailing list linux-ha@lists.linux-ha.org Cc: Andrew Beekhof and...@beekhof.net; Hermes Flying flyingher...@yahoo.com Sent: Monday, December 3, 2012 1:18 PM Subject: Re: [Linux-HA] Does stonith always succeed? On 12/03/2012 06:11 AM, Andrew Beekhof wrote: On Mon, Dec 3, 2012 at 9:03 PM, Hermes Flying flyingher...@yahoo.com wrote: Hi, My understanding is that HA using Pacemaker is based on fencing/STONITH . So my question is: Is STONITH guaranteed to ALWAYS succeed? No. Are there cases when it fails? When? E.g. in specific deployments (example geo-separation)? Misconfiguration. Hardware failure. Plenty of reasons. Thanks Nothing is computers is ever 100%. Anyone who tells you otherwise is a marketing drone. This is why, in my clusters, I always use two fence methods; I use IPMI (or iLO, DRAC, RSA) as the preferred fence device but, if that fails, I have switched PDUs as backup fence devices. A classic way that fencing can fail is, for example, the power feeding a server fails (like a fried power supply or blown motherboard) which cuts the power to the IPMI BMC. In this scenario, the IPMI BMC is down and can't reply to the other node. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
I see, I will look into this. There is one thing I am confused about. If I have Node-A/Node-B/Node-C and let's say Node-A has the VIP. What happens if corosync fails in all nodes? Will all 3 try to kill eachother? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Monday, December 3, 2012 1:20 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes Most Fujitsu servers work with fence_ipmilan. I suspect the same is true with the IBM servers. Ask the hardware people if they have out-of-band management on the servers (which is the generic/marketing way of saying IPMI or the like). On 12/03/2012 05:00 AM, Hermes Flying wrote: Some are IBM servers mentioning VMK (optional) and others are Fujitsu mentioning iRMC. Not sure what can I do in the IBM case *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:36 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes I don't know anything about your setup or office situation, so I can't speak to your specific issues. What I can say is that almost all actual servers have remote management; Fujitsu servers have IPMI, HP servers have iLO, Dell servers have DRAC, IBM servers have RSA and so on. Most generic servers have at least basic IPMI. So it's very likely that you will already have what you need to implement fencing. If you don't, then something like the APC AP7900 is usually about $500 canadian and will work as a fence device. I don't know what country you are in, so I can't speak more on availability, applicability or cost. You might seriously want to consider hiring someone to help you with this setup. Clustering is not a straight-forward topic. The cost to hire a consultant might save you enough time and headache that it's cheaper in the end than trying to do it yourself. On 12/02/2012 03:31 PM, Hermes Flying wrote: I am an application developer and so far I didn't care about HW specs (not my responsibility). Now it seems I must care and so since I am not sure what HW is being recommended for servers I will verify tomorrow and let you know. My concern is that by going to pacemaker (although seems to be exactly what I need) is that we would need to request for new HW on migrate to new deployment. I don't even know if these devices are expensive. Did you ever had an issue with this? If I have problem with this, should I be looking into different solution? Any suggestion for alternative? *From:* Digimer li...@alteeve.ca mailto:li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com mailto:flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:19 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes It must be an external device. If, for example, the kernel crashes hard, or if you get a spinlock, the system may not respond to anything or a service may never stop, blocking a reboot. You can not trust that the system is accessible or functioning in any way. Fencing *must* be an external device. Period. What kind of servers do you have? On 12/02/2012 03:15 PM, Hermes Flying wrote: If I have a requirement not to include external HW, is there any other way? I mean, I am not -by far- a linux expert, but how come it doesn't do a restart or halt? *From:* Digimer li...@alteeve.ca mailto:li...@alteeve.ca mailto:li...@alteeve.ca mailto:li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com mailto:flyingher...@yahoo.com mailto:flyingher...@yahoo.com mailto:flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:01 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 PM, Hermes Flying wrote: Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? Yes, there must be external hardware (out-of-band management counts as external, despite physically being on the server's mainboard). The most common fence device is IPMI, iLO, RSA, iDRAC and the like. Alternatives are switched PDUs, like APC's AP7900. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind
Re: [Linux-HA] Does stonith always succeed?
These kind of deployments, are they part of a Linux-HA best practices document? E.g. the kind of backup defense you are using with the PDU is it something of your flavor or is it common practice? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org; Andrew Beekhof and...@beekhof.net Sent: Monday, December 3, 2012 2:07 PM Subject: Re: [Linux-HA] Does stonith always succeed? Poorly. Stretch clusters are extremely difficult to build, which is why I do not recommend building them. With few exceptions, tradition remote-backup warm-spare servers is better than stretch clusters. On 12/03/2012 07:05 AM, Hermes Flying wrote: But this assumes that the servers are co-located, right? How is geo-separated nodes supported? *From:* Digimer li...@alteeve.ca *To:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Cc:* Andrew Beekhof and...@beekhof.net; Hermes Flying flyingher...@yahoo.com *Sent:* Monday, December 3, 2012 1:18 PM *Subject:* Re: [Linux-HA] Does stonith always succeed? On 12/03/2012 06:11 AM, Andrew Beekhof wrote: On Mon, Dec 3, 2012 at 9:03 PM, Hermes Flying flyingher...@yahoo.com mailto:flyingher...@yahoo.com wrote: Hi, My understanding is that HA using Pacemaker is based on fencing/STONITH . So my question is: Is STONITH guaranteed to ALWAYS succeed? No. Are there cases when it fails? When? E.g. in specific deployments (example geo-separation)? Misconfiguration. Hardware failure. Plenty of reasons. Thanks Nothing is computers is ever 100%. Anyone who tells you otherwise is a marketing drone. This is why, in my clusters, I always use two fence methods; I use IPMI (or iLO, DRAC, RSA) as the preferred fence device but, if that fails, I have switched PDUs as backup fence devices. A classic way that fencing can fail is, for example, the power feeding a server fails (like a fried power supply or blown motherboard) which cuts the power to the IPMI BMC. In this scenario, the IPMI BMC is down and can't reply to the other node. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
Hi, So you are saying I should not use the notion of primary ok. When I have 3 nodes, won't 1 node have the VIP? How is this node defined in Pacemaker's terminology if primary is inappropriate? Best Regards From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 8:22 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 AM, Hermes Flying wrote: Hi, For a cluster with 2 nodes I was explained what would happen. The other node will take over using fencing. It will take over *after* fencing. Two separate concepts. Fencing ensures that a lost node is truly gone and not just partitioned. Once fencing succeeds and the lost node is known to be down, _then_ recovery of service(s) that had been running on the victim will begin. But in clusters with 3+ nodes what happens when corosync fails? I assume that if the communication fails with the primary, all other nodes consider themselves eligible to become primaries. Is this the case? Corosync failing will be treated as a failure in the node and the node will be removed and fenced. Any services that had been running on it may or may not be recovered, depending on the rules defined for that given service. If it is recovered, then where it is restarted again depends on how each service was configured. 1)If a node has problem communicating with the primary AND has network problem with the rest of the network (clients) does it still try to become the primary (try to kill other nodes?) Please drop the idea of pacemaker being primary; that's the wrong way to look at it. If pacemaker (via corosync) loses contact with it's peer(s), then it checks the quorum policy. If quorum is enabled, it checks to see if it had quorum. If it does, it will try to fence it's peer. If it doesn't, it will shut down any services it might have been running. Likely in this case, one of the nodes with quorum will fence it shortly. 2) In practice if the corosync fails but the primary is still up and running and serving requests, is primary attempted to be killed by the other nodes?Or you use some other way to figure out that this is a network failure, primary has not crashed? Again, drop the notion of primary. Whether a node tries to fence it's peer is a question of whether it has quorum (or if quorum is disabled). Failing corosync is the same as failing the whole node. Pacemaker will fail is corosync dies. 3)Finally on corosync failure I assume the primary does nothing, as it does not care about the backups. Is this correct? This question doesn't make sense. Thank you! np -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
But the VIP service is in coordination with corosync right? I mean when you say: if the node hosting the VIP fails, pacemaker may try to restart it or it might relocate it, depending on how you've configured things What does failure mean? That the service crashed OR also that corosync failed (so can not reach rest of nodes)? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 8:46 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes As I said; each _service_ can have the concept of primary, just not pacemaker itself. I gave an example earlier; Pacemkaer might have two services; * DRBD, active on all nodes. * VIP, active on one node only. In this example, the DRBD service is Active/Active. If it fails on a given node, it will try to restart. If that fails, it will *not* relocate. Here, there is no primary. The VIP on the other hand runs on one node at a time only. Generally it will start on the first active node, but you might configure it to prefer one node. If that preferred node comes online later, pacemaker will migrate it. If there is no preferred node, then the VIP will stay where it is. If the node hosting the VIP fails, pacemaker may try to restart it or it might relocate it, depending on how you've configured things. In this case, the VIP service has the concept of primary, though it's better to think of it as Active. Make sense? On 12/02/2012 01:35 PM, Hermes Flying wrote: Hi, So you are saying I should not use the notion of primary ok. When I have 3 nodes, won't 1 node have the VIP? How is this node defined in Pacemaker's terminology if primary is inappropriate? Best Regards *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 8:22 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 AM, Hermes Flying wrote: Hi, For a cluster with 2 nodes I was explained what would happen. The other node will take over using fencing. It will take over *after* fencing. Two separate concepts. Fencing ensures that a lost node is truly gone and not just partitioned. Once fencing succeeds and the lost node is known to be down, _then_ recovery of service(s) that had been running on the victim will begin. But in clusters with 3+ nodes what happens when corosync fails? I assume that if the communication fails with the primary, all other nodes consider themselves eligible to become primaries. Is this the case? Corosync failing will be treated as a failure in the node and the node will be removed and fenced. Any services that had been running on it may or may not be recovered, depending on the rules defined for that given service. If it is recovered, then where it is restarted again depends on how each service was configured. 1)If a node has problem communicating with the primary AND has network problem with the rest of the network (clients) does it still try to become the primary (try to kill other nodes?) Please drop the idea of pacemaker being primary; that's the wrong way to look at it. If pacemaker (via corosync) loses contact with it's peer(s), then it checks the quorum policy. If quorum is enabled, it checks to see if it had quorum. If it does, it will try to fence it's peer. If it doesn't, it will shut down any services it might have been running. Likely in this case, one of the nodes with quorum will fence it shortly. 2) In practice if the corosync fails but the primary is still up and running and serving requests, is primary attempted to be killed by the other nodes?Or you use some other way to figure out that this is a network failure, primary has not crashed? Again, drop the notion of primary. Whether a node tries to fence it's peer is a question of whether it has quorum (or if quorum is disabled). Failing corosync is the same as failing the whole node. Pacemaker will fail is corosync dies. 3)Finally on corosync failure I assume the primary does nothing, as it does not care about the backups. Is this correct? This question doesn't make sense. Thank you! np -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
When you say: So if corosync dies, pacemaker loses it's foundation and falls over dead. You mean pacemaker instance in all of nodes, right? Also (my emphasis ) The other node(s) in the cluster will declare the node as failed, fence it and then decide if any services need to be restarted Which node are your talking about? Itself? It will try to restart itself if corosync i.e. pacemaker dies? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 9:10 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes Corosync is used for cluster communications, membership and quorum. It doesn't care what the cluster is used for. It coordinates nothing. Think if it like the foundation of a house. Pacemaker, in turn, doesn't care how members in the cluster come or go, it only cares about who is a member now and whether or not it has quorum. When membership changes, it looks at the service(s) on the cluster and decides if anything should be stopped, started, relocated or restarted. Think of this as the house on top of the foundation. So if corosync dies, pacemaker loses it's foundation and falls over dead. The other node(s) in the cluster will declare the node as failed, fence it and then decide if any services need to be restarted. So if the node had a VIP, and pacemaker was configured to restart the VIP elsewhere, it would then do so. Now, if the VIP itself fails, but all the nodes are healthy, then pacemaker may simply try to restart the VIP on the same machine. If it keeps failing, then it may relocate the VIP to another node in the cluster. This, again, depends on how pacemaker is configured to manage the VIP. On 12/02/2012 01:59 PM, Hermes Flying wrote: But the VIP service is in coordination with corosync right? I mean when you say: if the node hosting the VIP fails, pacemaker may try to restart it or it might relocate it, depending on how you've configured things What does failure mean? That the service crashed OR also that corosync failed (so can not reach rest of nodes)? *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 8:46 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes As I said; each _service_ can have the concept of primary, just not pacemaker itself. I gave an example earlier; Pacemkaer might have two services; * DRBD, active on all nodes. * VIP, active on one node only. In this example, the DRBD service is Active/Active. If it fails on a given node, it will try to restart. If that fails, it will *not* relocate. Here, there is no primary. The VIP on the other hand runs on one node at a time only. Generally it will start on the first active node, but you might configure it to prefer one node. If that preferred node comes online later, pacemaker will migrate it. If there is no preferred node, then the VIP will stay where it is. If the node hosting the VIP fails, pacemaker may try to restart it or it might relocate it, depending on how you've configured things. In this case, the VIP service has the concept of primary, though it's better to think of it as Active. Make sense? On 12/02/2012 01:35 PM, Hermes Flying wrote: Hi, So you are saying I should not use the notion of primary ok. When I have 3 nodes, won't 1 node have the VIP? How is this node defined in Pacemaker's terminology if primary is inappropriate? Best Regards *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 8:22 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 AM, Hermes Flying wrote: Hi, For a cluster with 2 nodes I was explained what would happen. The other node will take over using fencing. It will take over *after* fencing. Two separate concepts. Fencing ensures that a lost node is truly gone and not just partitioned. Once fencing succeeds and the lost node is known to be down, _then_ recovery of service(s) that had been running on the victim will begin. But in clusters with 3+ nodes what happens when corosync fails? I assume that if the communication fails with the primary, all other nodes consider themselves eligible to become primaries. Is this the case? Corosync failing will be treated as a failure in the node and the node will be removed and fenced. Any services that had been running on it may or may not be recovered, depending on the rules defined for that given service. If it is recovered, then where it is restarted again depends on how each service
Re: [Linux-HA] Corosync on cluster with 3+ nodes
Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 9:52 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:27 PM, Hermes Flying wrote: When you say: So if corosync dies, pacemaker loses it's foundation and falls over dead. You mean pacemaker instance in all of nodes, right? No, just the pacemaker instance on the machine that lost corosync. Also (my emphasis ) The other node(s) in the cluster will declare the node as failed, fence it and then decide if any services need to be restarted Which node are your talking about? Itself? It will try to restart itself if corosync i.e. pacemaker dies? Whichever node that lost it's corosync version. If corosync dies, pacemaker can't and won't do anything. It's dead. It will not restart. The other node(s) in the cluster will declare that node lost and will fence it. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
If I have a requirement not to include external HW, is there any other way? I mean, I am not -by far- a linux expert, but how come it doesn't do a restart or halt? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 10:01 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 PM, Hermes Flying wrote: Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? Yes, there must be external hardware (out-of-band management counts as external, despite physically being on the server's mainboard). The most common fence device is IPMI, iLO, RSA, iDRAC and the like. Alternatives are switched PDUs, like APC's AP7900. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
I am an application developer and so far I didn't care about HW specs (not my responsibility). Now it seems I must care and so since I am not sure what HW is being recommended for servers I will verify tomorrow and let you know. My concern is that by going to pacemaker (although seems to be exactly what I need) is that we would need to request for new HW on migrate to new deployment. I don't even know if these devices are expensive. Did you ever had an issue with this? If I have problem with this, should I be looking into different solution? Any suggestion for alternative? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 10:19 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes It must be an external device. If, for example, the kernel crashes hard, or if you get a spinlock, the system may not respond to anything or a service may never stop, blocking a reboot. You can not trust that the system is accessible or functioning in any way. Fencing *must* be an external device. Period. What kind of servers do you have? On 12/02/2012 03:15 PM, Hermes Flying wrote: If I have a requirement not to include external HW, is there any other way? I mean, I am not -by far- a linux expert, but how come it doesn't do a restart or halt? *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:01 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 PM, Hermes Flying wrote: Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? Yes, there must be external hardware (out-of-band management counts as external, despite physically being on the server's mainboard). The most common fence device is IPMI, iLO, RSA, iDRAC and the like. Alternatives are switched PDUs, like APC's AP7900. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Corosync on cluster with 3+ nodes
Unfortunatelly this is not an option that anyone asks my opinion :( I will checkout the specs and let you know about the actual server. Thank your for your help! From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Sunday, December 2, 2012 10:36 PM Subject: Re: [Linux-HA] Corosync on cluster with 3+ nodes I don't know anything about your setup or office situation, so I can't speak to your specific issues. What I can say is that almost all actual servers have remote management; Fujitsu servers have IPMI, HP servers have iLO, Dell servers have DRAC, IBM servers have RSA and so on. Most generic servers have at least basic IPMI. So it's very likely that you will already have what you need to implement fencing. If you don't, then something like the APC AP7900 is usually about $500 canadian and will work as a fence device. I don't know what country you are in, so I can't speak more on availability, applicability or cost. You might seriously want to consider hiring someone to help you with this setup. Clustering is not a straight-forward topic. The cost to hire a consultant might save you enough time and headache that it's cheaper in the end than trying to do it yourself. On 12/02/2012 03:31 PM, Hermes Flying wrote: I am an application developer and so far I didn't care about HW specs (not my responsibility). Now it seems I must care and so since I am not sure what HW is being recommended for servers I will verify tomorrow and let you know. My concern is that by going to pacemaker (although seems to be exactly what I need) is that we would need to request for new HW on migrate to new deployment. I don't even know if these devices are expensive. Did you ever had an issue with this? If I have problem with this, should I be looking into different solution? Any suggestion for alternative? *From:* Digimer li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:19 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes It must be an external device. If, for example, the kernel crashes hard, or if you get a spinlock, the system may not respond to anything or a service may never stop, blocking a reboot. You can not trust that the system is accessible or functioning in any way. Fencing *must* be an external device. Period. What kind of servers do you have? On 12/02/2012 03:15 PM, Hermes Flying wrote: If I have a requirement not to include external HW, is there any other way? I mean, I am not -by far- a linux expert, but how come it doesn't do a restart or halt? *From:* Digimer li...@alteeve.ca mailto:li...@alteeve.ca *To:* Hermes Flying flyingher...@yahoo.com mailto:flyingher...@yahoo.com *Cc:* General Linux-HA mailing list linux-ha@lists.linux-ha.org mailto:linux-ha@lists.linux-ha.org *Sent:* Sunday, December 2, 2012 10:01 PM *Subject:* Re: [Linux-HA] Corosync on cluster with 3+ nodes On 12/02/2012 02:56 PM, Hermes Flying wrote: Clear! In order to do fence and crash a node, is there a specific HW requirement to do this? Yes, there must be external hardware (out-of-band management counts as external, despite physically being on the server's mainboard). The most common fence device is IPMI, iLO, RSA, iDRAC and the like. Alternatives are switched PDUs, like APC's AP7900. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Thank you for this! One last thing I need to clear out before digging into your configuration specs etc. Since the pacemaker is a fail-over system rather than a load-balancing system (like Red Hat) as you say, my understanding is that one of my nodes will have the VIP until: 1) Tomcat crashes and can not restart (dead for some reason) -- Pacemaker migrates VIP 2) The network communication with the outside network is cut off. -- Pacemaker migrates VIP If these (2) are valid (are they?) then that means that there is no primary/backup concept using pacemaker (since I will assign to one of my nodes to have the VIP and my installed Load Balancer will distribute the load among my 2 Tomcats) and as a result there can not be a split-brain. Yet you imply that split-brain can occur even with Pacemaker if I don't have fencing properly set. But how? Since it seems to me that Pacemaker does not have a notion of primary/backup. Or you mean something else with fail-over system? Additionally you say that the coordination of Pacemaker instances is done via corosync which is over network messages right? So what happens in the event of communication/network failure but only in the communication paths used for corosync coordination and not the communication path with the clients? Hope this question makes sense as I am new in your facilities. Thank you for your help! From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Saturday, December 1, 2012 2:35 AM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 11/30/2012 05:04 PM, Hermes Flying wrote: Hi, I am looking into using your facilities to have high availability on my system. I am trying to figure out some things. I hope you guys could help me. I am interested in knowing how pacemaker migrates a VIP and how a splitbrain situation is address by your facilities. To be specific: I am interested in the following setup: 2 linux machines. Each machine runs a load balancer and a Tomcat instance. If I understand correctly pacemaker will be responsible to assign the main VIP to one of the nodes. Pacemaker can handle virtual IP addresses, but it's more of a fail-over system, rather than a load-balancing, round-robin system. For load balancing, look at Red Hat's LVS. My questions are: 1)Will pacemaker monitor/restart the load balancers on each machine in case of crash? It can monitor/recover/relocate any service that uses init.d style scripts. If a script/service responds properly to stop, status and start, you're good to go. 2) How does pacemaker decide to migrate the VIP to the other node? At the most simple; When the machine hosting the VIP fails, it will relocate. You can control how, when and where the VIP fails back (look at 'resource stickiness'). 3) Do the pacemakers in each machine communicate? If yes how do you handle network failure? Could I end up with split-brain? Pacemaker uses corosync for cluster membership, quorum and fencing. A properly configured fence device (aka stonith), will prevent a split brain. If you disable or fail to properly setup fencing, split brains are possible and even likely. 4) Generally how is split-brain addressed using pacemaker? Fencing to prevent it. 5) Could pacemaker monitor Tomcat? If it supports stop, start and status, yes. As you can see I am interested in maintain quorum in a two-node configuration. If you can help me with this info to find a proper direction it would be much appreciated! Quorum needs to be disabled in a two-node cluster. This is fine with good fencing. To learn more, please see the documentation available here: http://clusterlabs.org/doc/ -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Thanks for your reply. First of all I didn't get if the VIP will migrate if Tomcat or load balancer also fails. It will right? Also if I understand this correctly, I can end up with VIP on both nodes if corosync fails due to network failure. And you suggest redundant communication paths to avoid this. But if I understand the problem, if the VIP runs in my linux-1 and pacemaker is somehow via corosync ready to take over on failure from linux-2, if there is a network failure (despite redundant communication paths, unless you guys recommend some specific topology to the people using Pacemaker that you are 100% full proof) how can you detect if the other node is actually crashed or just corosync fails? In this case won't the linux-2 also wakeup to take VIP? Could you please help me understand this? Thank you! From: David Coulson da...@davidcoulson.net To: Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org Cc: Digimer li...@alteeve.ca Sent: Saturday, December 1, 2012 2:46 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/1/12 5:46 AM, Hermes Flying wrote: Thank you for this! One last thing I need to clear out before digging into your configuration specs etc. Since the pacemaker is a fail-over system rather than a load-balancing system (like Red Hat) as you say, my understanding is that one of my nodes will have the VIP until: 1) Tomcat crashes and can not restart (dead for some reason) -- Pacemaker migrates VIP 2) The network communication with the outside network is cut off. -- Pacemaker migrates VIP If these (2) are valid (are they?) then that means that there is no primary/backup concept using pacemaker (since I will assign to one of my nodes to have the VIP and my installed Load Balancer will distribute the load among my 2 Tomcats) and as a result there can not be a split-brain. In the event of a split brain with Pacemaker, and you don't have any fencing configured, you will end up with your VIP running on both systems. Chances in your configuration it won't be a big deal since your router/firewall/whatever will learn the ARP of one system, so you'll end up routing traffic properly - But it will be unpredictable, and difficult to troubleshoot. Yet you imply that split-brain can occur even with Pacemaker if I don't have fencing properly set. But how? Since it seems to me that Pacemaker does not have a notion of primary/backup. Or you mean something else with fail-over system? For each resource, Pacemaker knows there is a node where it is running, and 'other' nodes where it is not running (but could if the node running it failed). So from a resource perceptive, there is an active node and one or more backups. Additionally you say that the coordination of Pacemaker instances is done via corosync which is over network messages right? So what happens in the event of communication/network failure but only in the communication paths used for corosync coordination and not the communication path with the clients? Hope this question makes sense as I am new in your facilities. Split brain. That's why you need a redundant communications network, plus you need fencing. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Great help! Please allow me to trouble you with one last question. If I get this, when I use fencing and the corosync fails then linux-2 will attempt to crash linux-1 and take over. At this point though linux-1 won't try to do anything right? Since it knows it is the primary, I mean. Then you say:Any resource previously running on linux-1 will be started on linux-2. Now at this point: By resource you mean only pacemaker and its related modules, right? Because I want Tomcat to be up and running and receiving requests in Linux-2 as well, which will be forwarded by load balancer of linux-1. Is this correct? Also in your setup of 2 NICs or 2 switches I assume that the idea is that the probability of split-brain due to network failure is very low right? Because I have read that it is not possible to avoid split-brain without adding a third node. But I may be misunderstanding this From: David Coulson da...@davidcoulson.net To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org; Digimer li...@alteeve.ca Sent: Saturday, December 1, 2012 3:26 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/1/12 8:21 AM, Hermes Flying wrote: Thanks for your reply. First of all I didn't get if the VIP will migrate if Tomcat or load balancer also fails. It will right? If you configure Pacemaker correctly, yes. Also if I understand this correctly, I can end up with VIP on both nodes if corosync fails due to network failure. And you suggest redundant communication paths to avoid this. But if I understand the problem, if the VIP runs in my linux-1 and pacemaker is somehow via corosync ready to take over on failure from linux-2, if there is a network failure (despite redundant communication paths, unless you guys recommend some specific topology to the people using Pacemaker that you are 100% full proof) how can you detect if the other node is actually crashed or just corosync fails? In this case won't the linux-2 also wakeup to take VIP? That is what fencing is for. If linux-1 goes offline from the perspective of linux-2, linux-2 will attempt to crash/power-cycle/power-off linux-1 to ensure it is really dead. Any resource previously running on linux-1 will be started on linux-2. Usually with a two node config I take two NICs on each box and connect them directly to the other one - Also would work if you had two separate switches you could run each path through. Then I use Linux NIC bonding to provide redundancy and run corosync over the bond interface. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Actually each Tomcat uses a back-end database that has the notion of primary/backup. I am trying to figure out if by using Pacemaker facilities I can avoid splitbrain in the database as well. So far from what you described I seem to get away with it meaning that by fencing, linux-1 will stop so the secondary database in lunux-2 will become primary. Am I on the right track here? If you have any recommendations for my setup (2 linux running: 2 LB/2Tomcat/2Databases) please let me know! Thank you for your time! From: David Coulson da...@davidcoulson.net To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org; Digimer li...@alteeve.ca Sent: Saturday, December 1, 2012 3:53 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/1/12 8:48 AM, Hermes Flying wrote: Great help! Please allow me to trouble you with one last question. If I get this, when I use fencing and the corosync fails then linux-2 will attempt to crash linux-1 and take over. At this point though linux-1 won't try to do anything right? Since it knows it is the primary, I mean. linux-1 will be powered off or crashed, so i think that speaks for itself. Then you say:Any resource previously running on linux-1 will be started on linux-2. Now at this point: By resource you mean only pacemaker and its related modules, right? Because I want Tomcat to be up and running and receiving requests in Linux-2 as well, which will be forwarded by load balancer of linux-1. Is this correct? I mean 'resources managed by pacemaker'. So if you VIP was running on linux-1, and it fails, and linux-2 fences it, the only place the VIP can run is linux-2. linux-1 is totally down. Also in your setup of 2 NICs or 2 switches I assume that the idea is that the probability of split-brain due to network failure is very low right? Because I have read that it is not possible to avoid split-brain without adding a third node. But I may be misunderstanding this A third node will eliminate split brain by definition, as quorum will only be obtained if a minimum of two nodes are available. If you have a diverse network configuration and good change management, you're probably not going to experience a split brain unless you have a substantial environment failure that will probably impact your client ability to access anything. Since you are not running shared storage, you're not going to experience data loss which is typically the biggest concern with split brain. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Dear David, thank you very much for you help. I will look into the details now that you have given me a direction, myself. Again thank you for your time! Best Regards From: David Coulson da...@davidcoulson.net To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org; Digimer li...@alteeve.ca Sent: Saturday, December 1, 2012 4:04 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker Already told you. If you are running two-node, make sure your fencing works and you have reliable connectivity between nodes. if that isn't good enough, add a third node. On 12/1/12 8:58 AM, Hermes Flying wrote: Actually each Tomcat uses a back-end database that has the notion of primary/backup. I am trying to figure out if by using Pacemaker facilities I can avoid splitbrain in the database as well. So far from what you described I seem to get away with it meaning that by fencing, linux-1 will stop so the secondary database in lunux-2 will become primary. Am I on the right track here? If you have any recommendations for my setup (2 linux running: 2 LB/2Tomcat/2Databases) please let me know! Thank you for your time! From: David Coulson da...@davidcoulson.net To: Hermes Flying flyingher...@yahoo.com Cc: General Linux-HA mailing list linux-ha@lists.linux-ha.org; Digimer li...@alteeve.ca Sent: Saturday, December 1, 2012 3:53 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/1/12 8:48 AM, Hermes Flying wrote: Great help! Please allow me to trouble you with one last question. If I get this, when I use fencing and the corosync fails then linux-2 will attempt to crash linux-1 and take over. At this point though linux-1 won't try to do anything right? Since it knows it is the primary, I mean. linux-1 will be powered off or crashed, so i think that speaks for itself. Then you say:Any resource previously running on linux-1 will be started on linux-2. Now at this point: By resource you mean only pacemaker and its related modules, right? Because I want Tomcat to be up and running and receiving requests in Linux-2 as well, which will be forwarded by load balancer of linux-1. Is this correct? I mean 'resources managed by pacemaker'. So if you VIP was running on linux-1, and it fails, and linux-2 fences it, the only place the VIP can run is linux-2. linux-1 is totally down. Also in your setup of 2 NICs or 2 switches I assume that the idea is that the probability of split-brain due to network failure is very low right? Because I have read that it is not possible to avoid split-brain without adding a third node. But I may be misunderstanding this A third node will eliminate split brain by definition, as quorum will only be obtained if a minimum of two nodes are available. If you have a diverse network configuration and good change management, you're probably not going to experience a split brain unless you have a substantial environment failure that will probably impact your client ability to access anything. Since you are not running shared storage, you're not going to experience data loss which is typically the biggest concern with split brain. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
So only Primary/Actice supported. Good to know. In your description of 3+nodes, they all try to become the primary. If a node does not have access to the external network (besides the corosync paths) does it still try to become primary? From: Digimer li...@alteeve.ca To: David Coulson da...@davidcoulson.net Cc: Hermes Flying flyingher...@yahoo.com; General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Saturday, December 1, 2012 7:20 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/01/2012 08:53 AM, David Coulson wrote: On 12/1/12 8:48 AM, Hermes Flying wrote: Great help! Please allow me to trouble you with one last question. If I get this, when I use fencing and the corosync fails then linux-2 will attempt to crash linux-1 and take over. At this point though linux-1 won't try to do anything right? Since it knows it is the primary, I mean. linux-1 will be powered off or crashed, so i think that speaks for itself. Pacemaker does not inherently have a concept of Primary/Secondary. It can be configured to operate in such a mode, or it can be active/active, n-1, n-n, etc. It all comes down to how each given resource is configured. You can have multiple resources in multiple configurations on the same instance. For example, you may have a DRBD resource set to run on both nodes (active/active), and a VM that runs on only one node at a time (active/passive) that uses the DRBD storage. So, given this, there is no inherent mechanism to say always fence node X. So imagine you have two nodes and they lose communication; Both will think the other has failed and both will try to fence it's peer. The classic analogy is an old west shoot-out; Fastest node lives, slower nodes gets powered off. You can help ensure one node has a better chance of winning by setting a delay on it. So it the above config, if you had set a 5 second delay against node 1, then it would get a 5 second head start in fencing node 2 (node 2 will wait the set time before trying to fence node 1). In this way, you can help ensure one node lives in such a case. Sometime a delay is needed in any case because in some fence methods, like IPMI-based devices, it's technically possible that both nodes will get their fence call out before dieing, leaving both nodes powered off. Setting a delay will protect against this. Then you say:Any resource previously running on linux-1 will be started on linux-2. Now at this point: By resource you mean only pacemaker and its related modules, right? Because I want Tomcat to be up and running and receiving requests in Linux-2 as well, which will be forwarded by load balancer of linux-1. Is this correct? I mean 'resources managed by pacemaker'. So if you VIP was running on linux-1, and it fails, and linux-2 fences it, the only place the VIP can run is linux-2. linux-1 is totally down. Also in your setup of 2 NICs or 2 switches I assume that the idea is that the probability of split-brain due to network failure is very low right? Because I have read that it is not possible to avoid split-brain without adding a third node. But I may be misunderstanding this A third node will eliminate split brain by definition, as quorum will only be obtained if a minimum of two nodes are available. If you have a diverse network configuration and good change management, you're probably not going to experience a split brain unless you have a substantial environment failure that will probably impact your client ability to access anything. Since you are not running shared storage, you're not going to experience data loss which is typically the biggest concern with split brain. A couple of notes; Only mode=1 bonding (active/passive) works reliably. No other mode is supported (and I've tested all and found all other modes would fail). Quorum comes into play with 3+ nodes, but it does not eliminate the need for fencing. Imagine that a node was in the middle of a write to shared storage and then hung totally. The other two nodes declare it as failed, don't fence it, and proceed to clean up the shared storage to return it to a consistent state. Time passes and new data gets written to the same area of shared storage. Then the hung node thaws, has no idea time has passed, and just finishes the write thinking it's locks are still valid and that it still has quorum. Voila! Corrupt storage, despite quorum. Clustering all all about protecting against failures. Hand-waving a failure as unlikely is not a good practice in HA clustering. If it's possible, plan for it. :) -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org
Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
You say that after fencing and failure from init.d restart/status scripts it will try to start Tomcat on the other node? But I need Tomcat running on both nodes! The setup I am asking is: HAProxy1---Tomcat-1---DB-1 (Linux-1) | +++^ | |-+| | + | | HAProxy2Tomcat-2DB-2 (Linux-2) I need Tomcat1-Tomcat2 processing requests concurrently and DB-1, DB-2 running in active/passing mode. So you are saying I can only monitor the load balancer (e.g. HAProxy) via Pacemaker? Not Tomcat or my DB? From: Digimer li...@alteeve.ca To: Hermes Flying flyingher...@yahoo.com Cc: David Coulson da...@davidcoulson.net; General Linux-HA mailing list linux-ha@lists.linux-ha.org Sent: Saturday, December 1, 2012 7:25 PM Subject: Re: [Linux-HA] Some help on understanding how HA issues are addressed by pacemaker On 12/01/2012 08:58 AM, Hermes Flying wrote: Actually each Tomcat uses a back-end database that has the notion of primary/backup. I am trying to figure out if by using Pacemaker facilities I can avoid splitbrain in the database as well. So far from what you described I seem to get away with it meaning that by fencing, linux-1 will stop so the secondary database in lunux-2 will become primary. Am I on the right track here? If you have any recommendations for my setup (2 linux running: 2 LB/2Tomcat/2Databases) please let me know! Thank you for your time! You should check to see if there is a tomcat-specific resource agent for pacemaker. If there is, you should be able to do advanced checks. I can't speak to tomcat further as I do not use it. Speaking in general terms; Pacemaker will not normally fence a node simply because a resource on it has failed. Fencing (and quorum) are membership-level tools. If the node and it's cluster communication stack is healthy, it will be left alone. A resource failure will be addressed directly (by restarting it, relocating it, etc). So if you can do '/etc/init.d/tomcat status' and get a result, then pacemaker can tell when the status goes bad. At that time, it will call a 'stop' - 'start' - 'status' to see if it's ok. If it's not, it will try again or eventually simple stop it and try starting tomcat on the other node. (this I am guessing, as I mentioned earlier, I come from the rhcs world and have minimal pacemaker-specific experience). So if you tied your virtual IP and tomcat resources together in pacemaker, then the failure of one will trigger the restart or migration of both. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Corosync on cluster with 3+ nodes
Hi, For a cluster with 2 nodes I was explained what would happen. The other node will take over using fencing. But in clusters with 3+ nodes what happens when corosync fails? I assume that if the communication fails with the primary, all other nodes consider themselves eligible to become primaries. Is this the case? 1)If a node has problem communicating with the primary AND has network problem with the rest of the network (clients) does it still try to become the primary (try to kill other nodes?) 2) In practice if the corosync fails but the primary is still up and running and serving requests, is primary attempted to be killed by the other nodes?Or you use some other way to figure out that this is a network failure, primary has not crashed? 3)Finally on corosync failure I assume the primary does nothing, as it does not care about the backups. Is this correct? Thank you! ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Some help on understanding how HA issues are addressed by pacemaker
Hi, I am looking into using your facilities to have high availability on my system. I am trying to figure out some things. I hope you guys could help me. I am interested in knowing how pacemaker migrates a VIP and how a splitbrain situation is address by your facilities. To be specific: I am interested in the following setup: 2 linux machines. Each machine runs a load balancer and a Tomcat instance. If I understand correctly pacemaker will be responsible to assign the main VIP to one of the nodes. My questions are: 1)Will pacemaker monitor/restart the load balancers on each machine in case of crash? 2) How does pacemaker decide to migrate the VIP to the other node? 3) Do the pacemakers in each machine communicate? If yes how do you handle network failure? Could I end up with split-brain? 4) Generally how is split-brain addressed using pacemaker? 5) Could pacemaker monitor Tomcat? As you can see I am interested in maintain quorum in a two-node configuration. If you can help me with this info to find a proper direction it would be much appreciated! Thank you ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
