Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
On Mon, 2018-03-26 at 15:42 +, Sam Gardner wrote: > Thanks, Andrei and Alberto. > > Alberto, I will look into the node-constraint parameters, though I > suspect Andrei is correct - my "base" resource is DRBDFS in this > case, and the issue I'm seeing is that a failure in my secondary > resources does not cause the other secondary resources nor the "base" > resource to move to the other node. > > Andrei, I have no restrictions on the particulars of the rules that > I'm putting in place - I can completely discard the rules that I have > implemented already. > > Here's a simple diagram: > https://imgur.com/a/5LTmJ > > These are my restrictions: > 1) If any of DRBD-Master, DRBDFS, INIF-Master, or OUTIF-Master moves > to D2, all other resources should move to D2. > 2) If DRBDFS or DRBD-Master cannot run on either D1 or D2, all other > resources should be stopped. > 3) If INIF-Master or OUTIF-Master cannot run on either D1 or D2, no > other resources should be stopped. 4) Keep INIF-Master with a working IN interface and OUTIF-Master with a working OUT interface. One problem is that 4) conflicts with 1). It's possible for INIF to be working on only one node and OUTIF to be working only on the other node. I'm thinking you want something like this: * Don't constrain DRBD-Master * Colocate DRBDFS with DRBD-Master, +INFINITY * Colocate INIF-Master and OUTIF-Master with DRBDFS, +INFINITY (using separate individual constraints) * Keep your INIF/OUTIF rules, but use a finite negative score. That way, they *must* stay with DRBFDS and DRBD-Master (due to the previous constraints), but will *prefer* a node where their interface is up. If one of them is more important than the other, give it a stronger score, to break the tie when one interface is up on each node. * Order DRBDFS start after DRBD-Master promote No order is necessary on INIF/OUTIF, since an IP can work regardless of the file system. That should meet your intentions. > This sounds like a particular constraint that may not be possible to > do per our discussions in this thread. > > I can get pretty close with a workaround - I'm using ethmonitor on > the Master/Slave resources as you can see in the config, so if I > create new "heartbeat:Dummy" active resources with the same > ethmonitor location constraint, unplugging the interface will move > everything over. > > However, a failure of a different type on the master/slave VIPs that > would not also be apparent on the dummy base resource would not cause > a failover of the entire group, which isn't ideal (though admittedly > unlikely in this particular use case). > > Thanks much for all of the help, > -- > Sam Gardner > Trustwave | SMART SECURITY ON DEMAND > > > > > > > > > On 3/25/18, 6:06 AM, "Users on behalf of Andrei Borzenkov" n...@clusterlabs.org on behalf of arvidj...@gmail.com> wrote: > > > 25.03.2018 10:21, Alberto Mijares пишет: > > > On Sat, Mar 24, 2018 at 2:16 PM, Andrei Borzenkov > > l.com> wrote: > > > > 23.03.2018 20:42, Sam Gardner пишет: > > > > > Thanks, Ken. > > > > > > > > > > I just want all master-mode resources to be running wherever > > > > > DRBDFS is running (essentially). If the cluster detects that > > > > > any of the master-mode resources can't run on the current > > > > > node (but can run on the other per ethmon), all other master- > > > > > mode resources as well as DRBDFS should move over to the > > > > > other node. > > > > > > > > > > The current set of constraints I have will let DRBDFS move to > > > > > the standby node and "take" the Master mode resources with > > > > > it, but the Master mode resources failing over to the other > > > > > node won't take the other Master resources or DRBDFS. > > > > > > > > > > > > > I do not think it is possible. There is no way to express > > > > symmetrical > > > > colocation rule like "always run A and B together". You start > > > > with A and > > > > place B relative to A; but then A is not affected by B's state. > > > > Attempting now to place A relative to B will result in a loop > > > > and is > > > > ignored. See also old discussion: > > > > > > > > > > > > > It is possible. Check this thread > > > https://scanmail.trustwave.com/?c=4062&d=qYK32i8YnPIdkrPQRoURDTOy > > > qGVIytWo2- > > > H2bJ__2w&s=5&u=https%3a%2f%2flists%2eclusterlabs%2eorg%2fpipermai > > > l%2fusers%2f2017-November%2f006788%2ehtml > > > > > > > I do not see how it answers the question. It explains how to use > > other > > criteria than node name for colocating resources, but it does not > > change > > basic fact that colocating is asymmetrical. Actually this thread > > explicitly suggests "Pick one resource as your base resource that > > everything else should go along with". > > > > If you you actually have configuration that somehow implements > > symmetrical colocation between resources, I would appreciate if you > > could post your configuration. > > > > Regarding the original problem, the root cause is slightly > >
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
>> >> It is possible. Check this thread >> https://lists.clusterlabs.org/pipermail/users/2017-November/006788.html >> > > I do not see how it answers the question. It explains how to use other > criteria than node name for colocating resources, but it does not change > basic fact that colocating is asymmetrical. Actually this thread > explicitly suggests "Pick one resource as your base resource that > everything else should go along with". > To be honest, I didn't fully read the OP. I just read a fragment and it looked to me like the grouping constraint could work. Sorry, guys. Please ignore my post. Best regards, Alberto Mijares ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
Thanks, Andrei and Alberto. Alberto, I will look into the node-constraint parameters, though I suspect Andrei is correct - my "base" resource is DRBDFS in this case, and the issue I'm seeing is that a failure in my secondary resources does not cause the other secondary resources nor the "base" resource to move to the other node. Andrei, I have no restrictions on the particulars of the rules that I'm putting in place - I can completely discard the rules that I have implemented already. Here's a simple diagram: https://imgur.com/a/5LTmJ These are my restrictions: 1) If any of DRBD-Master, DRBDFS, INIF-Master, or OUTIF-Master moves to D2, all other resources should move to D2. 2) If DRBDFS or DRBD-Master cannot run on either D1 or D2, all other resources should be stopped. 3) If INIF-Master or OUTIF-Master cannot run on either D1 or D2, no other resources should be stopped. This sounds like a particular constraint that may not be possible to do per our discussions in this thread. I can get pretty close with a workaround - I'm using ethmonitor on the Master/Slave resources as you can see in the config, so if I create new "heartbeat:Dummy" active resources with the same ethmonitor location constraint, unplugging the interface will move everything over. However, a failure of a different type on the master/slave VIPs that would not also be apparent on the dummy base resource would not cause a failover of the entire group, which isn't ideal (though admittedly unlikely in this particular use case). Thanks much for all of the help, -- Sam Gardner Trustwave | SMART SECURITY ON DEMAND On 3/25/18, 6:06 AM, "Users on behalf of Andrei Borzenkov" wrote: >25.03.2018 10:21, Alberto Mijares пишет: >> On Sat, Mar 24, 2018 at 2:16 PM, Andrei Borzenkov >> wrote: >>> 23.03.2018 20:42, Sam Gardner пишет: Thanks, Ken. I just want all master-mode resources to be running wherever DRBDFS is running (essentially). If the cluster detects that any of the master-mode resources can't run on the current node (but can run on the other per ethmon), all other master-mode resources as well as DRBDFS should move over to the other node. The current set of constraints I have will let DRBDFS move to the standby node and "take" the Master mode resources with it, but the Master mode resources failing over to the other node won't take the other Master resources or DRBDFS. >>> >>> I do not think it is possible. There is no way to express symmetrical >>> colocation rule like "always run A and B together". You start with A and >>> place B relative to A; but then A is not affected by B's state. >>> Attempting now to place A relative to B will result in a loop and is >>> ignored. See also old discussion: >>> >> >> >> It is possible. Check this thread >> https://scanmail.trustwave.com/?c=4062&d=qYK32i8YnPIdkrPQRoURDTOyqGVIytWo2-H2bJ__2w&s=5&u=https%3a%2f%2flists%2eclusterlabs%2eorg%2fpipermail%2fusers%2f2017-November%2f006788%2ehtml >> > >I do not see how it answers the question. It explains how to use other >criteria than node name for colocating resources, but it does not change >basic fact that colocating is asymmetrical. Actually this thread >explicitly suggests "Pick one resource as your base resource that >everything else should go along with". > >If you you actually have configuration that somehow implements >symmetrical colocation between resources, I would appreciate if you >could post your configuration. > >Regarding the original problem, the root cause is slightly different though. > >@Sam, the behavior you describe is correct for your constraints that you >show. When colocating with resource set, all resources in the set must >be active on the same node. It means that in your case of > > id="pcs_rsc_colocation_set_drbdfs_set_drbd.master_inside-interface-sameip.master_outside-interface-sameip.master" >score="INFINITY"> > > > > id="pcs_rsc_set_drbd.master_inside-interface-sameip.master_outside-interface-sameip.master" >role="Master" sequential="false"> > > > > > > >if one IP resource (master) is moved to another node, dependent resource >(drbdfs) simply cannot run anywhere. > >Before discussing low level pacemaker implementation you really need to >have high level model of resources relationship. On one hand you >apparently intend to always run everything on the same node - on the >other hand you have two rules that independently decide where to place >two resources. That does not fit together. >___ >Users mailing list: Users@clusterlabs.org >https://scanmail.trustwave.com/?c=4062&d=qYK32i8YnPIdkrPQRoURDTOyqGVIytWo2-H0aZn72g&s=5&u=https%3a%2f%2flists%2eclusterlabs%2eorg%2fmailman%2flistinfo%2fusers > >Project Home: >http://scanmail.trustwave.com/?c=4062&d=qYK32i8YnPIdkrPQRoURDTOyqGVIytWo27bwaZCrhg&s=5&u=http%3a%2f%2fwww
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
25.03.2018 10:21, Alberto Mijares пишет: > On Sat, Mar 24, 2018 at 2:16 PM, Andrei Borzenkov wrote: >> 23.03.2018 20:42, Sam Gardner пишет: >>> Thanks, Ken. >>> >>> I just want all master-mode resources to be running wherever DRBDFS is >>> running (essentially). If the cluster detects that any of the master-mode >>> resources can't run on the current node (but can run on the other per >>> ethmon), all other master-mode resources as well as DRBDFS should move over >>> to the other node. >>> >>> The current set of constraints I have will let DRBDFS move to the standby >>> node and "take" the Master mode resources with it, but the Master mode >>> resources failing over to the other node won't take the other Master >>> resources or DRBDFS. >>> >> >> I do not think it is possible. There is no way to express symmetrical >> colocation rule like "always run A and B together". You start with A and >> place B relative to A; but then A is not affected by B's state. >> Attempting now to place A relative to B will result in a loop and is >> ignored. See also old discussion: >> > > > It is possible. Check this thread > https://lists.clusterlabs.org/pipermail/users/2017-November/006788.html > I do not see how it answers the question. It explains how to use other criteria than node name for colocating resources, but it does not change basic fact that colocating is asymmetrical. Actually this thread explicitly suggests "Pick one resource as your base resource that everything else should go along with". If you you actually have configuration that somehow implements symmetrical colocation between resources, I would appreciate if you could post your configuration. Regarding the original problem, the root cause is slightly different though. @Sam, the behavior you describe is correct for your constraints that you show. When colocating with resource set, all resources in the set must be active on the same node. It means that in your case of if one IP resource (master) is moved to another node, dependent resource (drbdfs) simply cannot run anywhere. Before discussing low level pacemaker implementation you really need to have high level model of resources relationship. On one hand you apparently intend to always run everything on the same node - on the other hand you have two rules that independently decide where to place two resources. That does not fit together. ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
On Sat, Mar 24, 2018 at 2:16 PM, Andrei Borzenkov wrote: > 23.03.2018 20:42, Sam Gardner пишет: >> Thanks, Ken. >> >> I just want all master-mode resources to be running wherever DRBDFS is >> running (essentially). If the cluster detects that any of the master-mode >> resources can't run on the current node (but can run on the other per >> ethmon), all other master-mode resources as well as DRBDFS should move over >> to the other node. >> >> The current set of constraints I have will let DRBDFS move to the standby >> node and "take" the Master mode resources with it, but the Master mode >> resources failing over to the other node won't take the other Master >> resources or DRBDFS. >> > > I do not think it is possible. There is no way to express symmetrical > colocation rule like "always run A and B together". You start with A and > place B relative to A; but then A is not affected by B's state. > Attempting now to place A relative to B will result in a loop and is > ignored. See also old discussion: > It is possible. Check this thread https://lists.clusterlabs.org/pipermail/users/2017-November/006788.html Best regards, Alberto Mijares ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
23.03.2018 20:42, Sam Gardner пишет: > Thanks, Ken. > > I just want all master-mode resources to be running wherever DRBDFS is > running (essentially). If the cluster detects that any of the master-mode > resources can't run on the current node (but can run on the other per > ethmon), all other master-mode resources as well as DRBDFS should move over > to the other node. > > The current set of constraints I have will let DRBDFS move to the standby > node and "take" the Master mode resources with it, but the Master mode > resources failing over to the other node won't take the other Master > resources or DRBDFS. > I do not think it is possible. There is no way to express symmetrical colocation rule like "always run A and B together". You start with A and place B relative to A; but then A is not affected by B's state. Attempting now to place A relative to B will result in a loop and is ignored. See also old discussion: https://lists.gt.net/linuxha/pacemaker/68099 > As a side note, there are other resources I have in play (some active/passive > like DRBDFS, some Master/Slave like the ship resources) that are related, but > not shown here - I'm just having a hard time reasoning about the generalized > form that my constraints should take to make this sort of thing work. > ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
The shared-ip resource agents are our own custom RA - they do what we want and the failover mechanism for the single resource works fine as far as I can tell - using conntrackd or some other non-IP master/slave resource would be the same sort of failure that we're running into because the constraints aren't set up correctly. -- Sam Gardner Trustwave | SMART SECURITY ON DEMAND On 3/23/18, 12:42 PM, "Users on behalf of Sam Gardner" wrote: >Thanks, Ken. > >I just want all master-mode resources to be running wherever DRBDFS is running >(essentially). If the cluster detects that any of the master-mode resources >can't run on the current node (but can run on the other per ethmon), all other >master-mode resources as well as DRBDFS should move over to the other node. > >The current set of constraints I have will let DRBDFS move to the standby node >and "take" the Master mode resources with it, but the Master mode resources >failing over to the other node won't take the other Master resources or DRBDFS. > >As a side note, there are other resources I have in play (some active/passive >like DRBDFS, some Master/Slave like the ship resources) that are related, but >not shown here - I'm just having a hard time reasoning about the generalized >form that my constraints should take to make this sort of thing work. >-- >Sam Gardner >Trustwave | SMART SECURITY ON DEMAND > > > > > > > > >On 3/23/18, 12:34 PM, "Users on behalf of Ken Gaillot" > wrote: > >>On Tue, 2018-03-20 at 16:34 +, Sam Gardner wrote: >>> Hi All - >>> >>> I've implemented a simple two-node cluster with DRBD and a couple of >>> network-based Master/Slave resources. >>> >>> Using the ethmonitor RA, I set up failover whenever the >>> Master/Primary node loses link on the specified ethernet physical >>> device by constraining the Master role only on nodes where the ethmon >>> variable is "1". >>> >>> Something is going wrong with my colocation constraint, however - if >>> I set up the DRBDFS resource to monitor link on eth1, unplugging eth1 >>> on the Primary node causes a failover as expected - all Master >>> resources are demoted to "slave" and promoted on the opposite node, >>> and the "normal" DRBDFS moves to the other node as expected. >>> >>> However, if I put the same ethmonitor constraint on the network-based >>> Master/Slave resource, only that specific resource fails over - >>> DRBDFS stays in the same location (though it stops) as do the other >>> Master/Slave resources. >>> >>> This *smells* like a constraints issue to me - does anyone know what >>> I might be doing wrong? >>> >>> PCS before: >>> Cluster name: >>> http://scanmail.trustwave.com/?c=4062&d=mby12vU1sHtzTmhG_RQD80FeoJfoSXp5PKS2oHmF9g&s=5&u=http%3a%2f%2fnode1%2ehostname%2ecom%5fnode2%2ehostname%2ecom >>> Stack: corosync >>> Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) >>> - partition with quorum >>> Last updated: Tue Mar 20 16:25:47 2018 >>> Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on >>> node2.hostname.com_0 >>> >>> 2 nodes configured >>> 11 resources configured >>> >>> Online: [ node1.hostname.com_0 node2.hostname.com_0 ] >>> >>> Full list of resources: >>> >>> Master/Slave Set: drbd.master [drbd.slave] >>> Masters: [ node1.hostname.com_0 ] >>> Slaves: [ node2.hostname.com_0 ] >>> drbdfs (ocf::heartbeat:Filesystem):Started node1.hostname.com_0 >>> Master/Slave Set: inside-interface-sameip.master [inside-interface- >>> sameip.slave] >>> Masters: [ node1.hostname.com_0 ] >>> Slaves: [ node2.hostname.com_0 ] >>> Master/Slave Set: outside-interface-sameip.master [outside- >>> interface-sameip.slave] >>> Masters: [ node1.hostname.com_0 ] >>> Slaves: [ node2.hostname.com_0 ] >>> Clone Set: monitor-eth1-clone [monitor-eth1] >>> Started: [ node1.hostname.com_0 node2.hostname.com_0 ] >>> Clone Set: monitor-eth2-clone [monitor-eth2] >>> Started: [ node1.hostname.com_0 node2.hostname.com_0 ] >> >>What agent are the two IP resources using? I'm not familiar with any IP >>resource agents that are master/slave clones. >> >>> Daemon Status: >>> corosync: active/enabled >>> pacemaker: active/enabled >>> pcsd: inactive/disabled >>> >>> PCS after: >>> Cluster name: >>> http://scanmail.trustwave.com/?c=4062&d=mby12vU1sHtzTmhG_RQD80FeoJfoSXp5PKS2oHmF9g&s=5&u=http%3a%2f%2fnode1%2ehostname%2ecom%5fnode2%2ehostname%2ecom >>> Stack: corosync >>> Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) >>> - partition with quorum >>> Last updated: Tue Mar 20 16:29:40 2018 >>> Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on >>> node2.hostname.com_0 >>> >>> 2 nodes configured >>> 11 resources configured >>> >>> Online: [ node1.hostname.com_0 node2.hostname.com_0 ] >>> >>> Full list of resources: >>> >>> Master/Slave Set: drbd.master [drbd.slave] >>> Masters: [ node1.hostname.com_0 ] >>> Slaves: [ node2.hostname.com_0 ] >>> drbdfs
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
Thanks, Ken. I just want all master-mode resources to be running wherever DRBDFS is running (essentially). If the cluster detects that any of the master-mode resources can't run on the current node (but can run on the other per ethmon), all other master-mode resources as well as DRBDFS should move over to the other node. The current set of constraints I have will let DRBDFS move to the standby node and "take" the Master mode resources with it, but the Master mode resources failing over to the other node won't take the other Master resources or DRBDFS. As a side note, there are other resources I have in play (some active/passive like DRBDFS, some Master/Slave like the ship resources) that are related, but not shown here - I'm just having a hard time reasoning about the generalized form that my constraints should take to make this sort of thing work. -- Sam Gardner Trustwave | SMART SECURITY ON DEMAND On 3/23/18, 12:34 PM, "Users on behalf of Ken Gaillot" wrote: >On Tue, 2018-03-20 at 16:34 +, Sam Gardner wrote: >> Hi All - >> >> I've implemented a simple two-node cluster with DRBD and a couple of >> network-based Master/Slave resources. >> >> Using the ethmonitor RA, I set up failover whenever the >> Master/Primary node loses link on the specified ethernet physical >> device by constraining the Master role only on nodes where the ethmon >> variable is "1". >> >> Something is going wrong with my colocation constraint, however - if >> I set up the DRBDFS resource to monitor link on eth1, unplugging eth1 >> on the Primary node causes a failover as expected - all Master >> resources are demoted to "slave" and promoted on the opposite node, >> and the "normal" DRBDFS moves to the other node as expected. >> >> However, if I put the same ethmonitor constraint on the network-based >> Master/Slave resource, only that specific resource fails over - >> DRBDFS stays in the same location (though it stops) as do the other >> Master/Slave resources. >> >> This *smells* like a constraints issue to me - does anyone know what >> I might be doing wrong? >> >> PCS before: >> Cluster name: >> http://scanmail.trustwave.com/?c=4062&d=o7q12l4ebrGiubIsdGUsPkgY15QLdpI4OU2Ogkq_xg&s=5&u=http%3a%2f%2fnode1%2ehostname%2ecom%5fnode2%2ehostname%2ecom >> Stack: corosync >> Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) >> - partition with quorum >> Last updated: Tue Mar 20 16:25:47 2018 >> Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on >> node2.hostname.com_0 >> >> 2 nodes configured >> 11 resources configured >> >> Online: [ node1.hostname.com_0 node2.hostname.com_0 ] >> >> Full list of resources: >> >> Master/Slave Set: drbd.master [drbd.slave] >> Masters: [ node1.hostname.com_0 ] >> Slaves: [ node2.hostname.com_0 ] >> drbdfs (ocf::heartbeat:Filesystem):Started node1.hostname.com_0 >> Master/Slave Set: inside-interface-sameip.master [inside-interface- >> sameip.slave] >> Masters: [ node1.hostname.com_0 ] >> Slaves: [ node2.hostname.com_0 ] >> Master/Slave Set: outside-interface-sameip.master [outside- >> interface-sameip.slave] >> Masters: [ node1.hostname.com_0 ] >> Slaves: [ node2.hostname.com_0 ] >> Clone Set: monitor-eth1-clone [monitor-eth1] >> Started: [ node1.hostname.com_0 node2.hostname.com_0 ] >> Clone Set: monitor-eth2-clone [monitor-eth2] >> Started: [ node1.hostname.com_0 node2.hostname.com_0 ] > >What agent are the two IP resources using? I'm not familiar with any IP >resource agents that are master/slave clones. > >> Daemon Status: >> corosync: active/enabled >> pacemaker: active/enabled >> pcsd: inactive/disabled >> >> PCS after: >> Cluster name: >> http://scanmail.trustwave.com/?c=4062&d=o7q12l4ebrGiubIsdGUsPkgY15QLdpI4OU2Ogkq_xg&s=5&u=http%3a%2f%2fnode1%2ehostname%2ecom%5fnode2%2ehostname%2ecom >> Stack: corosync >> Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) >> - partition with quorum >> Last updated: Tue Mar 20 16:29:40 2018 >> Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on >> node2.hostname.com_0 >> >> 2 nodes configured >> 11 resources configured >> >> Online: [ node1.hostname.com_0 node2.hostname.com_0 ] >> >> Full list of resources: >> >> Master/Slave Set: drbd.master [drbd.slave] >> Masters: [ node1.hostname.com_0 ] >> Slaves: [ node2.hostname.com_0 ] >> drbdfs (ocf::heartbeat:Filesystem):Stopped >> Master/Slave Set: inside-interface-sameip.master [inside-interface- >> sameip.slave] >> Masters: [ node2.hostname.com_0 ] >> Stopped: [ node1.hostname.com_0 ] >> Master/Slave Set: outside-interface-sameip.master [outside- >> interface-sameip.slave] >> Masters: [ node1.hostname.com_0 ] >> Slaves: [ node2.hostname.com_0 ] >> Clone Set: monitor-eth1-clone [monitor-eth1] >> Started: [ node1.hostname.com_0 node2.hostname.com_0 ] >> Clone Set: monitor-eth2-clone [monitor-eth2] >> Started: [ node1.ho
Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources
On Tue, 2018-03-20 at 16:34 +, Sam Gardner wrote: > Hi All - > > I've implemented a simple two-node cluster with DRBD and a couple of > network-based Master/Slave resources. > > Using the ethmonitor RA, I set up failover whenever the > Master/Primary node loses link on the specified ethernet physical > device by constraining the Master role only on nodes where the ethmon > variable is "1". > > Something is going wrong with my colocation constraint, however - if > I set up the DRBDFS resource to monitor link on eth1, unplugging eth1 > on the Primary node causes a failover as expected - all Master > resources are demoted to "slave" and promoted on the opposite node, > and the "normal" DRBDFS moves to the other node as expected. > > However, if I put the same ethmonitor constraint on the network-based > Master/Slave resource, only that specific resource fails over - > DRBDFS stays in the same location (though it stops) as do the other > Master/Slave resources. > > This *smells* like a constraints issue to me - does anyone know what > I might be doing wrong? > > PCS before: > Cluster name: node1.hostname.com_node2.hostname.com > Stack: corosync > Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) > - partition with quorum > Last updated: Tue Mar 20 16:25:47 2018 > Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on > node2.hostname.com_0 > > 2 nodes configured > 11 resources configured > > Online: [ node1.hostname.com_0 node2.hostname.com_0 ] > > Full list of resources: > > Master/Slave Set: drbd.master [drbd.slave] > Masters: [ node1.hostname.com_0 ] > Slaves: [ node2.hostname.com_0 ] > drbdfs (ocf::heartbeat:Filesystem): Started node1.hostname.com_0 > Master/Slave Set: inside-interface-sameip.master [inside-interface- > sameip.slave] > Masters: [ node1.hostname.com_0 ] > Slaves: [ node2.hostname.com_0 ] > Master/Slave Set: outside-interface-sameip.master [outside- > interface-sameip.slave] > Masters: [ node1.hostname.com_0 ] > Slaves: [ node2.hostname.com_0 ] > Clone Set: monitor-eth1-clone [monitor-eth1] > Started: [ node1.hostname.com_0 node2.hostname.com_0 ] > Clone Set: monitor-eth2-clone [monitor-eth2] > Started: [ node1.hostname.com_0 node2.hostname.com_0 ] What agent are the two IP resources using? I'm not familiar with any IP resource agents that are master/slave clones. > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: inactive/disabled > > PCS after: > Cluster name: node1.hostname.com_node2.hostname.com > Stack: corosync > Current DC: node2.hostname.com_0 (version 1.1.16-12.el7_4.4-94ff4df) > - partition with quorum > Last updated: Tue Mar 20 16:29:40 2018 > Last change: Tue Mar 20 16:00:33 2018 by hacluster via crmd on > node2.hostname.com_0 > > 2 nodes configured > 11 resources configured > > Online: [ node1.hostname.com_0 node2.hostname.com_0 ] > > Full list of resources: > > Master/Slave Set: drbd.master [drbd.slave] > Masters: [ node1.hostname.com_0 ] > Slaves: [ node2.hostname.com_0 ] > drbdfs (ocf::heartbeat:Filesystem): Stopped > Master/Slave Set: inside-interface-sameip.master [inside-interface- > sameip.slave] > Masters: [ node2.hostname.com_0 ] > Stopped: [ node1.hostname.com_0 ] > Master/Slave Set: outside-interface-sameip.master [outside- > interface-sameip.slave] > Masters: [ node1.hostname.com_0 ] > Slaves: [ node2.hostname.com_0 ] > Clone Set: monitor-eth1-clone [monitor-eth1] > Started: [ node1.hostname.com_0 node2.hostname.com_0 ] > Clone Set: monitor-eth2-clone [monitor-eth2] > Started: [ node1.hostname.com_0 node2.hostname.com_0 ] > > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: inactive/disabled > > This is the "constraints" section of my CIB (full CIB is attached): > id="pcs_rsc_colocation_set_drbdfs_set_drbd.master_inside-interface- > sameip.master_outside-interface-sameip.master" score="INFINITY"> > > > > sequential="false"> > > > > > Resource sets can be confusing in the best of cases. The above constraint says: Place drbdfs only on a node where the master instances of drbd.master and the two IPs are running (without any dependencies between those resources). This explains why the master instances can run on different nodes, and why drbdfs was stopped when they did. > kind="Serialize" symmetrical="false"> > id="pcs_rsc_set_drbd.master_inside-interface-sameip.master_outside- > interface-sameip.master-1" role="Master"> > > > > > > > > The above constraint says: if promoting any of drbd.master and the two interfaces and/or starting drbdfs, do each action one at a time (in any order). Other actions (including demoting and stopping) can happen in any order. > r