Re: [Pacemaker] drbd under pacemaker - always get split brain
> It is not. > > Pacemaker may just be quicker to promote now, > or in your setup other things may have changed > which also changed the timing behaviour. > > But what you are trying to do has always been broken, > and will always be broken. > Hello Lars, You were right, fixing configuration indeed fixed my issue. I humbly apologize for my ignorance and will further immerse in documentation :) have a nice day! nik -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpz99YkZSCI8.pgp Description: PGP signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] drbd under pacemaker - always get split brain
On Thu, Jul 12, 2012 at 04:23:51PM +0200, Nikola Ciprich wrote: > Hello Lars, > > thanks for Your reply.. > > > You "Problem" is this: > > > > DRBD config: > >allow-two-primaries, > >but *NO* fencing policy, > >and *NO* fencing handler. > > > > And, as if that was not bad enough already, > > Pacemaker config: > > no-quorum-policy="ignore" \ > > stonith-enabled="false" > > yes, I've written it's just test cluster on virtual machines. therefore no > fencing devices. > > however I don't think it's the whole problem source, I've tried starting > node2 much later > after node1 (actually node1 has been running for about 1 day), and got right > into same situation.. > pacemaker just doesn't wait long enough before the drbds can connect at all > and seems to promote them both. > it really seems to be regression to me, as this was always working well... It is not. Pacemaker may just be quicker to promote now, or in your setup other things may have changed which also changed the timing behaviour. But what you are trying to do has always been broken, and will always be broken. > even though I've set no-quorum-policy to freeze, the problem returns as soon > as cluster becomes quorate.. > I have all split-brain and fencing scripts in drbd disabled intentionaly so I > had chance to investigate, otherwise > one of the nodes always commited suicide but there should be no reason for > split brain.. Right. That's why "shooting" as in stonith is not good enough a fencing mechanism in a drbd dual Primary cluster. You also need to tell the peer that it is outdated, respectively must not become "Primary" or "Master" until it synced up (or at least, *starts* to sync up). You can do that using the crm-fence-peer.sh (it does not actually tell DRBD that it is outdated, but it tells Pacemaker to not promote that other node, which is even better, if the rest of the system is properly set up. crm-fence-peer.sh alone is also not good enough in certain situations. That's why you need both, the drbd "fence-peer" mechanism *and* stonith. > > cheers! > > nik > > > > > > D'oh. > > > > And then, well, > > your nodes come up some minute+ after each other, > > and Pacemaker and DRBD behave exactly as configured: > > > > > > Jul 10 06:00:12 vmnci20 crmd: [3569]: info: do_state_transition: All 1 > > cluster nodes are eligible to run resources. > > > > > > Note the *1* ... > > > > So it starts: > > Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Start > > drbd-sas0:0(vmnci20) > > > > But leaves: > > Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Leave > > drbd-sas0:1(Stopped) > > as there is no peer node yet. > > > > > > And on the next iteration, we still have only one node: > > Jul 10 06:00:15 vmnci20 crmd: [3569]: info: do_state_transition: All 1 > > cluster nodes are eligible to run resources. > > > > So we promote: > > Jul 10 06:00:15 vmnci20 pengine: [3568]: notice: LogActions: Promote > > drbd-sas0:0(Slave -> Master vmnci20) > > > > > > And only some minute later, the peer node joins: > > Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: State > > transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED > > cause=C_FSA_INTERNAL origin=check_join_state ] > > Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: All 2 > > cluster nodes responded to the join offer. > > > > So now we can start the peer: > > > > Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Leave > > drbd-sas0:0(Master vmnci20) > > Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Start > > drbd-sas0:1(vmnci21) > > > > > > And it even is promoted right away: > > Jul 10 06:01:36 vmnci20 pengine: [3568]: notice: LogActions: Promote > > drbd-sas0:1(Slave -> Master vmnci21) > > > > And within those 3 seconds, DRBD was not able to establish the connection > > yet. > > > > > > You configured DRBD and Pacemaker to produce data divergence. > > Not suprisingly, that is exactly what you get. > > > > > > > > Fix your Problem. > > See above; hint: fencing resource-and-stonith, > > crm-fence-peer.sh + stonith_admin, > > add stonith, maybe add a third node so you don't need to ignore quorum, > > ... > > > > And all will be well. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] drbd under pacemaker - always get split brain
Hello Lars, thanks for Your reply.. > You "Problem" is this: > > DRBD config: > allow-two-primaries, > but *NO* fencing policy, > and *NO* fencing handler. > > And, as if that was not bad enough already, > Pacemaker config: > no-quorum-policy="ignore" \ > stonith-enabled="false" yes, I've written it's just test cluster on virtual machines. therefore no fencing devices. however I don't think it's the whole problem source, I've tried starting node2 much later after node1 (actually node1 has been running for about 1 day), and got right into same situation.. pacemaker just doesn't wait long enough before the drbds can connect at all and seems to promote them both. it really seems to be regression to me, as this was always working well... even though I've set no-quorum-policy to freeze, the problem returns as soon as cluster becomes quorate.. I have all split-brain and fencing scripts in drbd disabled intentionaly so I had chance to investigate, otherwise one of the nodes always commited suicide but there should be no reason for split brain.. cheers! nik > D'oh. > > And then, well, > your nodes come up some minute+ after each other, > and Pacemaker and DRBD behave exactly as configured: > > > Jul 10 06:00:12 vmnci20 crmd: [3569]: info: do_state_transition: All 1 > cluster nodes are eligible to run resources. > > > Note the *1* ... > > So it starts: > Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Start > drbd-sas0:0 (vmnci20) > > But leaves: > Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Leave > drbd-sas0:1 (Stopped) > as there is no peer node yet. > > > And on the next iteration, we still have only one node: > Jul 10 06:00:15 vmnci20 crmd: [3569]: info: do_state_transition: All 1 > cluster nodes are eligible to run resources. > > So we promote: > Jul 10 06:00:15 vmnci20 pengine: [3568]: notice: LogActions: Promote > drbd-sas0:0 (Slave -> Master vmnci20) > > > And only some minute later, the peer node joins: > Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: State > transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED > cause=C_FSA_INTERNAL origin=check_join_state ] > Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: All 2 > cluster nodes responded to the join offer. > > So now we can start the peer: > > Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Leave > drbd-sas0:0 (Master vmnci20) > Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Start > drbd-sas0:1 (vmnci21) > > > And it even is promoted right away: > Jul 10 06:01:36 vmnci20 pengine: [3568]: notice: LogActions: Promote > drbd-sas0:1 (Slave -> Master vmnci21) > > And within those 3 seconds, DRBD was not able to establish the connection yet. > > > You configured DRBD and Pacemaker to produce data divergence. > Not suprisingly, that is exactly what you get. > > > > Fix your Problem. > See above; hint: fencing resource-and-stonith, > crm-fence-peer.sh + stonith_admin, > add stonith, maybe add a third node so you don't need to ignore quorum, > ... > > And all will be well. > > > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgplQrfWtrRJa.pgp Description: PGP signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] drbd under pacemaker - always get split brain
On Wed, Jul 11, 2012 at 11:38:52AM +0200, Nikola Ciprich wrote: > > Well, I'd expect that to be safer as your current configuration ... > > discard-zero-changes will never overwrite data automatically have > > you tried adding the start-delay to DRBD start operation? I'm curious if > > that is already sufficient for your problem. > Hi, > > tried > timeout="240s"/> > (I hope it's the setting You've meant, although I'm not sure, I haven't found > any documentation > on start-delay option) > > but didn't help.. Of course not. You "Problem" is this: DRBD config: allow-two-primaries, but *NO* fencing policy, and *NO* fencing handler. And, as if that was not bad enough already, Pacemaker config: no-quorum-policy="ignore" \ stonith-enabled="false" D'oh. And then, well, your nodes come up some minute+ after each other, and Pacemaker and DRBD behave exactly as configured: Jul 10 06:00:12 vmnci20 crmd: [3569]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Note the *1* ... So it starts: Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Start drbd-sas0:0(vmnci20) But leaves: Jul 10 06:00:12 vmnci20 pengine: [3568]: notice: LogActions: Leave drbd-sas0:1(Stopped) as there is no peer node yet. And on the next iteration, we still have only one node: Jul 10 06:00:15 vmnci20 crmd: [3569]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. So we promote: Jul 10 06:00:15 vmnci20 pengine: [3568]: notice: LogActions: Promote drbd-sas0:0(Slave -> Master vmnci20) And only some minute later, the peer node joins: Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jul 10 06:01:33 vmnci20 crmd: [3569]: info: do_state_transition: All 2 cluster nodes responded to the join offer. So now we can start the peer: Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Leave drbd-sas0:0(Master vmnci20) Jul 10 06:01:33 vmnci20 pengine: [3568]: notice: LogActions: Start drbd-sas0:1(vmnci21) And it even is promoted right away: Jul 10 06:01:36 vmnci20 pengine: [3568]: notice: LogActions: Promote drbd-sas0:1(Slave -> Master vmnci21) And within those 3 seconds, DRBD was not able to establish the connection yet. You configured DRBD and Pacemaker to produce data divergence. Not suprisingly, that is exactly what you get. Fix your Problem. See above; hint: fencing resource-and-stonith, crm-fence-peer.sh + stonith_admin, add stonith, maybe add a third node so you don't need to ignore quorum, ... And all will be well. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] drbd under pacemaker - always get split brain
> Well, I'd expect that to be safer as your current configuration ... > discard-zero-changes will never overwrite data automatically have > you tried adding the start-delay to DRBD start operation? I'm curious if > that is already sufficient for your problem. Hi, tried (I hope it's the setting You've meant, although I'm not sure, I haven't found any documentation on start-delay option) but didn't help.. > > Regards, > Andreas > > > > >>> > >>> Best Regards, > >>> Andreas > >>> > >>> -- > >>> Need help with Pacemaker? > >>> http://www.hastexo.com/now > >>> > > thanks for Your time. > n. > > > > > > Regards, > > Andreas > > > > -- > > Need help with Pacemaker? > > http://www.hastexo.com/now > > > >> > >> thanks a lot in advance > >> > >> nik > >> > >> > >> On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: > >>> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: > hello, > > I'm trying to solve quite mysterious problem here.. > I've got new cluster with bunch of SAS disks for testing purposes. > I've configured DRBDs (in primary/primary configuration) > > when I start drbd using drbdadm, it get's up nicely (both nodes > are Primary, connected). > however when I start it using corosync, I always get split-brain, > although > there are no data written, no network disconnection, anything.. > >>> > >>> your full drbd and Pacemaker configuration please ... some snippets > >>> from > >>> something are very seldom helpful ... > >>> > >>> Regards, > >>> Andreas > >>> > >>> -- > >>> Need help with Pacemaker? > >>> http://www.hastexo.com/now > >>> > > here's drbd resource config: > primitive drbd-sas0 ocf:linbit:drbd \ > params drbd_resource="drbd-sas0" \ > operations $id="drbd-sas0-operations" \ > op start interval="0" timeout="240s" \ > op stop interval="0" timeout="200s" \ > op promote interval="0" timeout="200s" \ > op demote interval="0" timeout="200s" \ > op monitor interval="179s" role="Master" timeout="150s" \ > op monitor interval="180s" role="Slave" timeout="150s" > > ms ms-drbd-sas0 drbd-sas0 \ > meta clone-max="2" clone-node-max="1" master-max="2" > master-node-max="1" notify="true" globally-unique="false" > interleave="true" target-role="Started" > > > here's the dmesg output when pacemaker tries to promote drbd, > causing the splitbrain: > [ 157.646292] block drbd2: Starting worker thread (from drbdsetup > [6892]) > [ 157.646539] block drbd2: disk( Diskless -> Attaching ) > [ 157.650364] block drbd2: Found 1 transactions (1 active extents) > in activity log. > [ 157.650560] block drbd2: Method to ensure write ordering: drain > [ 157.650688] block drbd2: drbd_bm_resize called with capacity == > 584667688 > [ 157.653442] block drbd2: resync bitmap: bits=73083461 > words=1141930 pages=2231 > [ 157.653760] block drbd2: size = 279 GB (292333844 KB) > [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies > [ 157.673722] block drbd2: recounting of set bits took additional 2 > jiffies > [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on > disk bit-map. > [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) > [ 157.674100] block drbd2: attached to UUIDs > 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 > [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) > [ 157.685704] block drbd2: Starting receiver thread (from > drbd2_worker [6893]) > [ 157.685928] block drbd2: receiver (re)started > [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) > [ 158.960577] block drbd2: role( Secondary -> Primary ) > [ 158.960815] block drbd2: new current UUID > 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > [ 162.686990] block drbd2: Handshake successful: Agreed network > protocol version 96 > [ 162.687183] block drbd2: conn( WFConnection -> WFReportParams ) > [ 162.687404] block drbd2: Starting asender thread (from > drbd2_receiver [6927]) > [ 162.687741] block drbd2: data-integrity-alg: > [ 162.687930] block drbd2: drbd_sync_handshake: > [ 162.688057] block drbd2: self > 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > bits:0 flags:0 > [ 162.688244] block drbd2: peer >
Re: [Pacemaker] drbd under pacemaker - always get split brain
On 07/11/2012 09:23 AM, Nikola Ciprich wrote: >>> It really really looks like Pacemaker is too fast when promoting to >>> primary ... before the connection to the already up second node can be >>> established. >> >> Do you mean we're violating a constraint? >> Or is it a problem of the RA returning too soon? > dunno, I tried older drbd userspaces to check if it's not problem > of newer RA, to no avail... > >> >>> I see in your logs you have DRBD 8.3.13 userland but >>> 8.3.11 DRBD module installed ... can you test with 8.3.13 kernel module >>> ... there have been fixes that look like addressing this problem. > tried 8.3.13 userspace + 8.3.13 module (on top of 3.0.36 kernel), > unfortunately same result.. > >>> >>> Another quick-fix, that should also do: add a start-delay of some >>> seconds to the start operation of DRBD >>> >>> ... or fix your after-split-brain policies to automatically solve this >>> special type of split-brain (with 0 blocks to sync). > I'll try that, although I'd not like to use this for production :) Well, I'd expect that to be safer as your current configuration ... discard-zero-changes will never overwrite data automatically have you tried adding the start-delay to DRBD start operation? I'm curious if that is already sufficient for your problem. Regards, Andreas > >>> >>> Best Regards, >>> Andreas >>> >>> -- >>> Need help with Pacemaker? >>> http://www.hastexo.com/now >>> thanks for Your time. n. > > Regards, > Andreas > > -- > Need help with Pacemaker? > http://www.hastexo.com/now > >> >> thanks a lot in advance >> >> nik >> >> >> On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: >>> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: hello, I'm trying to solve quite mysterious problem here.. I've got new cluster with bunch of SAS disks for testing purposes. I've configured DRBDs (in primary/primary configuration) when I start drbd using drbdadm, it get's up nicely (both nodes are Primary, connected). however when I start it using corosync, I always get split-brain, although there are no data written, no network disconnection, anything.. >>> >>> your full drbd and Pacemaker configuration please ... some snippets from >>> something are very seldom helpful ... >>> >>> Regards, >>> Andreas >>> >>> -- >>> Need help with Pacemaker? >>> http://www.hastexo.com/now >>> here's drbd resource config: primitive drbd-sas0 ocf:linbit:drbd \ params drbd_resource="drbd-sas0" \ operations $id="drbd-sas0-operations" \ op start interval="0" timeout="240s" \ op stop interval="0" timeout="200s" \ op promote interval="0" timeout="200s" \ op demote interval="0" timeout="200s" \ op monitor interval="179s" role="Master" timeout="150s" \ op monitor interval="180s" role="Slave" timeout="150s" ms ms-drbd-sas0 drbd-sas0 \ meta clone-max="2" clone-node-max="1" master-max="2" master-node-max="1" notify="true" globally-unique="false" interleave="true" target-role="Started" here's the dmesg output when pacemaker tries to promote drbd, causing the splitbrain: [ 157.646292] block drbd2: Starting worker thread (from drbdsetup [6892]) [ 157.646539] block drbd2: disk( Diskless -> Attaching ) [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in activity log. [ 157.650560] block drbd2: Method to ensure write ordering: drain [ 157.650688] block drbd2: drbd_bm_resize called with capacity == 584667688 [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 pages=2231 [ 157.653760] block drbd2: size = 279 GB (292333844 KB) [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies [ 157.673722] block drbd2: recounting of set bits took additional 2 jiffies [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map. [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) [ 157.674100] block drbd2: attached to UUIDs 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) [ 157.685704] block drbd2: Starting receiver thread (from drbd2_worker [6893]) [ 157.685928] block drbd2: receiver (re)started [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) [ 158.960577] block drbd2: role( Secondary -> Primary ) [ 158.960815] block drbd2: new current UUID 015E111F18D089
Re: [Pacemaker] drbd under pacemaker - always get split brain
On 07/11/2012 04:50 AM, Andrew Beekhof wrote: > On Wed, Jul 11, 2012 at 8:06 AM, Andreas Kurz wrote: >> On Tue, Jul 10, 2012 at 8:12 AM, Nikola Ciprich >> wrote: >>> Hello Andreas, Why not using the RA that comes with the resource-agent package? >>> well, I've historically used my scripts, haven't even noticed when LVM >>> resource appeared.. I switched to it now.., thanks for the hint.. this "become-primary-on" was never activated? >>> nope. >>> >>> Is the drbd init script deactivated on system boot? Cluster logs should give more insights >>> yes, it's deactivated. I tried resyncinc drbd by hand, deleted logs, >>> rebooted both nodes, checked drbd ain't started and started corosync. >>> result is here: >>> http://nelide.cz/nik/logs.tar.gz >> >> It really really looks like Pacemaker is too fast when promoting to >> primary ... before the connection to the already up second node can be >> established. > > Do you mean we're violating a constraint? > Or is it a problem of the RA returning too soon? It looks like a RA problem ... notifications after the start of the resource and the following promote are very fast and DRBD is still not finished with establishing the connection to the peer. I can't remember seeing this before. Regards, Andreas > >> I see in your logs you have DRBD 8.3.13 userland but >> 8.3.11 DRBD module installed ... can you test with 8.3.13 kernel module >> ... there have been fixes that look like addressing this problem. >> >> Another quick-fix, that should also do: add a start-delay of some >> seconds to the start operation of DRBD >> >> ... or fix your after-split-brain policies to automatically solve this >> special type of split-brain (with 0 blocks to sync). >> >> Best Regards, >> Andreas >> >> -- >> Need help with Pacemaker? >> http://www.hastexo.com/now >> >>> >>> thanks for Your time. >>> n. >>> >>> Regards, Andreas -- Need help with Pacemaker? http://www.hastexo.com/now > > thanks a lot in advance > > nik > > > On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: >> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: >>> hello, >>> >>> I'm trying to solve quite mysterious problem here.. >>> I've got new cluster with bunch of SAS disks for testing purposes. >>> I've configured DRBDs (in primary/primary configuration) >>> >>> when I start drbd using drbdadm, it get's up nicely (both nodes >>> are Primary, connected). >>> however when I start it using corosync, I always get split-brain, >>> although >>> there are no data written, no network disconnection, anything.. >> >> your full drbd and Pacemaker configuration please ... some snippets from >> something are very seldom helpful ... >> >> Regards, >> Andreas >> >> -- >> Need help with Pacemaker? >> http://www.hastexo.com/now >> >>> >>> here's drbd resource config: >>> primitive drbd-sas0 ocf:linbit:drbd \ >>> params drbd_resource="drbd-sas0" \ >>> operations $id="drbd-sas0-operations" \ >>> op start interval="0" timeout="240s" \ >>> op stop interval="0" timeout="200s" \ >>> op promote interval="0" timeout="200s" \ >>> op demote interval="0" timeout="200s" \ >>> op monitor interval="179s" role="Master" timeout="150s" \ >>> op monitor interval="180s" role="Slave" timeout="150s" >>> >>> ms ms-drbd-sas0 drbd-sas0 \ >>>meta clone-max="2" clone-node-max="1" master-max="2" >>> master-node-max="1" notify="true" globally-unique="false" >>> interleave="true" target-role="Started" >>> >>> >>> here's the dmesg output when pacemaker tries to promote drbd, causing >>> the splitbrain: >>> [ 157.646292] block drbd2: Starting worker thread (from drbdsetup >>> [6892]) >>> [ 157.646539] block drbd2: disk( Diskless -> Attaching ) >>> [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in >>> activity log. >>> [ 157.650560] block drbd2: Method to ensure write ordering: drain >>> [ 157.650688] block drbd2: drbd_bm_resize called with capacity == >>> 584667688 >>> [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 >>> pages=2231 >>> [ 157.653760] block drbd2: size = 279 GB (292333844 KB) >>> [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies >>> [ 157.673722] block drbd2: recounting of set bits took additional 2 >>> jiffies >>> [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk >>> bit-map. >>> [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) >>> [ 157.674100] block drbd2: attached to UUIDs >>> 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 >>> [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) >>> [ 157.685704] block drbd2: Starting r
Re: [Pacemaker] drbd under pacemaker - always get split brain
> > It really really looks like Pacemaker is too fast when promoting to > > primary ... before the connection to the already up second node can be > > established. > > Do you mean we're violating a constraint? > Or is it a problem of the RA returning too soon? dunno, I tried older drbd userspaces to check if it's not problem of newer RA, to no avail... > > > I see in your logs you have DRBD 8.3.13 userland but > > 8.3.11 DRBD module installed ... can you test with 8.3.13 kernel module > > ... there have been fixes that look like addressing this problem. tried 8.3.13 userspace + 8.3.13 module (on top of 3.0.36 kernel), unfortunately same result.. > > > > Another quick-fix, that should also do: add a start-delay of some > > seconds to the start operation of DRBD > > > > ... or fix your after-split-brain policies to automatically solve this > > special type of split-brain (with 0 blocks to sync). I'll try that, although I'd not like to use this for production :) > > > > Best Regards, > > Andreas > > > > -- > > Need help with Pacemaker? > > http://www.hastexo.com/now > > > >> > >> thanks for Your time. > >> n. > >> > >> > >>> > >>> Regards, > >>> Andreas > >>> > >>> -- > >>> Need help with Pacemaker? > >>> http://www.hastexo.com/now > >>> > >>> > > >>> > thanks a lot in advance > >>> > > >>> > nik > >>> > > >>> > > >>> > On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: > >>> >> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: > >>> >>> hello, > >>> >>> > >>> >>> I'm trying to solve quite mysterious problem here.. > >>> >>> I've got new cluster with bunch of SAS disks for testing purposes. > >>> >>> I've configured DRBDs (in primary/primary configuration) > >>> >>> > >>> >>> when I start drbd using drbdadm, it get's up nicely (both nodes > >>> >>> are Primary, connected). > >>> >>> however when I start it using corosync, I always get split-brain, > >>> >>> although > >>> >>> there are no data written, no network disconnection, anything.. > >>> >> > >>> >> your full drbd and Pacemaker configuration please ... some snippets > >>> >> from > >>> >> something are very seldom helpful ... > >>> >> > >>> >> Regards, > >>> >> Andreas > >>> >> > >>> >> -- > >>> >> Need help with Pacemaker? > >>> >> http://www.hastexo.com/now > >>> >> > >>> >>> > >>> >>> here's drbd resource config: > >>> >>> primitive drbd-sas0 ocf:linbit:drbd \ > >>> >>> params drbd_resource="drbd-sas0" \ > >>> >>> operations $id="drbd-sas0-operations" \ > >>> >>> op start interval="0" timeout="240s" \ > >>> >>> op stop interval="0" timeout="200s" \ > >>> >>> op promote interval="0" timeout="200s" \ > >>> >>> op demote interval="0" timeout="200s" \ > >>> >>> op monitor interval="179s" role="Master" timeout="150s" \ > >>> >>> op monitor interval="180s" role="Slave" timeout="150s" > >>> >>> > >>> >>> ms ms-drbd-sas0 drbd-sas0 \ > >>> >>>meta clone-max="2" clone-node-max="1" master-max="2" > >>> >>> master-node-max="1" notify="true" globally-unique="false" > >>> >>> interleave="true" target-role="Started" > >>> >>> > >>> >>> > >>> >>> here's the dmesg output when pacemaker tries to promote drbd, causing > >>> >>> the splitbrain: > >>> >>> [ 157.646292] block drbd2: Starting worker thread (from drbdsetup > >>> >>> [6892]) > >>> >>> [ 157.646539] block drbd2: disk( Diskless -> Attaching ) > >>> >>> [ 157.650364] block drbd2: Found 1 transactions (1 active extents) > >>> >>> in activity log. > >>> >>> [ 157.650560] block drbd2: Method to ensure write ordering: drain > >>> >>> [ 157.650688] block drbd2: drbd_bm_resize called with capacity == > >>> >>> 584667688 > >>> >>> [ 157.653442] block drbd2: resync bitmap: bits=73083461 > >>> >>> words=1141930 pages=2231 > >>> >>> [ 157.653760] block drbd2: size = 279 GB (292333844 KB) > >>> >>> [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies > >>> >>> [ 157.673722] block drbd2: recounting of set bits took additional 2 > >>> >>> jiffies > >>> >>> [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on > >>> >>> disk bit-map. > >>> >>> [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) > >>> >>> [ 157.674100] block drbd2: attached to UUIDs > >>> >>> 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 > >>> >>> [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) > >>> >>> [ 157.685704] block drbd2: Starting receiver thread (from > >>> >>> drbd2_worker [6893]) > >>> >>> [ 157.685928] block drbd2: receiver (re)started > >>> >>> [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) > >>> >>> [ 158.960577] block drbd2: role( Secondary -> Primary ) > >>> >>> [ 158.960815] block drbd2: new current UUID > >>> >>> 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > >>> >>> [ 162.686990] block drbd2: Handshake successful: Agreed network > >>> >>> protocol version 96 > >>> >>> [ 162.687183] block drbd2: conn( WFConnection -> WFReportParams ) > >>> >>> [ 162.
Re: [Pacemaker] drbd under pacemaker - always get split brain
On Wed, Jul 11, 2012 at 8:06 AM, Andreas Kurz wrote: > On Tue, Jul 10, 2012 at 8:12 AM, Nikola Ciprich > wrote: >> Hello Andreas, >>> Why not using the RA that comes with the resource-agent package? >> well, I've historically used my scripts, haven't even noticed when LVM >> resource appeared.. I switched to it now.., thanks for the hint.. >>> this "become-primary-on" was never activated? >> nope. >> >> >>> Is the drbd init script deactivated on system boot? Cluster logs should >>> give more insights >> yes, it's deactivated. I tried resyncinc drbd by hand, deleted logs, >> rebooted both nodes, checked drbd ain't started and started corosync. >> result is here: >> http://nelide.cz/nik/logs.tar.gz > > It really really looks like Pacemaker is too fast when promoting to > primary ... before the connection to the already up second node can be > established. Do you mean we're violating a constraint? Or is it a problem of the RA returning too soon? > I see in your logs you have DRBD 8.3.13 userland but > 8.3.11 DRBD module installed ... can you test with 8.3.13 kernel module > ... there have been fixes that look like addressing this problem. > > Another quick-fix, that should also do: add a start-delay of some > seconds to the start operation of DRBD > > ... or fix your after-split-brain policies to automatically solve this > special type of split-brain (with 0 blocks to sync). > > Best Regards, > Andreas > > -- > Need help with Pacemaker? > http://www.hastexo.com/now > >> >> thanks for Your time. >> n. >> >> >>> >>> Regards, >>> Andreas >>> >>> -- >>> Need help with Pacemaker? >>> http://www.hastexo.com/now >>> >>> > >>> > thanks a lot in advance >>> > >>> > nik >>> > >>> > >>> > On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: >>> >> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: >>> >>> hello, >>> >>> >>> >>> I'm trying to solve quite mysterious problem here.. >>> >>> I've got new cluster with bunch of SAS disks for testing purposes. >>> >>> I've configured DRBDs (in primary/primary configuration) >>> >>> >>> >>> when I start drbd using drbdadm, it get's up nicely (both nodes >>> >>> are Primary, connected). >>> >>> however when I start it using corosync, I always get split-brain, >>> >>> although >>> >>> there are no data written, no network disconnection, anything.. >>> >> >>> >> your full drbd and Pacemaker configuration please ... some snippets from >>> >> something are very seldom helpful ... >>> >> >>> >> Regards, >>> >> Andreas >>> >> >>> >> -- >>> >> Need help with Pacemaker? >>> >> http://www.hastexo.com/now >>> >> >>> >>> >>> >>> here's drbd resource config: >>> >>> primitive drbd-sas0 ocf:linbit:drbd \ >>> >>> params drbd_resource="drbd-sas0" \ >>> >>> operations $id="drbd-sas0-operations" \ >>> >>> op start interval="0" timeout="240s" \ >>> >>> op stop interval="0" timeout="200s" \ >>> >>> op promote interval="0" timeout="200s" \ >>> >>> op demote interval="0" timeout="200s" \ >>> >>> op monitor interval="179s" role="Master" timeout="150s" \ >>> >>> op monitor interval="180s" role="Slave" timeout="150s" >>> >>> >>> >>> ms ms-drbd-sas0 drbd-sas0 \ >>> >>>meta clone-max="2" clone-node-max="1" master-max="2" >>> >>> master-node-max="1" notify="true" globally-unique="false" >>> >>> interleave="true" target-role="Started" >>> >>> >>> >>> >>> >>> here's the dmesg output when pacemaker tries to promote drbd, causing >>> >>> the splitbrain: >>> >>> [ 157.646292] block drbd2: Starting worker thread (from drbdsetup >>> >>> [6892]) >>> >>> [ 157.646539] block drbd2: disk( Diskless -> Attaching ) >>> >>> [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in >>> >>> activity log. >>> >>> [ 157.650560] block drbd2: Method to ensure write ordering: drain >>> >>> [ 157.650688] block drbd2: drbd_bm_resize called with capacity == >>> >>> 584667688 >>> >>> [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 >>> >>> pages=2231 >>> >>> [ 157.653760] block drbd2: size = 279 GB (292333844 KB) >>> >>> [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies >>> >>> [ 157.673722] block drbd2: recounting of set bits took additional 2 >>> >>> jiffies >>> >>> [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk >>> >>> bit-map. >>> >>> [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) >>> >>> [ 157.674100] block drbd2: attached to UUIDs >>> >>> 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 >>> >>> [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) >>> >>> [ 157.685704] block drbd2: Starting receiver thread (from drbd2_worker >>> >>> [6893]) >>> >>> [ 157.685928] block drbd2: receiver (re)started >>> >>> [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) >>> >>> [ 158.960577] block drbd2: role( Secondary -> Primary ) >>> >>> [ 158.960815] block drbd2: new current UUID >>> >>> 015E111F18D08945:0150944D23F16BAE:8C175205284E3262
Re: [Pacemaker] drbd under pacemaker - always get split brain
On Tue, Jul 10, 2012 at 8:12 AM, Nikola Ciprich wrote: > Hello Andreas, >> Why not using the RA that comes with the resource-agent package? > well, I've historically used my scripts, haven't even noticed when LVM > resource appeared.. I switched to it now.., thanks for the hint.. >> this "become-primary-on" was never activated? > nope. > > >> Is the drbd init script deactivated on system boot? Cluster logs should >> give more insights > yes, it's deactivated. I tried resyncinc drbd by hand, deleted logs, > rebooted both nodes, checked drbd ain't started and started corosync. > result is here: > http://nelide.cz/nik/logs.tar.gz It really really looks like Pacemaker is too fast when promoting to primary ... before the connection to the already up second node can be established. I see in your logs you have DRBD 8.3.13 userland but 8.3.11 DRBD module installed ... can you test with 8.3.13 kernel module ... there have been fixes that look like addressing this problem. Another quick-fix, that should also do: add a start-delay of some seconds to the start operation of DRBD ... or fix your after-split-brain policies to automatically solve this special type of split-brain (with 0 blocks to sync). Best Regards, Andreas -- Need help with Pacemaker? http://www.hastexo.com/now > > thanks for Your time. > n. > > >> >> Regards, >> Andreas >> >> -- >> Need help with Pacemaker? >> http://www.hastexo.com/now >> >> > >> > thanks a lot in advance >> > >> > nik >> > >> > >> > On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: >> >> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: >> >>> hello, >> >>> >> >>> I'm trying to solve quite mysterious problem here.. >> >>> I've got new cluster with bunch of SAS disks for testing purposes. >> >>> I've configured DRBDs (in primary/primary configuration) >> >>> >> >>> when I start drbd using drbdadm, it get's up nicely (both nodes >> >>> are Primary, connected). >> >>> however when I start it using corosync, I always get split-brain, >> >>> although >> >>> there are no data written, no network disconnection, anything.. >> >> >> >> your full drbd and Pacemaker configuration please ... some snippets from >> >> something are very seldom helpful ... >> >> >> >> Regards, >> >> Andreas >> >> >> >> -- >> >> Need help with Pacemaker? >> >> http://www.hastexo.com/now >> >> >> >>> >> >>> here's drbd resource config: >> >>> primitive drbd-sas0 ocf:linbit:drbd \ >> >>> params drbd_resource="drbd-sas0" \ >> >>> operations $id="drbd-sas0-operations" \ >> >>> op start interval="0" timeout="240s" \ >> >>> op stop interval="0" timeout="200s" \ >> >>> op promote interval="0" timeout="200s" \ >> >>> op demote interval="0" timeout="200s" \ >> >>> op monitor interval="179s" role="Master" timeout="150s" \ >> >>> op monitor interval="180s" role="Slave" timeout="150s" >> >>> >> >>> ms ms-drbd-sas0 drbd-sas0 \ >> >>>meta clone-max="2" clone-node-max="1" master-max="2" >> >>> master-node-max="1" notify="true" globally-unique="false" >> >>> interleave="true" target-role="Started" >> >>> >> >>> >> >>> here's the dmesg output when pacemaker tries to promote drbd, causing >> >>> the splitbrain: >> >>> [ 157.646292] block drbd2: Starting worker thread (from drbdsetup >> >>> [6892]) >> >>> [ 157.646539] block drbd2: disk( Diskless -> Attaching ) >> >>> [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in >> >>> activity log. >> >>> [ 157.650560] block drbd2: Method to ensure write ordering: drain >> >>> [ 157.650688] block drbd2: drbd_bm_resize called with capacity == >> >>> 584667688 >> >>> [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 >> >>> pages=2231 >> >>> [ 157.653760] block drbd2: size = 279 GB (292333844 KB) >> >>> [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies >> >>> [ 157.673722] block drbd2: recounting of set bits took additional 2 >> >>> jiffies >> >>> [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk >> >>> bit-map. >> >>> [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) >> >>> [ 157.674100] block drbd2: attached to UUIDs >> >>> 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 >> >>> [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) >> >>> [ 157.685704] block drbd2: Starting receiver thread (from drbd2_worker >> >>> [6893]) >> >>> [ 157.685928] block drbd2: receiver (re)started >> >>> [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) >> >>> [ 158.960577] block drbd2: role( Secondary -> Primary ) >> >>> [ 158.960815] block drbd2: new current UUID >> >>> 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 >> >>> [ 162.686990] block drbd2: Handshake successful: Agreed network >> >>> protocol version 96 >> >>> [ 162.687183] block drbd2: conn( WFConnection -> WFReportParams ) >> >>> [ 162.687404] block drbd2: Starting asender thread (from drbd2_receiver >> >>> [6927]) >
Re: [Pacemaker] drbd under pacemaker - always get split brain
Hello Andreas, > Why not using the RA that comes with the resource-agent package? well, I've historically used my scripts, haven't even noticed when LVM resource appeared.. I switched to it now.., thanks for the hint.. > this "become-primary-on" was never activated? nope. > Is the drbd init script deactivated on system boot? Cluster logs should > give more insights yes, it's deactivated. I tried resyncinc drbd by hand, deleted logs, rebooted both nodes, checked drbd ain't started and started corosync. result is here: http://nelide.cz/nik/logs.tar.gz thanks for Your time. n. > > Regards, > Andreas > > -- > Need help with Pacemaker? > http://www.hastexo.com/now > > > > > thanks a lot in advance > > > > nik > > > > > > On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote: > >> On 07/02/2012 11:49 PM, Nikola Ciprich wrote: > >>> hello, > >>> > >>> I'm trying to solve quite mysterious problem here.. > >>> I've got new cluster with bunch of SAS disks for testing purposes. > >>> I've configured DRBDs (in primary/primary configuration) > >>> > >>> when I start drbd using drbdadm, it get's up nicely (both nodes > >>> are Primary, connected). > >>> however when I start it using corosync, I always get split-brain, although > >>> there are no data written, no network disconnection, anything.. > >> > >> your full drbd and Pacemaker configuration please ... some snippets from > >> something are very seldom helpful ... > >> > >> Regards, > >> Andreas > >> > >> -- > >> Need help with Pacemaker? > >> http://www.hastexo.com/now > >> > >>> > >>> here's drbd resource config: > >>> primitive drbd-sas0 ocf:linbit:drbd \ > >>> params drbd_resource="drbd-sas0" \ > >>> operations $id="drbd-sas0-operations" \ > >>> op start interval="0" timeout="240s" \ > >>> op stop interval="0" timeout="200s" \ > >>> op promote interval="0" timeout="200s" \ > >>> op demote interval="0" timeout="200s" \ > >>> op monitor interval="179s" role="Master" timeout="150s" \ > >>> op monitor interval="180s" role="Slave" timeout="150s" > >>> > >>> ms ms-drbd-sas0 drbd-sas0 \ > >>>meta clone-max="2" clone-node-max="1" master-max="2" > >>> master-node-max="1" notify="true" globally-unique="false" > >>> interleave="true" target-role="Started" > >>> > >>> > >>> here's the dmesg output when pacemaker tries to promote drbd, causing the > >>> splitbrain: > >>> [ 157.646292] block drbd2: Starting worker thread (from drbdsetup [6892]) > >>> [ 157.646539] block drbd2: disk( Diskless -> Attaching ) > >>> [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in > >>> activity log. > >>> [ 157.650560] block drbd2: Method to ensure write ordering: drain > >>> [ 157.650688] block drbd2: drbd_bm_resize called with capacity == > >>> 584667688 > >>> [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 > >>> pages=2231 > >>> [ 157.653760] block drbd2: size = 279 GB (292333844 KB) > >>> [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies > >>> [ 157.673722] block drbd2: recounting of set bits took additional 2 > >>> jiffies > >>> [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk > >>> bit-map. > >>> [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) > >>> [ 157.674100] block drbd2: attached to UUIDs > >>> 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 > >>> [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) > >>> [ 157.685704] block drbd2: Starting receiver thread (from drbd2_worker > >>> [6893]) > >>> [ 157.685928] block drbd2: receiver (re)started > >>> [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) > >>> [ 158.960577] block drbd2: role( Secondary -> Primary ) > >>> [ 158.960815] block drbd2: new current UUID > >>> 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > >>> [ 162.686990] block drbd2: Handshake successful: Agreed network protocol > >>> version 96 > >>> [ 162.687183] block drbd2: conn( WFConnection -> WFReportParams ) > >>> [ 162.687404] block drbd2: Starting asender thread (from drbd2_receiver > >>> [6927]) > >>> [ 162.687741] block drbd2: data-integrity-alg: > >>> [ 162.687930] block drbd2: drbd_sync_handshake: > >>> [ 162.688057] block drbd2: self > >>> 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > >>> bits:0 flags:0 > >>> [ 162.688244] block drbd2: peer > >>> 7EC38CBFC3D28FFF:0150944D23F16BAF:8C175205284E3263:8C165205284E3263 > >>> bits:0 flags:0 > >>> [ 162.688428] block drbd2: uuid_compare()=100 by rule 90 > >>> [ 162.688544] block drbd2: helper command: /sbin/drbdadm > >>> initial-split-brain minor-2 > >>> [ 162.691332] block drbd2: helper command: /sbin/drbdadm > >>> initial-split-brain minor-2 exit code 0 (0x0) > >>> > >>> to me it seems to be that it's promoting it too early, and I also wonder > >>> why there is the > >>> "new current UUID" stuff? > >>> > >>> I'm using centos6,
Re: [Pacemaker] drbd under pacemaker - always get split brain
On 07/09/2012 12:58 PM, Nikola Ciprich wrote:
> Hello Andreas,
>
> yes, You're right. I should have sent those in the initial post. Sorry about
> that.
> I've created very simple test configuration on which I'm able to simulate the
> problem.
> there's no stonith etc, since it's just two virtual machines for the test.
>
> crm conf:
>
> primitive drbd-sas0 ocf:linbit:drbd \
> params drbd_resource="drbd-sas0" \
> operations $id="drbd-sas0-operations" \
> op start interval="0" timeout="240s" \
> op stop interval="0" timeout="200s" \
> op promote interval="0" timeout="200s" \
> op demote interval="0" timeout="200s" \
> op monitor interval="179s" role="Master" timeout="150s" \
> op monitor interval="180s" role="Slave" timeout="150s"
>
> primitive lvm ocf:lbox:lvm.ocf \
Why not using the RA that comes with the resource-agent package?
> op start interval="0" timeout="180" \
> op stop interval="0" timeout="180"
>
> ms ms-drbd-sas0 drbd-sas0 \
>meta clone-max="2" clone-node-max="1" master-max="2" master-node-max="1"
> notify="true" globally-unique="false" interleave="true" target-role="Started"
>
> clone cl-lvm lvm \
> meta globally-unique="false" ordered="false" interleave="true"
> notify="false" target-role="Started" \
> params lvm-clone-max="2" lvm-clone-node-max="1"
>
> colocation col-lvm-drbd-sas0 inf: cl-lvm ms-drbd-sas0:Master
>
> order ord-drbd-sas0-lvm inf: ms-drbd-sas0:promote cl-lvm:start
>
> property $id="cib-bootstrap-options" \
>dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558" \
>cluster-infrastructure="openais" \
>expected-quorum-votes="2" \
>no-quorum-policy="ignore" \
>stonith-enabled="false"
>
> lvm resource starts vgshared volume group on top of drbd (LVM filters are set
> to
> use /dev/drbd* devices only)
>
> drbd configuration:
>
> global {
>usage-count no;
> }
>
> common {
>protocol C;
>
> handlers {
> pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; ";
> pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; ";
> local-io-error "/usr/lib/drbd/notify-io-error.sh;
> /usr/lib/drbd/notify-emergency-shutdown.sh; ";
>
> #pri-on-incon-degr
> "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> #pri-lost-after-sb
> "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> #local-io-error "/usr/lib/drbd/notify-io-error.sh;
> /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;
> halt -f";
> # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
> # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
> # before-resync-target
> "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
> # after-resync-target
> /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
> }
>
> net {
> allow-two-primaries;
> after-sb-0pri discard-zero-changes;
> after-sb-1pri discard-secondary;
> after-sb-2pri call-pri-lost-after-sb;
> #rr-conflict disconnect;
> max-buffers 8000;
> max-epoch-size 8000;
> sndbuf-size 0;
> ping-timeout 50;
> }
>
> syncer {
> rate 100M;
> al-extents 3833;
> # al-extents 257;
> # verify-alg sha1;
> }
>
> disk {
> on-io-error detach;
> no-disk-barrier;
> no-disk-flushes;
> no-md-flushes;
> }
>
> startup {
> # wfc-timeout 0;
> degr-wfc-timeout 120;# 2 minutes.
> # become-primary-on both;
this "become-primary-on" was never activated?
>
> }
> }
>
> note that pri-on-incon-degr etc handlers are intentionally commented out so I
> can
> see what's going on.. otherwise machine always got immediate reboot..
>
> any idea?
Is the drbd init script deactivated on system boot? Cluster logs should
give more insights
Regards,
Andreas
--
Need help with Pacemaker?
http://www.hastexo.com/now
>
> thanks a lot in advance
>
> nik
>
>
> On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote:
>> On 07/02/2012 11:49 PM, Nikola Ciprich wrote:
>>> hello,
>>>
>>> I'm trying to solve quite mysterious problem here..
>>> I've got new cluster with bunch of SAS disks for testing p
Re: [Pacemaker] drbd under pacemaker - always get split brain
Hello Andreas,
yes, You're right. I should have sent those in the initial post. Sorry about
that.
I've created very simple test configuration on which I'm able to simulate the
problem.
there's no stonith etc, since it's just two virtual machines for the test.
crm conf:
primitive drbd-sas0 ocf:linbit:drbd \
params drbd_resource="drbd-sas0" \
operations $id="drbd-sas0-operations" \
op start interval="0" timeout="240s" \
op stop interval="0" timeout="200s" \
op promote interval="0" timeout="200s" \
op demote interval="0" timeout="200s" \
op monitor interval="179s" role="Master" timeout="150s" \
op monitor interval="180s" role="Slave" timeout="150s"
primitive lvm ocf:lbox:lvm.ocf \
op start interval="0" timeout="180" \
op stop interval="0" timeout="180"
ms ms-drbd-sas0 drbd-sas0 \
meta clone-max="2" clone-node-max="1" master-max="2" master-node-max="1"
notify="true" globally-unique="false" interleave="true" target-role="Started"
clone cl-lvm lvm \
meta globally-unique="false" ordered="false" interleave="true"
notify="false" target-role="Started" \
params lvm-clone-max="2" lvm-clone-node-max="1"
colocation col-lvm-drbd-sas0 inf: cl-lvm ms-drbd-sas0:Master
order ord-drbd-sas0-lvm inf: ms-drbd-sas0:promote cl-lvm:start
property $id="cib-bootstrap-options" \
dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
no-quorum-policy="ignore" \
stonith-enabled="false"
lvm resource starts vgshared volume group on top of drbd (LVM filters are set to
use /dev/drbd* devices only)
drbd configuration:
global {
usage-count no;
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; ";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; ";
local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; ";
#pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot
-f";
#pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot
-f";
#local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt
-f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target
"/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target
/usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
#rr-conflict disconnect;
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
ping-timeout 50;
}
syncer {
rate 100M;
al-extents 3833;
# al-extents 257;
# verify-alg sha1;
}
disk {
on-io-error detach;
no-disk-barrier;
no-disk-flushes;
no-md-flushes;
}
startup {
# wfc-timeout 0;
degr-wfc-timeout 120;# 2 minutes.
# become-primary-on both;
}
}
note that pri-on-incon-degr etc handlers are intentionally commented out so I
can
see what's going on.. otherwise machine always got immediate reboot..
any idea?
thanks a lot in advance
nik
On Sun, Jul 08, 2012 at 12:47:16AM +0200, Andreas Kurz wrote:
> On 07/02/2012 11:49 PM, Nikola Ciprich wrote:
> > hello,
> >
> > I'm trying to solve quite mysterious problem here..
> > I've got new cluster with bunch of SAS disks for testing purposes.
> > I've configured DRBDs (in primary/primary configuration)
> >
> > when I start drbd using drbdadm, it get's up nicely (both nodes
> > are Primary, connected).
> > however when I start it using corosync, I always get split-brain, although
> > there are no data written, no network disconnection, anything..
>
> your full drbd and Pacemaker configuration please ... some snippets from
> something are very seldom helpful ...
>
> Regards,
> Andreas
>
> --
> Need help with Pacemaker?
> http://www.hastexo.com/now
>
> >
> > here's drbd resource config:
>
Re: [Pacemaker] drbd under pacemaker - always get split brain
On 07/02/2012 11:49 PM, Nikola Ciprich wrote: > hello, > > I'm trying to solve quite mysterious problem here.. > I've got new cluster with bunch of SAS disks for testing purposes. > I've configured DRBDs (in primary/primary configuration) > > when I start drbd using drbdadm, it get's up nicely (both nodes > are Primary, connected). > however when I start it using corosync, I always get split-brain, although > there are no data written, no network disconnection, anything.. your full drbd and Pacemaker configuration please ... some snippets from something are very seldom helpful ... Regards, Andreas -- Need help with Pacemaker? http://www.hastexo.com/now > > here's drbd resource config: > primitive drbd-sas0 ocf:linbit:drbd \ > params drbd_resource="drbd-sas0" \ > operations $id="drbd-sas0-operations" \ > op start interval="0" timeout="240s" \ > op stop interval="0" timeout="200s" \ > op promote interval="0" timeout="200s" \ > op demote interval="0" timeout="200s" \ > op monitor interval="179s" role="Master" timeout="150s" \ > op monitor interval="180s" role="Slave" timeout="150s" > > ms ms-drbd-sas0 drbd-sas0 \ >meta clone-max="2" clone-node-max="1" master-max="2" master-node-max="1" > notify="true" globally-unique="false" interleave="true" target-role="Started" > > > here's the dmesg output when pacemaker tries to promote drbd, causing the > splitbrain: > [ 157.646292] block drbd2: Starting worker thread (from drbdsetup [6892]) > [ 157.646539] block drbd2: disk( Diskless -> Attaching ) > [ 157.650364] block drbd2: Found 1 transactions (1 active extents) in > activity log. > [ 157.650560] block drbd2: Method to ensure write ordering: drain > [ 157.650688] block drbd2: drbd_bm_resize called with capacity == 584667688 > [ 157.653442] block drbd2: resync bitmap: bits=73083461 words=1141930 > pages=2231 > [ 157.653760] block drbd2: size = 279 GB (292333844 KB) > [ 157.671626] block drbd2: bitmap READ of 2231 pages took 18 jiffies > [ 157.673722] block drbd2: recounting of set bits took additional 2 jiffies > [ 157.673846] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk > bit-map. > [ 157.673972] block drbd2: disk( Attaching -> UpToDate ) > [ 157.674100] block drbd2: attached to UUIDs > 0150944D23F16BAE::8C175205284E3262:8C165205284E3263 > [ 157.685539] block drbd2: conn( StandAlone -> Unconnected ) > [ 157.685704] block drbd2: Starting receiver thread (from drbd2_worker > [6893]) > [ 157.685928] block drbd2: receiver (re)started > [ 157.686071] block drbd2: conn( Unconnected -> WFConnection ) > [ 158.960577] block drbd2: role( Secondary -> Primary ) > [ 158.960815] block drbd2: new current UUID > 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 > [ 162.686990] block drbd2: Handshake successful: Agreed network protocol > version 96 > [ 162.687183] block drbd2: conn( WFConnection -> WFReportParams ) > [ 162.687404] block drbd2: Starting asender thread (from drbd2_receiver > [6927]) > [ 162.687741] block drbd2: data-integrity-alg: > [ 162.687930] block drbd2: drbd_sync_handshake: > [ 162.688057] block drbd2: self > 015E111F18D08945:0150944D23F16BAE:8C175205284E3262:8C165205284E3263 bits:0 > flags:0 > [ 162.688244] block drbd2: peer > 7EC38CBFC3D28FFF:0150944D23F16BAF:8C175205284E3263:8C165205284E3263 bits:0 > flags:0 > [ 162.688428] block drbd2: uuid_compare()=100 by rule 90 > [ 162.688544] block drbd2: helper command: /sbin/drbdadm initial-split-brain > minor-2 > [ 162.691332] block drbd2: helper command: /sbin/drbdadm initial-split-brain > minor-2 exit code 0 (0x0) > > to me it seems to be that it's promoting it too early, and I also wonder why > there is the > "new current UUID" stuff? > > I'm using centos6, kernel 3.0.36, drbd-8.3.13, pacemaker-1.1.6 > > could anybody please try to advice me? I'm sure I'm doing something stupid, > but can't figure out what... > > thanks a lot in advance > > with best regards > > nik > > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
