Comments in-line. Also in-line with Pacemaker config at the bottom.
HTH
Jake
----- Original Message -----
> From: "Bob Schatz" <bsch...@yahoo.com>
> To: drbd-user@lists.linbit.com
> Sent: Thursday, August 11, 2011 1:09:56 PM
> Subject: [DRBD-user] Fw: DRBD STONITH - how is Pacemaker constraint
> cleared?
> Hi,
> Does anyone know the answer to the question below about DRBD STONITH
> setting Pacemaker location constraints?
> Thanks!
> Bob
> ----- Forwarded Message -----
> From: Bob Schatz <bsch...@yahoo.com>
> To: "drbd-user@lists.linbit.com" <drbd-user@lists.linbit.com>
> Sent: Tuesday, August 2, 2011 12:21 PM
> Subject: [DRBD-user] DRBD STONITH - how is Pacemaker constraint
> cleared?
> Hi,
> I setup DRBD and Pacemaker using STONITH for DRBD and for Pacemaker.
> (Configs at bottom of email)
> When I reboot the PRIMARY DRBD node (cnode-1-3-6), Pacemaker shows
> this location constraint:
> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \
> rule $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master"
> -inf: #uname ne cnode-1-3-5
> and transitions the SECONDARY to PRIMARY. This makes sense to me.
> However, when I restart cnode-1-3-6 (cnode-1-3-5 still up as PRIMARY)
> the location constraint is not cleared as I would have expected.
> Also, DRBD is not started (I assume because of the location
> constraint). I would expect that since cnode-1-3-5 is still up the
> constraint would be moved and DRBD would change to SECONDARY.
The location constraint would only prevent DRBD from allowing glance-drbd to be
promoted to Master on cnode-1-3-6. Basicly it says that the role of
ms-glance-drbd:Master can only be on node named cnode-1-3-5. It doesn't care
about ms-glance-drbd:Secondary. It would not prevent DRBD from starting either
(though your ordering could cause it not to start...). Could you clarify what
you mean by "DRBD is not stared"?
> Am I correct that this location constraint should be cleared?
> I assumed this would be cleared by the DRBD handler
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh" script but I
> do not believe it is called.
That is the handler that would clear the location constraint. You should see it
cleared after the resync is complete. If DRBD isn't running it will never
resync which means it will never run the after-resync-target commands. Have
you checked that cnode-1-3-6 is UpToDate (cat /proc/drbd). Here's an excerpt
of how it should look in the logs as the constraint is removed (should log on
cnode-1-3-6):
kernel: [ 77.131564] block drbd4: Resync done (total 1 sec; paused 0 sec; 0
K/sec)
kernel: [ 77.131573] block drbd4: conn( SyncTarget -> Connected ) disk(
Inconsistent -> UpToDate )
kernel: [ 77.131585] block drbd4: helper command: /sbin/drbdadm
after-resync-target minor-4
crm-unfence-peer.sh[3024]: invoked for bind <-- drbd4
kernel: [ 77.261360] block drbd4: helper command: /sbin/drbdadm
after-resync-target minor-4 exit code 0 (0x0)
> BTW, I am pretty sure I have ordering duplications in my Pacemaker
> configuration (pointed out by Andrew on the Pacemaker mailing list)
> but I am not sure if that is the problem.
> Thanks,
> Bob
> drbd.conf file:
> global {
> usage-count yes;
> }
> common {
> protocol C;
> }
> resource glance-repos-drbd {
> disk {
> fencing resource-and-stonith;
> }
> handlers {
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
> }
> on cnode-1-3-5 {
> device /dev/drbd1;
> disk /dev/glance-repos/glance-repos-vol;
> address 10.4.1.29:7789;
> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;
> }
> on cnode-1-3-6 {
> device /dev/drbd1;
> disk /dev/glance-repos/glance-repos-vol;
> address 10.4.1.30:7789;
> flexible-meta-disk /dev/glance-repos/glance-repos-drbd-meta-vol;
> }
> syncer {
> rate 40M;
> }
> }
> Pacemaker configuration:
> node cnode-1-3-5
> node cnode-1-3-6
> primitive glance-drbd-p ocf:linbit:drbd \ params
> drbd_resource="glance-repos-drbd" \ op start interval="0"
> timeout="240" \ op stop interval="0" timeout="100" \ op monitor
> interval="59s" role="Master" timeout="30s" \ op monitor
> interval="61s" role="Slave" timeout="30s"
> primitive glance-fs-p ocf:heartbeat:Filesystem \ params
> device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \ op
> start interval="0" timeout="60" \ op monitor interval="60"
> timeout="60" OCF_CHECK_LEVEL="20" \ op stop interval="0"
> timeout="120"
> primitive glance-ip-p ocf:heartbeat:IPaddr2 \ params ip="10.4.0.25"
> nic="br100" \ op monitor interval="5s"
> primitive glance-lvm-p ocf:heartbeat:LVM \ params
> volgrpname="glance-repos" exclusive="true" \ op start interval="0"
> timeout="30" \ op stop interval="0" timeout="30" \ meta
> target-role="Started"
I don't understand why you have this primitive?
> primitive node-stonith-5-p stonith:external/ipmi \ op monitor
> interval="10m" timeout="1m" target_role="Started" \ params
> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.99"
> userid="ADMIN" passwd="foo" interface="lan"
> primitive node-stonith-6-p stonith:external/ipmi \ op monitor
> interval="10m" timeout="1m" target_role="Started" \ params
> hostname="cnode-1-3-5 cnode-1-3-6" ipaddr="172.23.8.100"
> userid="ADMIN" passwd="foo" interface="lan"
> group group-glance-fs glance-fs-p glance-ip-p \ meta
> target-role="Started"
> ms ms-glance-drbd glance-drbd-p \ meta master-node-max="1"
> clone-max="2" clone-node-max="1" globally-unique="false"
> notify="true" target-role="Master"
> clone cloneLvm glance-lvm-p
> location drbd-fence-by-handler-ms-glance-drbd ms-glance-drbd \ rule
> $id="drbd-fence-by-handler-rule-ms-glance-drbd" $role="Master" -inf:
> #uname ne cnode-1-3-5
> location loc-node-stonith-5 node-stonith-5-p \ rule
> $id="loc-node-stonith-5-rule" -inf: #uname eq cnode-1-3-5
> location loc-node-stonith-6 node-stonith-6-p \ rule
> $id="loc-node-stonith-6-rule" -inf: #uname eq cnode-1-3-6
> colocation coloc-drbd-and-fs-group inf: ms-glance-drbd:Master
> group-glance-fs
This is backwards I believe... group-glance-fs runs on the
ms-glance-drbd:Master correct?
Colocation reads x on y so this would say that the ms-glance-drbd:Master has to
run on the group-glance-fs. That means if group-glance-fs isn't running then
ms-glance-drbd:Master can never run on that node.
Quote from Pacemaker Docs:
<rsc_colocation id="colocate" rsc="resource1" with-rsc="resource2"
score="INFINITY"/>
Remember, because INFINITY was used, if resource2 can't run on any of the
cluster nodes (for whatever reason) then resource1 will not be allowed to run.
> order order-glance-drbd-demote-before-stop-drbd inf:
> ms-glance-drbd:demote ms-glance-drbd:stop
Not needed
> order order-glance-drbd-promote-before-fs-group inf:
> ms-glance-drbd:promote group-glance-fs:start
Ordering statements are applied in reverse when stopping so the above
statements handles the demote/stop also making the ordering statements with
demote unneeded.
> order order-glance-drbd-start-before-drbd-promote inf:
> ms-glance-drbd:start ms-glance-drbd:promote
Not needed - start for ms resources... they should be started normally.
> order order-glance-fs-stop-before-demote-drbd inf:
> group-glance-fs:stop ms-glance-drbd:demote
Not needed
> order order-glance-lvm-before-drbd 0: cloneLvm ms-glance-drbd:start
> property $id="cib-bootstrap-options" \
> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
> cluster-infrastructure="openais" \ expected-quorum-votes="2" \
> stonith-enabled="true" \ no-quorum-policy="ignore" \
> last-lrm-refresh="1311899021"
> rsc_defaults $id="rsc-options" \ resource-stickiness="100"
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
