On 06/28/2016 11:24 AM, Marcin Dulak wrote:
>
>
> On Tue, Jun 28, 2016 at 5:04 AM, Andrew Beekhof <abeek...@redhat.com
> <mailto:abeek...@redhat.com>> wrote:
>
>     On Sun, Jun 26, 2016 at 6:05 AM, Marcin Dulak
>     <marcin.du...@gmail.com <mailto:marcin.du...@gmail.com>> wrote:
>     > Hi,
>     >
>     > I'm trying to get familiar with STONITH Block Devices (SBD) on a
>     3-node
>     > CentOS7 built in VirtualBox.
>     > The complete setup is available at
>     > https://github.com/marcindulak/vagrant-sbd-tutorial-centos7.git
>     > so hopefully with some help I'll be able to make it work.
>     >
>     > Question 1:
>     > The shared device /dev/sbd1 is the VirtualBox's "shareable hard
>     disk"
>     > https://www.virtualbox.org/manual/ch05.html#hdimagewrites
>     > will SBD fencing work with that type of storage?
>
>     unknown
>
>     >
>     > I start the cluster using vagrant_1.8.1 and virtualbox-4.3 with:
>     > $ vagrant up  # takes ~15 minutes
>     >
>     > The setup brings up the nodes, installs the necessary packages,
>     and prepares
>     > for the configuration of the pcs cluster.
>     > You can see which scripts the nodes execute at the bottom of the
>     > Vagrantfile.
>     > While there is 'yum -y install sbd' on CentOS7 the fence_sbd
>     agent has not
>     > been packaged yet.
>
>     you're not supposed to use it
>
>     > Therefore I rebuild Fedora 24 package using the latest
>     > https://github.com/ClusterLabs/fence-agents/archive/v4.0.22.tar.gz
>     > plus the update to the fence_sbd from
>     > https://github.com/ClusterLabs/fence-agents/pull/73
>     >
>     > The configuration is inspired by
>     > https://www.novell.com/support/kb/doc.php?id=7009485 and
>     >
>     
> https://www.suse.com/documentation/sle-ha-12/book_sleha/data/sec_ha_storage_protect_fencing.html
>     >
>     > Question 2:
>     > After reading
>     http://blog.clusterlabs.org/blog/2015/sbd-fun-and-profit I
>     > expect with just one stonith resource configured
>
>     there shouldn't be any stonith resources configured
>
>
> It's a test setup.
> Foundhttps://www.suse.com/documentation/sle-ha-12/book_sleha/data/sec_ha_storage_protect_fencing.html
>
> crm configure
> property stonith-enabled="true"
> property stonith-timeout="40s"
> primitive stonith_sbd stonith:external/sbd op start interval="0"
> timeout="15" start-delay="10"
> commit
> quit

For what is supported (self-fencing by watchdog) the stonith-resource is
just not needed because
of sbd and pacemaker interacting via cib.

>
>
> and trying to configure CentOS7 similarly.
>  
>
>
>     > a node will be fenced when I stop pacemaker and corosync `pcs
>     cluster stop
>     > node-1` or just `stonith_admin -F node-1`, but this is not the case.
>     >
>     > As can be seen below from uptime, the node-1 is not shutdown by
>     `pcs cluster
>     > stop node-1` executed on itself.
>     > I found some discussions on users@clusterlabs.org
>     <mailto:users@clusterlabs.org> about whether a node
>     > running SBD resource can fence itself,
>     > but the conclusion was not clear to me.
>
>     on RHEL and derivatives it can ONLY fence itself. the disk based
>     posion pill isn't supported yet
>
>
> once it's supported on RHEL I'll be ready :)
Meaning not supported in this case doesn't (just) mean that you will
receive - if at
all - very limited help, but that sbd is built with "--disable-shared-disk".
So unless you rebuild the package accordingly (with the other type of not
supported then ;-) ) testing with a block-device won't make much sense I
guess.
I'm already a little surprised that you get what you get ;-)
>
>
>     >
>     > Question 3:
>     > Neither node-1 is fenced by `stonith_admin -F node-1` executed
>     on node-2,
>     > despite the fact
>     > /var/log/messages on node-2 (the one currently running
>     MyStonith) reporting:
>     > ...
>     > notice: Operation 'off' [3309] (call 2 from stonith_admin.3288)
>     for host
>     > 'node-1' with device 'MyStonith' returned: 0 (OK)
>     > ...
>     > What is happening here?
>
>     have you tried looking at the sbd logs?
>     is the watchdog device functioning correctly?
>
>
> it turned out (suggested here
> http://clusterlabs.org/pipermail/users/2016-June/003355.html) that the
> reason for node-1 not being fenced by `stonith_admin -F node-1`
> executed on node-2
> was the previously executed `pcs cluster stop node-1`. In my setup SBD
> seems integrated with corosync/pacemaker and the latter command
> stopped the sbd service on node-1.
> Killing corosync on node-1 instead of `pcs cluster stop node-1` fences
> node-1 as expected:
>
> [root at node-1 ~]# killall -15 corosync
> Broadcast message from systemd-journald at node-1 (Sat 2016-06-25
> 21:55:07 EDT):
> sbd[4761]:  /dev/sdb1:    emerg: do_exit: Rebooting system: off
>
> I'm left with further questions: how to setup fence_sbd for the fenced
> node to shutdown instead of reboot?
> Both action=off or mode=onoff action=off options passed to fence_sbd
> when creating the MyStonith resource result in a reboot.
>
> [root at node-2 ~]# pcs stonith show MyStonith
>  Resource: MyStonith (class=stonith type=fence_sbd)
>   Attributes: devices=/dev/sdb1 power_timeout=21 action=off
>   Operations: monitor interval=60s (MyStonith-monitor-interval-60s)
>
> [root@node-2 ~]# pcs status
> Cluster name: mycluster
> Last updated: Tue Jun 28 04:55:43 2016        Last change: Tue Jun 28
> 04:48:03 2016 by root via cibadmin on node-1
> Stack: corosync
> Current DC: node-3 (version 1.1.13-10.el7_2.2-44eb2dd) - partition
> with quorum
> 3 nodes and 1 resource configured
>
> Online: [ node-1 node-2 node-3 ]
>
> Full list of resources:
>
>  MyStonith    (stonith:fence_sbd):    Started node-2
>
> PCSD Status:
>   node-1: Online
>   node-2: Online
>   node-3: Online
>
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
>
> Starting from the above cluster state:
> [root@node-2 ~]# stonith_admin -F node-1
> results also in a reboot of node-1 instead of shutdown.
>
> /var/log/messages after the last command show "reboot" on node-2
> ...
> Jun 28 04:49:39 localhost stonith-ng[3081]:  notice: Client
> stonith_admin.3179.fbc038ee wants to fence (off) 'node-1' with device
> '(any)'
> Jun 28 04:49:39 localhost stonith-ng[3081]:  notice: Initiating remote
> operation off for node-1: 8aea4f12-538d-41ab-bf20-0c8b0f72e2a3 (0)
> Jun 28 04:49:39 localhost stonith-ng[3081]:  notice: watchdog can not
> fence (off) node-1: static-list
> Jun 28 04:49:40 localhost stonith-ng[3081]:  notice: MyStonith can
> fence (off) node-1: dynamic-list
> Jun 28 04:49:40 localhost stonith-ng[3081]:  notice: watchdog can not
> fence (off) node-1: static-list
> Jun 28 04:49:44 localhost stonith-ng[3081]:  notice:
> crm_update_peer_proc: Node node-1[1] - state is now lost (was member)
> Jun 28 04:49:44 localhost stonith-ng[3081]:  notice: Removing node-1/1
> from the membership list
> Jun 28 04:49:44 localhost stonith-ng[3081]:  notice: Purged 1 peers
> with id=1 and/or uname=node-1 from the membership cache
> Jun 28 04:49:45 localhost stonith-ng[3081]:  notice: MyStonith can
> fence (reboot) node-1: dynamic-list
> Jun 28 04:49:45 localhost stonith-ng[3081]:  notice: watchdog can not
> fence (reboot) node-1: static-list
> Jun 28 04:49:46 localhost stonith-ng[3081]:  notice: Operation reboot
> of node-1 by node-3 for crmd.3063@node-3.36859c4e: OK
> Jun 28 04:50:00 localhost stonith-ng[3081]:  notice: Operation 'off'
> [3200] (call 2 from stonith_admin.3179) for host 'node-1' with device
> 'MyStonith' returned: 0 (OK)
> Jun 28 04:50:00 localhost stonith-ng[3081]:  notice: Operation off of
> node-1 by node-2 for stonith_admin.3179@node-2.8aea4f12: OK
> ...
>
>
> Another question (I think the question is valid also for a potential
> SUSE setup): What is the proper way of operating a cluster with SBD
> after node-1 was fenced?
>
> [root at node-2 ~]# sbd -d /dev/sdb1 list
> 0    node-3    clear
> 1    node-2    clear
> 2    node-1    off    node-2
>
> I found that executing sbd watch on node-1 clears the SBD status:
> [root at node-1 ~]# sbd -d /dev/sdb1 watch
> [root at node-1 ~]# sbd -d /dev/sdb1 list
> 0    node-3    clear
> 1    node-2    clear
> 2    node-1    clear
> Making sure that sbd is not running on node-1 (I can do that because
> node-1 is currently not a part of the cluster)
> [root at node-1 ~]# killall -15 sbd
> I have to kill sbd because it's integrated with corosync and corosync
> fails to start on node-1 with sbd already running.
>
> I can now join node-1 to the cluster from node-2:
> [root at node-2 ~]# pcs cluster start node-1
>
>
> Marcin
>  
>
>     >
>     > Question 4 (for the future):
>     > Assuming the node-1 was fenced, what is the way of operating SBD?
>     > I see the sbd lists now:
>     > 0       node-3  clear
>     > 1       node-1  off    node-2
>     > 2       node-2  clear
>     > How to clear the status of node-1?
>     >
>     > Question 5 (also for the future):
>     > While the relation 'stonith-timeout = Timeout (msgwait) + 20%'
>     presented
>     > at
>     >
>     
> https://www.suse.com/documentation/sle_ha/book_sleha/data/sec_ha_storage_protect_fencing.html
>     > is clearly described, I wonder about the relation of
>     'stonith-timeout'
>     > to other timeouts like the 'monitor interval=60s' reported by
>     `pcs stonith
>     > show MyStonith`.
>     >
>     > Here is how I configure the cluster and test it. The run.sh
>     script is
>     > attached.
>     >
>     > $ sh -x run01.sh 2>&1 | tee run01.txt
>     >
>     > with the result:
>     >
>     > $ cat run01.txt
>     >
>     > Each block below shows the executed ssh command and the result.
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs cluster auth -u hacluster -p
>     password node-1
>     > node-2 node-3'
>     > node-1: Authorized
>     > node-3: Authorized
>     > node-2: Authorized
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs cluster setup --name mycluster
>     node-1 node-2
>     > node-3'
>     > Shutting down pacemaker/corosync services...
>     > Redirecting to /bin/systemctl stop  pacemaker.service
>     > Redirecting to /bin/systemctl stop  corosync.service
>     > Killing any remaining services...
>     > Removing all cluster configuration files...
>     > node-1: Succeeded
>     > node-2: Succeeded
>     > node-3: Succeeded
>     > Synchronizing pcsd certificates on nodes node-1, node-2, node-3...
>     > node-1: Success
>     > node-3: Success
>     > node-2: Success
>     > Restaring pcsd on the nodes in order to reload the certificates...
>     > node-1: Success
>     > node-3: Success
>     > node-2: Success
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs cluster start --all'
>     > node-3: Starting Cluster...
>     > node-2: Starting Cluster...
>     > node-1: Starting Cluster...
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'corosync-cfgtool -s'
>     > Printing ring status.
>     > Local node ID 1
>     > RING ID 0
>     >     id    = 192.168.10.11
>     >     status    = ring 0 active with no faults
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs status corosync'
>     > Membership information
>     > ----------------------
>     >     Nodeid      Votes Name
>     >          1          1 node-1 (local)
>     >          2          1 node-2
>     >          3          1 node-3
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs status'
>     > Cluster name: mycluster
>     > WARNING: no stonith devices and stonith-enabled is not false
>     > Last updated: Sat Jun 25 15:40:51 2016        Last change: Sat
>     Jun 25
>     > 15:40:33 2016 by hacluster via crmd on node-2
>     > Stack: corosync
>     > Current DC: node-2 (version 1.1.13-10.el7_2.2-44eb2dd) -
>     partition with
>     > quorum
>     > 3 nodes and 0 resources configured
>     > Online: [ node-1 node-2 node-3 ]
>     > Full list of resources:
>     > PCSD Status:
>     >   node-1: Online
>     >   node-2: Online
>     >   node-3: Online
>     > Daemon Status:
>     >   corosync: active/disabled
>     >   pacemaker: active/disabled
>     >   pcsd: active/enabled
>     >
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'sbd -d /dev/sdb1 list'
>     > 0    node-3    clear
>     > 1    node-2    clear
>     > 2    node-1    clear
>     >
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'sbd -d /dev/sdb1 dump'
>     > ==Dumping header on disk /dev/sdb1
>     > Header version     : 2.1
>     > UUID               : 79f28167-a207-4f2a-a723-aa1c00bf1dee
>     > Number of slots    : 255
>     > Sector size        : 512
>     > Timeout (watchdog) : 10
>     > Timeout (allocate) : 2
>     > Timeout (loop)     : 1
>     > Timeout (msgwait)  : 20
>     > ==Header on disk /dev/sdb1 is dumped
>     >
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs stonith list'
>     > fence_sbd - Fence agent for sbd
>     >
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs stonith create MyStonith fence_sbd
>     > devices=/dev/sdb1 power_timeout=21 action=off'
>     > ssh node-1 -c sudo su - -c 'pcs property set stonith-enabled=true'
>     > ssh node-1 -c sudo su - -c 'pcs property set stonith-timeout=24s'
>     > ssh node-1 -c sudo su - -c 'pcs property'
>     > Cluster Properties:
>     >  cluster-infrastructure: corosync
>     >  cluster-name: mycluster
>     >  dc-version: 1.1.13-10.el7_2.2-44eb2dd
>     >  have-watchdog: true
>     >  stonith-enabled: true
>     >  stonith-timeout: 24s
>     >  stonith-watchdog-timeout: 10s
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs stonith show MyStonith'
>     >  Resource: MyStonith (class=stonith type=fence_sbd)
>     >   Attributes: devices=/dev/sdb1 power_timeout=21 action=off
>     >   Operations: monitor interval=60s (MyStonith-monitor-interval-60s)
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'pcs cluster stop node-1 '
>     > node-1: Stopping Cluster (pacemaker)...
>     > node-1: Stopping Cluster (corosync)...
>     >
>     >
>     >
>     > ############################
>     > ssh node-2 -c sudo su - -c 'pcs status'
>     > Cluster name: mycluster
>     > Last updated: Sat Jun 25 15:42:29 2016        Last change: Sat
>     Jun 25
>     > 15:41:09 2016 by root via cibadmin on node-1
>     > Stack: corosync
>     > Current DC: node-2 (version 1.1.13-10.el7_2.2-44eb2dd) -
>     partition with
>     > quorum
>     > 3 nodes and 1 resource configured
>     > Online: [ node-2 node-3 ]
>     > OFFLINE: [ node-1 ]
>     > Full list of resources:
>     >  MyStonith    (stonith:fence_sbd):    Started node-2
>     > PCSD Status:
>     >   node-1: Online
>     >   node-2: Online
>     >   node-3: Online
>     > Daemon Status:
>     >   corosync: active/disabled
>     >   pacemaker: active/disabled
>     >   pcsd: active/enabled
>     >
>     >
>     >
>     > ############################
>     > ssh node-2 -c sudo su - -c 'stonith_admin -F node-1 '
>     >
>     >
>     >
>     > ############################
>     > ssh node-2 -c sudo su - -c 'grep stonith-ng /var/log/messages'
>     > Jun 25 15:40:11 localhost stonith-ng[3102]:  notice: Additional
>     logging
>     > available in /var/log/cluster/corosync.log
>     > Jun 25 15:40:11 localhost stonith-ng[3102]:  notice: Connecting
>     to cluster
>     > infrastructure: corosync
>     > Jun 25 15:40:11 localhost stonith-ng[3102]:  notice:
>     crm_update_peer_proc:
>     > Node node-2[2] - state is now member (was (null))
>     > Jun 25 15:40:12 localhost stonith-ng[3102]:  notice: Watching
>     for stonith
>     > topology changes
>     > Jun 25 15:40:12 localhost stonith-ng[3102]:  notice: Added
>     'watchdog' to the
>     > device list (1 active devices)
>     > Jun 25 15:40:12 localhost stonith-ng[3102]:  notice:
>     crm_update_peer_proc:
>     > Node node-3[3] - state is now member (was (null))
>     > Jun 25 15:40:12 localhost stonith-ng[3102]:  notice:
>     crm_update_peer_proc:
>     > Node node-1[1] - state is now member (was (null))
>     > Jun 25 15:40:12 localhost stonith-ng[3102]:  notice: New
>     watchdog timeout
>     > 10s (was 0s)
>     > Jun 25 15:41:03 localhost stonith-ng[3102]:  notice: Relying on
>     watchdog
>     > integration for fencing
>     > Jun 25 15:41:04 localhost stonith-ng[3102]:  notice: Added
>     'MyStonith' to
>     > the device list (2 active devices)
>     > Jun 25 15:41:54 localhost stonith-ng[3102]:  notice:
>     crm_update_peer_proc:
>     > Node node-1[1] - state is now lost (was member)
>     > Jun 25 15:41:54 localhost stonith-ng[3102]:  notice: Removing
>     node-1/1 from
>     > the membership list
>     > Jun 25 15:41:54 localhost stonith-ng[3102]:  notice: Purged 1
>     peers with
>     > id=1 and/or uname=node-1 from the membership cache
>     > Jun 25 15:42:33 localhost stonith-ng[3102]:  notice: Client
>     > stonith_admin.3288.eb400ac9 wants to fence (off) 'node-1' with
>     device
>     > '(any)'
>     > Jun 25 15:42:33 localhost stonith-ng[3102]:  notice: Initiating
>     remote
>     > operation off for node-1: 848cd1e9-55e4-4abc-8d7a-3762eaaf9ab4 (0)
>     > Jun 25 15:42:33 localhost stonith-ng[3102]:  notice: watchdog
>     can not fence
>     > (off) node-1: static-list
>     > Jun 25 15:42:33 localhost stonith-ng[3102]:  notice: MyStonith
>     can fence
>     > (off) node-1: dynamic-list
>     > Jun 25 15:42:33 localhost stonith-ng[3102]:  notice: watchdog
>     can not fence
>     > (off) node-1: static-list
>     > Jun 25 15:42:54 localhost stonith-ng[3102]:  notice: Operation
>     'off' [3309]
>     > (call 2 from stonith_admin.3288) for host 'node-1' with device
>     'MyStonith'
>     > returned: 0 (OK)
>     > Jun 25 15:42:54 localhost stonith-ng[3102]:  notice: Operation
>     off of node-1
>     > by node-2 for stonith_admin.3288@node-2.848cd1e9: OK
>     > Jun 25 15:42:54 localhost stonith-ng[3102]: warning:
>     new_event_notification
>     > (3102-3288-12 <tel:%283102-3288-12>): Broken pipe (32)
>     > Jun 25 15:42:54 localhost stonith-ng[3102]: warning: st_notify_fence
>     > notification of client stonith_admin.3288.eb400a failed: Broken
>     pipe (-32)
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'sbd -d /dev/sdb1 list'
>     > 0    node-3    clear
>     > 1    node-2    clear
>     > 2    node-1    off    node-2
>     >
>     >
>     >
>     > ############################
>     > ssh node-1 -c sudo su - -c 'uptime'
>     >  15:43:31 up 21 min,  2 users,  load average: 0.25, 0.18, 0.11
>     >
>     >
>     >
>     > Cheers,
>     >
>     > Marcin
>     >
>     >
>     > _______________________________________________
>     > Users mailing list: Users@clusterlabs.org
>     <mailto:Users@clusterlabs.org>
>     > http://clusterlabs.org/mailman/listinfo/users
>     >
>     > Project Home: http://www.clusterlabs.org
>     > Getting started:
>     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     > Bugs: http://bugs.clusterlabs.org
>     >
>
>     _______________________________________________
>     Users mailing list: Users@clusterlabs.org
>     <mailto:Users@clusterlabs.org>
>     http://clusterlabs.org/mailman/listinfo/users
>
>     Project Home: http://www.clusterlabs.org
>     Getting started:
>     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     Bugs: http://bugs.clusterlabs.org
>
>
>
>
> _______________________________________________
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


_______________________________________________
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Reply via email to