[ClusterLabs] Questions about SBD behavior
Hi, I am checking the watchdog function of SBD (without shared block-device). In a two-node cluster, if one cluster is stopped, watchdog is triggered on the remaining node. Is this the designed behavior? [vmrh75b]# cat /etc/corosync/corosync.conf (snip) quorum { provider: corosync_votequorum two_node: 1 } [vmrh75b]# cat /etc/sysconfig/sbd # This file has been generated by pcs. SBD_DELAY_START=no ## SBD_DEVICE="/dev/vdb1" SBD_OPTS="-vvv" SBD_PACEMAKER=yes SBD_STARTMODE=always SBD_WATCHDOG_DEV=/dev/watchdog SBD_WATCHDOG_TIMEOUT=5 [vmrh75b]# crm_mon -r1 Stack: corosync Current DC: vmrh75a (version 2.0.0-0.1.rc4.el7-2.0.0-rc4) - partition with quorum Last updated: Fri May 25 13:36:07 2018 Last change: Fri May 25 13:35:22 2018 by root via cibadmin on vmrh75a 2 nodes configured 0 resources configured Online: [ vmrh75a vmrh75b ] No resources [vmrh75b]# pcs property show Cluster Properties: cluster-infrastructure: corosync cluster-name: my_cluster dc-version: 2.0.0-0.1.rc4.el7-2.0.0-rc4 have-watchdog: true stonith-enabled: false [vmrh75b]# ps -ef | egrep "sbd|coro|pace" root 2169 1 0 13:34 ?00:00:00 sbd: inquisitor root 2170 2169 0 13:34 ?00:00:00 sbd: watcher: Pacemaker root 2171 2169 0 13:34 ?00:00:00 sbd: watcher: Cluster root 2172 1 0 13:34 ?00:00:00 corosync root 2179 1 0 13:34 ?00:00:00 /usr/sbin/pacemakerd -f haclust+ 2180 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-based root 2181 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-fenced root 2182 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-execd haclust+ 2183 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-attrd haclust+ 2184 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-schedulerd haclust+ 2185 2179 0 13:34 ?00:00:00 /usr/libexec/pacemaker/pacemaker-controld [vmrh75b]# pcs cluster stop vmrh75a vmrh75a: Stopping Cluster (pacemaker)... vmrh75a: Stopping Cluster (corosync)... [vmrh75b]# tail -F /var/log/messages May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: Our peer on the DC (vmrh75a) is dead May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition S_NOT_DC -> S_ELECTION May 25 13:37:00 vmrh75b pacemaker-controld[2185]: notice: State transition S_ELECTION -> S_INTEGRATION May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Node vmrh75a state is now lost May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Removing all vmrh75a attributes for peer loss May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Lost attribute writer vmrh75a May 25 13:37:00 vmrh75b pacemaker-attrd[2183]: notice: Purged 1 peer with id=1 and/or uname=vmrh75a from the membership cache May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Node vmrh75a state is now lost May 25 13:37:00 vmrh75b pacemaker-fenced[2181]: notice: Purged 1 peer with id=1 and/or uname=vmrh75a from the membership cache May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Node vmrh75a state is now lost May 25 13:37:00 vmrh75b pacemaker-based[2180]: notice: Purged 1 peer with id=1 and/or uname=vmrh75a from the membership cache May 25 13:37:00 vmrh75b pacemaker-controld[2185]: warning: Input I_ELECTION_DC received in state S_INTEGRATION from do_election_check May 25 13:37:01 vmrh75b sbd[2171]: cluster: warning: set_servant_health: Connected to corosync but requires both nodes present May 25 13:37:01 vmrh75b sbd[2171]: cluster: warning: notify_parent: Notifying parent: UNHEALTHY (6) May 25 13:37:01 vmrh75b sbd[2169]: warning: inquisitor_child: cluster health check: UNHEALTHY May 25 13:37:01 vmrh75b sbd[2169]: warning: inquisitor_child: Servant cluster is outdated (age: 226) May 25 13:37:01 vmrh75b sbd[2170]: pcmk: notice: unpack_config: Watchdog will be used via SBD if fencing is required May 25 13:37:01 vmrh75b sbd[2170]: pcmk: info: determine_online_status: Node vmrh75b is online May 25 13:37:01 vmrh75b sbd[2170]: pcmk: info: unpack_node_loop: Node 2 is already processed May 25 13:37:01 vmrh75b sbd[2170]: pcmk: info: unpack_node_loop: Node 2 is already processed May 25 13:37:01 vmrh75b sbd[2171]: cluster: warning: notify_parent: Notifying parent: UNHEALTHY (6) May 25 13:37:01 vmrh75b corosync[2172]: [TOTEM ] A new membership (192.168.28.132:5712) was formed. Members left: 1 May 25 13:37:01 vmrh75b corosync[2172]: [QUORUM] Members[1]: 2 May 25 13:37:01 vmrh75b corosync[2172]: [MAIN ] Completed service synchronization, ready to provide service. May 25 13:37:01 vmrh75b pacemakerd[2179]: notice: Node vmrh75a state is now lost May 25 13:37:01 vmrh75b pacemaker-controld[2185]: notice: Node vmrh75a state is now lost May 25 13:37:01 vmrh75b pacemaker-controld[2185]: warning: Stonith/shutdown of node vmrh75a was not expected May 25 13:37:02 vmrh75b sbd[2171]: cluster: warning:
Re: [ClusterLabs] DLM fencing
On Thu, May 24, 2018 at 10:40 AM, Ken Gaillotwrote: > On Thu, 2018-05-24 at 16:14 +0200, Klaus Wenninger wrote: >> On 05/24/2018 04:03 PM, Ken Gaillot wrote: >> > On Thu, 2018-05-24 at 06:47 -0400, Jason Gauthier wrote: >> > > On Thu, May 24, 2018 at 12:19 AM, Andrei Borzenkov > > > il.c >> > > om> wrote: >> > > > 24.05.2018 02:57, Jason Gauthier пишет: >> > > > > I'm fairly new to clustering under Linux. I've basically >> > > > > have >> > > > > one shared >> > > > > storage resource right now, using dlm, and gfs2. >> > > > > I'm using fibre channel and when both of my nodes are up (2 >> > > > > node >> > > > > cluster) >> > > > > dlm and gfs2 seem to be operating perfectly. >> > > > > If I reboot node B, node A works fine and vice-versa. >> > > > > >> > > > > When node B goes offline unexpectedly, and become unclean, >> > > > > dlm >> > > > > seems to >> > > > > block all IO to the shared storage. >> > > > > >> > > > > dlm knows node B is down: >> > > > > >> > > > > # dlm_tool status >> > > > > cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 >> > > > > daemon now 865695 fence_pid 18186 >> > > > > fence 1084772369 nodedown pid 18186 actor 1084772368 fail >> > > > > 1527119246 fence >> > > > > 0 now 1527119524 >> > > > > node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 >> > > > > node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 >> > > > > at 0 >> > > > > 0 >> > > > > >> > > > > on the same server, I see these messages in my daemon.log >> > > > > May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: >> > > > > Could >> > > > > not kick >> > > > > (reboot) node 1084772369/(null) : No route to host (-113) >> > > > > May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error >> > > > > -113 >> > > > > nodeid >> > > > > 1084772369 >> > > > > >> > > > > I can recover from the situation by forcing it (or bring the >> > > > > other node >> > > > > back online) >> > > > > dlm_tool fence_ack 1084772369 >> > > > > >> > > > > cluster config is pretty straighforward. >> > > > > node 1084772368: alpha >> > > > > node 1084772369: beta >> > > > > primitive p_dlm_controld ocf:pacemaker:controld \ >> > > > > op monitor interval=60 timeout=60 \ >> > > > > meta target-role=Started \ >> > > > > params args="-K -L -s 1" >> > > > > primitive p_fs_gfs2 Filesystem \ >> > > > > params device="/dev/sdb2" directory="/vms" >> > > > > fstype=gfs2 >> > > > > primitive stonith_sbd stonith:external/sbd \ >> > > > > params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ >> > > > > meta target-role=Started >> > > > >> > > > What is the status of stonith resource? Did you configure SBD >> > > > fencing >> > > > properly? >> > > >> > > I believe so. It's shown above in my cluster config. >> > > >> > > > Is sbd daemon up and running with proper parameters? >> > > >> > > Well, no, apparently sbd isn't running.With dlm, and gfs2, >> > > the >> > > cluster controls handling launching of the daemons. >> > > I assumed the same here, since the resource shows that it is up. >> > >> > Unlike other services, sbd must be up before the cluster starts in >> > order for the cluster to use it properly. (Notice the "have- >> > watchdog=false" in your cib-bootstrap-options ... that means the >> > cluster didn't find sbd running.) >> > >> > Also, even storage-based sbd requires a working hardware watchdog >> > for >> > the actual self-fencing. SBD_WATCHDOG_DEV in /etc/sysconfig/sbd >> > should >> > list the watchdog device. Also sbd_device in your cluster config >> > should >> > match SBD_DEVICE in /etc/sysconfig/sbd. >> > >> > If you want the cluster to recover services elsewhere after a node >> > self-fences (which I'm sure you do), you also need to set the >> > stonith- >> > watchdog-timeout cluster property to something greater than the >> > value >> > of SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd. The cluster will >> > wait >> > that long and then assume the node fenced itself. Thanks. So, for whatever reason, sbd was not running. I went ahead and got /etc/default/sbd (debian) configured. I can't start the service manually due to dependencies, but I rebooted node B and it came up. Node A would not, I ended up rebooting both nodes at the same time, and sbd was running on both. I forced a failure of node B, and after a few seconds node A was able to access the shared storage. Definite improvement! >> Actually for the case that there is a shared disk a successful >> fencing-attempt via the sbd-fencing-resource should be enough >> for the node to be assumed down. >> In case of a 2-node-setup I would even discourage setting >> stonith-watchdog-timeout as we need a real quorum-mechanism >> for that to work. > > Ah, thanks -- I've updated the wiki how-to, feel free to clarify > further: > > https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker > >> >> Regards, >> Klaus >> >> > >> > > Online: [ alpha beta ] >> > > >> > > Full
Re: [ClusterLabs] pcsd processes using 100% CPU
> gcore is part of gdb: > https://packages.ubuntu.com/xenial/amd64/gdb/filelist > > Note that using the utility should have no observable influence > on the running process in question. When I ran gcore on the pid, it produced a whole bunch of memory read errors like this: warning: Memory read failed for corefile section, 8192 bytes at 0x93177000. warning: Memory read failed for corefile section, 1048576 bytes at 0x93179000. warning: Memory read failed for corefile section, 4096 bytes at 0x93378000. warning: Memory read failed for corefile section, 4096 bytes at 0x93379000. warning: Memory read failed for corefile section, 8192 bytes at 0x9337a000. warning: Memory read failed for corefile section, 1048576 bytes at 0x9337c000. warning: Memory read failed for corefile section, 4096 bytes at 0x9357c000. warning: Memory read failed for corefile section, 4096 bytes at 0x9357d000. warning: Memory read failed for corefile section, 8192 bytes at 0x9357e000. warning: Memory read failed for corefile section, 1048576 bytes at 0x9358. warning: Memory read failed for corefile section, 4096 bytes at 0x9377f000. warning: Memory read failed for corefile section, 4096 bytes at 0x9378. warning: Memory read failed for corefile section, 4096 bytes at 0x93982000. warning: Memory read failed for corefile section, 4096 bytes at 0x93983000. warning: Memory read failed for corefile section, 4096 bytes at 0x93fac000. warning: Memory read failed for corefile section, 4096 bytes at 0x93fad000. warning: Memory read failed for corefile section, 4096 bytes at 0x941b6000. warning: Memory read failed for corefile section, 4096 bytes at 0x941b7000. warning: Memory read failed for corefile section, 188416 bytes at 0x941b8000. warning: Memory read failed for corefile section, 4096 bytes at 0x943e8000. warning: Memory read failed for corefile section, 4096 bytes at 0x943e9000. warning: Memory read failed for corefile section, 4096 bytes at 0x94668000. warning: Memory read failed for corefile section, 4096 bytes at 0x94669000. Nonetheless, it did produce a core file. > load it to gdb (together with appropriate > debug symbols) and from here, you can investigate further. I'm sorry, but I don't really know how to do that. Could you provide some instructions of what I should do? I tried running `gdb --core=core.24923` but that threw errors: Failed to read a valid object file image from memory. Core was generated by `/usr/bin/ruby'. Program terminated with signal SIGSTOP, Stopped (signal). #0 0x0254a790 in ?? () [Current thread is 1 (LWP 24923)] When I then looked at ps again, the process was no longer running, so I guess one of the above commands killed it. So, I don't have a running process to work with anymore. The core file is 1GB large! It gzipped down to 5MB. Would it be useful for me to send this to you to look at directly? Thanks, -- Casey ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DLM fencing
On Thu, 2018-05-24 at 16:14 +0200, Klaus Wenninger wrote: > On 05/24/2018 04:03 PM, Ken Gaillot wrote: > > On Thu, 2018-05-24 at 06:47 -0400, Jason Gauthier wrote: > > > On Thu, May 24, 2018 at 12:19 AM, Andrei Borzenkov> > il.c > > > om> wrote: > > > > 24.05.2018 02:57, Jason Gauthier пишет: > > > > > I'm fairly new to clustering under Linux. I've basically > > > > > have > > > > > one shared > > > > > storage resource right now, using dlm, and gfs2. > > > > > I'm using fibre channel and when both of my nodes are up (2 > > > > > node > > > > > cluster) > > > > > dlm and gfs2 seem to be operating perfectly. > > > > > If I reboot node B, node A works fine and vice-versa. > > > > > > > > > > When node B goes offline unexpectedly, and become unclean, > > > > > dlm > > > > > seems to > > > > > block all IO to the shared storage. > > > > > > > > > > dlm knows node B is down: > > > > > > > > > > # dlm_tool status > > > > > cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 > > > > > daemon now 865695 fence_pid 18186 > > > > > fence 1084772369 nodedown pid 18186 actor 1084772368 fail > > > > > 1527119246 fence > > > > > 0 now 1527119524 > > > > > node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 > > > > > node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 > > > > > at 0 > > > > > 0 > > > > > > > > > > on the same server, I see these messages in my daemon.log > > > > > May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: > > > > > Could > > > > > not kick > > > > > (reboot) node 1084772369/(null) : No route to host (-113) > > > > > May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error > > > > > -113 > > > > > nodeid > > > > > 1084772369 > > > > > > > > > > I can recover from the situation by forcing it (or bring the > > > > > other node > > > > > back online) > > > > > dlm_tool fence_ack 1084772369 > > > > > > > > > > cluster config is pretty straighforward. > > > > > node 1084772368: alpha > > > > > node 1084772369: beta > > > > > primitive p_dlm_controld ocf:pacemaker:controld \ > > > > > op monitor interval=60 timeout=60 \ > > > > > meta target-role=Started \ > > > > > params args="-K -L -s 1" > > > > > primitive p_fs_gfs2 Filesystem \ > > > > > params device="/dev/sdb2" directory="/vms" > > > > > fstype=gfs2 > > > > > primitive stonith_sbd stonith:external/sbd \ > > > > > params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ > > > > > meta target-role=Started > > > > > > > > What is the status of stonith resource? Did you configure SBD > > > > fencing > > > > properly? > > > > > > I believe so. It's shown above in my cluster config. > > > > > > > Is sbd daemon up and running with proper parameters? > > > > > > Well, no, apparently sbd isn't running.With dlm, and gfs2, > > > the > > > cluster controls handling launching of the daemons. > > > I assumed the same here, since the resource shows that it is up. > > > > Unlike other services, sbd must be up before the cluster starts in > > order for the cluster to use it properly. (Notice the "have- > > watchdog=false" in your cib-bootstrap-options ... that means the > > cluster didn't find sbd running.) > > > > Also, even storage-based sbd requires a working hardware watchdog > > for > > the actual self-fencing. SBD_WATCHDOG_DEV in /etc/sysconfig/sbd > > should > > list the watchdog device. Also sbd_device in your cluster config > > should > > match SBD_DEVICE in /etc/sysconfig/sbd. > > > > If you want the cluster to recover services elsewhere after a node > > self-fences (which I'm sure you do), you also need to set the > > stonith- > > watchdog-timeout cluster property to something greater than the > > value > > of SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd. The cluster will > > wait > > that long and then assume the node fenced itself. > > Actually for the case that there is a shared disk a successful > fencing-attempt via the sbd-fencing-resource should be enough > for the node to be assumed down. > In case of a 2-node-setup I would even discourage setting > stonith-watchdog-timeout as we need a real quorum-mechanism > for that to work. Ah, thanks -- I've updated the wiki how-to, feel free to clarify further: https://wiki.clusterlabs.org/wiki/Using_SBD_with_Pacemaker > > Regards, > Klaus > > > > > > Online: [ alpha beta ] > > > > > > Full list of resources: > > > > > > stonith_sbd(stonith:external/sbd): Started alpha > > > Clone Set: cl_gfs2 [g_gfs2] > > > Started: [ alpha beta ] > > > > > > > > > > What is output of > > > > sbd -d /dev/sdb1 dump > > > > sbd -d /dev/sdb1 list > > > > > > Both nodes seem fine. > > > > > > 0 alpha testbeta > > > 1 betatestalpha > > > > > > > > > > on both nodes? Does > > > > > > > > sbd -d /dev/sdb1 message test > > > > > > > > work in both directions? > > > > > > It doesn't return an error, yet without a daemon running, I don't > > > think the message is
Re: [ClusterLabs] DLM fencing
On 05/24/2018 04:03 PM, Ken Gaillot wrote: > On Thu, 2018-05-24 at 06:47 -0400, Jason Gauthier wrote: >> On Thu, May 24, 2018 at 12:19 AM, Andrei Borzenkov> om> wrote: >>> 24.05.2018 02:57, Jason Gauthier пишет: I'm fairly new to clustering under Linux. I've basically have one shared storage resource right now, using dlm, and gfs2. I'm using fibre channel and when both of my nodes are up (2 node cluster) dlm and gfs2 seem to be operating perfectly. If I reboot node B, node A works fine and vice-versa. When node B goes offline unexpectedly, and become unclean, dlm seems to block all IO to the shared storage. dlm knows node B is down: # dlm_tool status cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 daemon now 865695 fence_pid 18186 fence 1084772369 nodedown pid 18186 actor 1084772368 fail 1527119246 fence 0 now 1527119524 node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 at 0 0 on the same server, I see these messages in my daemon.log May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: Could not kick (reboot) node 1084772369/(null) : No route to host (-113) May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error -113 nodeid 1084772369 I can recover from the situation by forcing it (or bring the other node back online) dlm_tool fence_ack 1084772369 cluster config is pretty straighforward. node 1084772368: alpha node 1084772369: beta primitive p_dlm_controld ocf:pacemaker:controld \ op monitor interval=60 timeout=60 \ meta target-role=Started \ params args="-K -L -s 1" primitive p_fs_gfs2 Filesystem \ params device="/dev/sdb2" directory="/vms" fstype=gfs2 primitive stonith_sbd stonith:external/sbd \ params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ meta target-role=Started >>> What is the status of stonith resource? Did you configure SBD >>> fencing >>> properly? >> I believe so. It's shown above in my cluster config. >> >>> Is sbd daemon up and running with proper parameters? >> Well, no, apparently sbd isn't running.With dlm, and gfs2, the >> cluster controls handling launching of the daemons. >> I assumed the same here, since the resource shows that it is up. > Unlike other services, sbd must be up before the cluster starts in > order for the cluster to use it properly. (Notice the "have- > watchdog=false" in your cib-bootstrap-options ... that means the > cluster didn't find sbd running.) > > Also, even storage-based sbd requires a working hardware watchdog for > the actual self-fencing. SBD_WATCHDOG_DEV in /etc/sysconfig/sbd should > list the watchdog device. Also sbd_device in your cluster config should > match SBD_DEVICE in /etc/sysconfig/sbd. > > If you want the cluster to recover services elsewhere after a node > self-fences (which I'm sure you do), you also need to set the stonith- > watchdog-timeout cluster property to something greater than the value > of SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd. The cluster will wait > that long and then assume the node fenced itself. Actually for the case that there is a shared disk a successful fencing-attempt via the sbd-fencing-resource should be enough for the node to be assumed down. In case of a 2-node-setup I would even discourage setting stonith-watchdog-timeout as we need a real quorum-mechanism for that to work. Regards, Klaus > >> Online: [ alpha beta ] >> >> Full list of resources: >> >> stonith_sbd(stonith:external/sbd): Started alpha >> Clone Set: cl_gfs2 [g_gfs2] >> Started: [ alpha beta ] >> >> >>> What is output of >>> sbd -d /dev/sdb1 dump >>> sbd -d /dev/sdb1 list >> Both nodes seem fine. >> >> 0 alpha testbeta >> 1 betatestalpha >> >> >>> on both nodes? Does >>> >>> sbd -d /dev/sdb1 message test >>> >>> work in both directions? >> It doesn't return an error, yet without a daemon running, I don't >> think the message is received either. >> >> >>> Does manual fencing using stonith_admin work? >> I'm not sure at the moment. I think I need to look into why the >> daemon isn't running. >> group g_gfs2 p_dlm_controld p_fs_gfs2 clone cl_gfs2 g_gfs2 \ meta interleave=true target-role=Started location cli-prefer-cl_gfs2 cl_gfs2 role=Started inf: alpha property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.16-94ff4df \ cluster-infrastructure=corosync \ cluster-name=zeta \ last-lrm-refresh=1525523370 \ stonith-enabled=true \ stonith-timeout=20s Any pointers would be appreciated. I feel like this should be working but I'm not sure if I've missed
Re: [ClusterLabs] DLM fencing
On Thu, 2018-05-24 at 06:47 -0400, Jason Gauthier wrote: > On Thu, May 24, 2018 at 12:19 AM, Andrei Borzenkovom> wrote: > > 24.05.2018 02:57, Jason Gauthier пишет: > > > I'm fairly new to clustering under Linux. I've basically have > > > one shared > > > storage resource right now, using dlm, and gfs2. > > > I'm using fibre channel and when both of my nodes are up (2 node > > > cluster) > > > dlm and gfs2 seem to be operating perfectly. > > > If I reboot node B, node A works fine and vice-versa. > > > > > > When node B goes offline unexpectedly, and become unclean, dlm > > > seems to > > > block all IO to the shared storage. > > > > > > dlm knows node B is down: > > > > > > # dlm_tool status > > > cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 > > > daemon now 865695 fence_pid 18186 > > > fence 1084772369 nodedown pid 18186 actor 1084772368 fail > > > 1527119246 fence > > > 0 now 1527119524 > > > node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 > > > node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 at 0 > > > 0 > > > > > > on the same server, I see these messages in my daemon.log > > > May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: Could > > > not kick > > > (reboot) node 1084772369/(null) : No route to host (-113) > > > May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error -113 > > > nodeid > > > 1084772369 > > > > > > I can recover from the situation by forcing it (or bring the > > > other node > > > back online) > > > dlm_tool fence_ack 1084772369 > > > > > > cluster config is pretty straighforward. > > > node 1084772368: alpha > > > node 1084772369: beta > > > primitive p_dlm_controld ocf:pacemaker:controld \ > > > op monitor interval=60 timeout=60 \ > > > meta target-role=Started \ > > > params args="-K -L -s 1" > > > primitive p_fs_gfs2 Filesystem \ > > > params device="/dev/sdb2" directory="/vms" fstype=gfs2 > > > primitive stonith_sbd stonith:external/sbd \ > > > params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ > > > meta target-role=Started > > > > What is the status of stonith resource? Did you configure SBD > > fencing > > properly? > > I believe so. It's shown above in my cluster config. > > > Is sbd daemon up and running with proper parameters? > > Well, no, apparently sbd isn't running.With dlm, and gfs2, the > cluster controls handling launching of the daemons. > I assumed the same here, since the resource shows that it is up. Unlike other services, sbd must be up before the cluster starts in order for the cluster to use it properly. (Notice the "have- watchdog=false" in your cib-bootstrap-options ... that means the cluster didn't find sbd running.) Also, even storage-based sbd requires a working hardware watchdog for the actual self-fencing. SBD_WATCHDOG_DEV in /etc/sysconfig/sbd should list the watchdog device. Also sbd_device in your cluster config should match SBD_DEVICE in /etc/sysconfig/sbd. If you want the cluster to recover services elsewhere after a node self-fences (which I'm sure you do), you also need to set the stonith- watchdog-timeout cluster property to something greater than the value of SBD_WATCHDOG_TIMEOUT in /etc/sysconfig/sbd. The cluster will wait that long and then assume the node fenced itself. > > Online: [ alpha beta ] > > Full list of resources: > > stonith_sbd(stonith:external/sbd): Started alpha > Clone Set: cl_gfs2 [g_gfs2] > Started: [ alpha beta ] > > > > What is output of > > sbd -d /dev/sdb1 dump > > sbd -d /dev/sdb1 list > > Both nodes seem fine. > > 0 alpha testbeta > 1 betatestalpha > > > > on both nodes? Does > > > > sbd -d /dev/sdb1 message test > > > > work in both directions? > > It doesn't return an error, yet without a daemon running, I don't > think the message is received either. > > > > Does manual fencing using stonith_admin work? > > I'm not sure at the moment. I think I need to look into why the > daemon isn't running. > > > > group g_gfs2 p_dlm_controld p_fs_gfs2 > > > clone cl_gfs2 g_gfs2 \ > > > meta interleave=true target-role=Started > > > location cli-prefer-cl_gfs2 cl_gfs2 role=Started inf: alpha > > > property cib-bootstrap-options: \ > > > have-watchdog=false \ > > > dc-version=1.1.16-94ff4df \ > > > cluster-infrastructure=corosync \ > > > cluster-name=zeta \ > > > last-lrm-refresh=1525523370 \ > > > stonith-enabled=true \ > > > stonith-timeout=20s > > > > > > Any pointers would be appreciated. I feel like this should be > > > working but > > > I'm not sure if I've missed something. > > > > > > Thanks, > > > > > > Jason > > > > > > > > > > > > ___ > > > Users mailing list: Users@clusterlabs.org > > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > > > Project Home: http://www.clusterlabs.org > > > Getting started:
Re: [ClusterLabs] pcsd processes using 100% CPU
On 23/05/18 12:43 -0600, Casey & Gina wrote: > I don't have gcore installed and don't know which package might > provide it. I also don't have experience with gdb but am happy to > try anything suggested to help figure out what's going on. gcore is part of gdb: https://packages.ubuntu.com/xenial/amd64/gdb/filelist Note that using the utility should have no observable influence on the running process in question. > The pcs version is 0.9.149, as packaged by Debian and inherited by > Ubuntu. Thanks, this will at least help to pinpoint any other similar observations. -- Jan (Poki) pgpnyRRtQekGC.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] gfs2-utils 3.2.0 released
Hi all, I am happy to announce the 3.2.0 release of gfs2-utils. This is an important release adding support for new on-disk features introduced in the 4.16 kernel. In fact it is required when building against 4.16 and later kernel headers due to poor assumptions made by earlier gfs2-utils relating to structure size changes. Building earlier gfs2-utils against 4.16 headers will result in test suite failures. (Thanks to Valentin Vidic for reporting these issues.) This release adds basic support for new gfs2 on-disk features: * Resource group header CRCs * "Next resource group" pointers in resource groups * Journal log header block CRCs * Journal log header timestamp fields * Statfs accounting fields in journal log headers Future releases will build on this work to take advantage of these new features, particularly for improving checking and performance. Other notable changes: * mkfs.gfs2 now scales down the journal size to make better use of small devices by default * Better detection of bad device topology * fsck.gfs2 no longer accepts conflicting -p, -n and -y options * Saving of symlinks in gfs2_edit savemeta has been fixed * Fixes for issues caught by static analysis and new compiler warnings * New test cases in the testsuite * Various minor code cleanups and improvements See below for a complete list of changes. The source tarball is available from: https://releases.pagure.org/gfs2-utils/gfs2-utils-3.2.0.tar.gz Please report bugs against the gfs2-utils component of Fedora rawhide: https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora=gfs2-utils=rawhide Regards, Andy Changes since version 3.1.10: Andrew Price (66): gfs2_grow: Disable rgrp alignment when dev topology is unsuitable mkfs.gfs2: Free unnecessary cached pages, disable readahead mkfs.gfs2: Fix resource group alignment issue libgfs2: Issue one write per rgrp when creating them libgfs2: Switch gfs2_dinode_out to use a char buffer libgfs2: Switch gfs2_log_header_out to use a char buffer gfs2-utils: Change instances of "gfs2_fsck" to "fsck.gfs2" gfs2_convert: Fix fgets return value warning fsck.gfs2: Fix snprintf truncation warning fsck.gfs2: Fix unchecked return value warning gfs2_grow: Fix unchecked ftruncate return value warning gfs2_grow: Remove unnecessary nesting in fix_rindex() gfs2_edit savemeta: Fix up saving of dinodes/symlinks gfs2_edit savemeta: Use size_t for saved structure lengths fsck.gfs2: Make -p, -n and -y conflicting options gfs2_edit: Print offsets of indirect pointers gfs2-utils configure: Check for rg_skip libgfs2: Add rgrp_skip support mkfs.gfs2: Pull place_journals() out of place_rgrps() mkfs.gfs2: Set the rg_skip field in new rgrps gfs2-utils configure: Check for rg_data0, rg_data and rg_bitbytes libgfs2: Add support for rg_data0, rg_data and rg_bitbytes mkfs.gfs2: Set the rg_data0, rg_data and rg_bitbytes fields libgfs2: Add support for rg_crc Add basic support for v2 log headers mkfs.gfs2: Scale down journal size for smaller devices gfs2-utils: Remove make-tarball.sh glocktop: Remove a non-existent flag from the usage string fsck.gfs2: Don't check lh_crc for older filesystems libgfs2: Remove unused lock* fields from gfs2_sbd libgfs2: Remove sb_addr from gfs2_sbd libgfs2: Plug an alignment hole in gfs2_sbd libgfs2: Plug an alignment hole in gfs2_buffer_head libgfs2: Plug an alignment hole in gfs2_inode libgfs2: Remove gfs2_meta_header_out_bh() libgfs2: Don't pass an extlen to block_map where not required libgfs2: Don't use a buffer_head in gfs2_meta_header_in libgfs2: Don't use buffer_heads in gfs2_sb_in libgfs2: Don't use buffer_heads in gfs2_rgrp_in libgfs2: Remove gfs2_rgrp_out_bh libgfs2: Don't use buffer_heads in gfs2_dinode_in libgfs2: Remove gfs2_dinode_out_bh libgfs2: Don't use buffer_heads in gfs2_leaf_{in,out} libgfs2: Don't use buffer_heads in gfs2_log_header_in libgfs2: Remove gfs2_log_header_out_bh libgfs2: Don't use buffer_heads in gfs2_log_descriptor_{in,out} libgfs2: Don't use buffer_heads in gfs2_quota_change_{in,out} libgfs2: Fix two unused variable warnings mkfs.gfs2: Silence an integer overflow warning libgfs2: Fix a thinko in write_journal() gfs2-utils tests: Add a fsck.gfs2 test for rebuilding journals gfs2_edit: Fix null pointer deref in dump_journal() libgfs2: Remove dead code from gfs2_rgrp_read() fsck.gfs2: Avoid int overflow in find_next_rgrp_dist gfs2_edit: Avoid potential int overflow in find_journal_block() gfs2_edit: Avoid a potential int overflow in dump_journal gfs2-utils: Avoid some more potential in overflows libgfs2: Fix a memory leak in lgfs2_build_jindex glocktop: Fix memory leak
Re: [ClusterLabs] DLM fencing
On Thu, May 24, 2018 at 12:19 AM, Andrei Borzenkovwrote: > 24.05.2018 02:57, Jason Gauthier пишет: >> I'm fairly new to clustering under Linux. I've basically have one shared >> storage resource right now, using dlm, and gfs2. >> I'm using fibre channel and when both of my nodes are up (2 node cluster) >> dlm and gfs2 seem to be operating perfectly. >> If I reboot node B, node A works fine and vice-versa. >> >> When node B goes offline unexpectedly, and become unclean, dlm seems to >> block all IO to the shared storage. >> >> dlm knows node B is down: >> >> # dlm_tool status >> cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 >> daemon now 865695 fence_pid 18186 >> fence 1084772369 nodedown pid 18186 actor 1084772368 fail 1527119246 fence >> 0 now 1527119524 >> node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 >> node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 at 0 0 >> >> on the same server, I see these messages in my daemon.log >> May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: Could not kick >> (reboot) node 1084772369/(null) : No route to host (-113) >> May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error -113 nodeid >> 1084772369 >> >> I can recover from the situation by forcing it (or bring the other node >> back online) >> dlm_tool fence_ack 1084772369 >> >> cluster config is pretty straighforward. >> node 1084772368: alpha >> node 1084772369: beta >> primitive p_dlm_controld ocf:pacemaker:controld \ >> op monitor interval=60 timeout=60 \ >> meta target-role=Started \ >> params args="-K -L -s 1" >> primitive p_fs_gfs2 Filesystem \ >> params device="/dev/sdb2" directory="/vms" fstype=gfs2 >> primitive stonith_sbd stonith:external/sbd \ >> params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ >> meta target-role=Started > > What is the status of stonith resource? Did you configure SBD fencing > properly? I believe so. It's shown above in my cluster config. > Is sbd daemon up and running with proper parameters? Well, no, apparently sbd isn't running.With dlm, and gfs2, the cluster controls handling launching of the daemons. I assumed the same here, since the resource shows that it is up. Online: [ alpha beta ] Full list of resources: stonith_sbd(stonith:external/sbd): Started alpha Clone Set: cl_gfs2 [g_gfs2] Started: [ alpha beta ] > What is output of > sbd -d /dev/sdb1 dump > sbd -d /dev/sdb1 list Both nodes seem fine. 0 alpha testbeta 1 betatestalpha > on both nodes? Does > > sbd -d /dev/sdb1 message test > > work in both directions? It doesn't return an error, yet without a daemon running, I don't think the message is received either. > Does manual fencing using stonith_admin work? I'm not sure at the moment. I think I need to look into why the daemon isn't running. >> group g_gfs2 p_dlm_controld p_fs_gfs2 >> clone cl_gfs2 g_gfs2 \ >> meta interleave=true target-role=Started >> location cli-prefer-cl_gfs2 cl_gfs2 role=Started inf: alpha >> property cib-bootstrap-options: \ >> have-watchdog=false \ >> dc-version=1.1.16-94ff4df \ >> cluster-infrastructure=corosync \ >> cluster-name=zeta \ >> last-lrm-refresh=1525523370 \ >> stonith-enabled=true \ >> stonith-timeout=20s >> >> Any pointers would be appreciated. I feel like this should be working but >> I'm not sure if I've missed something. >> >> Thanks, >> >> Jason >> >> >> >> ___ >> Users mailing list: Users@clusterlabs.org >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DLM fencing
On 05/24/2018 06:19 AM, Andrei Borzenkov wrote: > 24.05.2018 02:57, Jason Gauthier пишет: >> I'm fairly new to clustering under Linux. I've basically have one shared >> storage resource right now, using dlm, and gfs2. >> I'm using fibre channel and when both of my nodes are up (2 node cluster) >> dlm and gfs2 seem to be operating perfectly. >> If I reboot node B, node A works fine and vice-versa. >> >> When node B goes offline unexpectedly, and become unclean, dlm seems to >> block all IO to the shared storage. >> >> dlm knows node B is down: >> >> # dlm_tool status >> cluster nodeid 1084772368 quorate 1 ring seq 32644 32644 >> daemon now 865695 fence_pid 18186 >> fence 1084772369 nodedown pid 18186 actor 1084772368 fail 1527119246 fence >> 0 now 1527119524 >> node 1084772368 M add 861439 rem 0 fail 0 fence 0 at 0 0 >> node 1084772369 X add 865239 rem 865416 fail 865416 fence 0 at 0 0 >> >> on the same server, I see these messages in my daemon.log >> May 23 19:52:47 alpha stonith-api[18186]: stonith_api_kick: Could not kick >> (reboot) node 1084772369/(null) : No route to host (-113) >> May 23 19:52:47 alpha dlm_stonith[18186]: kick_helper error -113 nodeid >> 1084772369 >> >> I can recover from the situation by forcing it (or bring the other node >> back online) >> dlm_tool fence_ack 1084772369 >> >> cluster config is pretty straighforward. >> node 1084772368: alpha >> node 1084772369: beta >> primitive p_dlm_controld ocf:pacemaker:controld \ >> op monitor interval=60 timeout=60 \ >> meta target-role=Started \ >> params args="-K -L -s 1" >> primitive p_fs_gfs2 Filesystem \ >> params device="/dev/sdb2" directory="/vms" fstype=gfs2 >> primitive stonith_sbd stonith:external/sbd \ >> params pcmk_delay_max=30 sbd_device="/dev/sdb1" \ >> meta target-role=Started > What is the status of stonith resource? Did you configure SBD fencing > properly? Is sbd daemon up and running with proper parameters? What is > output of > > sbd -d /dev/sdb1 dump > sbd -d /dev/sdb1 list > > on both nodes? Does > > sbd -d /dev/sdb1 message test > > work in both directions? > > Does manual fencing using stonith_admin work? And checkout that your sbd (1.3.1 to be on the safe side) is new enough otherwise it won't work properly with 2-node enabled in corosync. But this wouldn't describe your problem - would rather be the other way round like still giving you access to the device while it might not be assured that the sbd-fenced node would properly watchdog-suicide in case that it looses access to the storage. Regards, Klaus > >> group g_gfs2 p_dlm_controld p_fs_gfs2 >> clone cl_gfs2 g_gfs2 \ >> meta interleave=true target-role=Started >> location cli-prefer-cl_gfs2 cl_gfs2 role=Started inf: alpha >> property cib-bootstrap-options: \ >> have-watchdog=false \ >> dc-version=1.1.16-94ff4df \ >> cluster-infrastructure=corosync \ >> cluster-name=zeta \ >> last-lrm-refresh=1525523370 \ >> stonith-enabled=true \ >> stonith-timeout=20s >> >> Any pointers would be appreciated. I feel like this should be working but >> I'm not sure if I've missed something. >> >> Thanks, >> >> Jason >> >> >> >> ___ >> Users mailing list: Users@clusterlabs.org >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: pcsd processes using 100% CPU
Tried that, it doesn't seem to do anything but prefix the lines with the pid: [pid 24923] sched_yield() = 0 [pid 24923] sched_yield() = 0 [pid 24923] sched_yield() = 0 Regards, -- Casey > On May 23, 2018, at 11:40 PM, Ulrich Windl >wrote: > Casey & Gina schrieb am 23.05.2018 um 20:43 in > Nachricht <3b8567a0-ef36-44af-bbad-0d494b08f...@icloud.com>: > [...] >> I ran `strace ‑p `, and the screen filled with the following line >> repeating as fast as my terminal can render: >> sched_yield() = 0 >> sched_yield() = 0 >> sched_yield() = 0 > > I wonder whether such a process is multi-threaded and whether adding option > "-f" to strace would make a difference... > > [...] > Regards, > Ulrich > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org