Re: [ClusterLabs] Question about fence_mpath
On Fri, Apr 28, 2017 at 11:25 AM, Chris Adams wrote: > Once upon a time, Seth Reid said: > > I've had a similar problem. Make sure that the /dev/mapper object above > > isn't a symlink. In my multiparth.conf, I had added an alias to a device > > name, like that, to make it easier, but if I pointed my fencing at that, > it > > didn't work. However, I could point directly to the object that the > symlink > > pointed to, and it work. I actually had better luck not even specifying a > > device at all. It found the right one. > > But the multipath devices are always symlinks - how do you access it? > The only "real" devices are /dev/dm-, but those names are not > stable, so you can't use those for anything. This confused me too when I set up my cluster. I found that everything worked better if I didn't specify a device path. I think there was documentation on Redhat that led me to try removing the "device" options. Also thinking back, I'm using fence_scsi, but in a multipath environment. Fence_scsi detected the shared device and everything worked very very smoothly for me. I apologize for not realizing that I had used fence_scsi (and not fence_mpath), but give it a shot removing the "device" options from your stonith command. > ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Question about fence_mpath
So, I'm still not getting fence_mpath working. This is all on CentOS 7.
> Here's what I did:
>
> - on each node, put a unique "reservation_key" in /etc/multipath.conf
> defaults{} section (node1=20170001 and node2=20170002)
>
> - created the STONITH device on node1 with:
> pcs stonith create multipath fence_mpath devices=/dev/mapper/
> 36000d31001309643 key=20170001 meta provides=unfencing
>
> It creates, but any time anything tries to fence (manually or by
> rebooting a node), I get errors in /var/log/messages. Trying to
> manually fence a node gets:
>
> # pcs stonith fence node2 --off
> Error: unable to fence 'node2'
> Command failed: No such device
>
I've had a similar problem. Make sure that the /dev/mapper object above
isn't a symlink. In my multiparth.conf, I had added an alias to a device
name, like that, to make it easier, but if I pointed my fencing at that, it
didn't work. However, I could point directly to the object that the symlink
pointed to, and it work. I actually had better luck not even specifying a
device at all. It found the right one.
>
> Another issue I run into is that fence_mpath tries to access/write to
> /var/run/cluster/mpath.devices, but nothing else creates that directory
> (and it seems that fence_mpath tries to read from it before writing it
> out).
>
>
This is because the device isn't "correct". It can read from a symlink, and
it doesn't follow the symlink, so nothing gets written.
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Question about fence_mpath
>
> The man page talks about it
>
unique to each node (but you only create the STONITH object from one
> node, right?).
The key is unique to each node, but there is only on stonith object. For
fence_mpath, they are putting scsi keys on a shared stonith device.
It also says it has to be set in /etc/multipath.conf
> but then doesn't say how/where.
>
This is part of my multipath.conf that shows the key.
$ cat /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
reservation_key 33c5
}
I am trying to set up a new cluster using fence_mpath. I'm not sure
> what to use for the "key" value though.
I used the fence_scsi program, directly, to figure out the key when I
initially set up my clusters. With what I know now. I could make up a key.
Its a 8 character hex key. When you see it with mpath_persist, it will look
like 0x33c5 (in the example above. It usually looks like that in the
logs, but don't put it like that in the config.
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
We are only using one mount, and that mount has nothing on it currently. I have fixed the problem. Our OS is Ubuntu 16.04 LTS (Xenial). I added the 17.04 (Zesty) repo to get newer a newer version of Corosync. I upgraded Corosync, which upgraded a long list of other related packages (Pacemaker and gfs2 among them). My fencing now works properly. If a node loses network connection to the cluster, only that node is fenced. Presumably, it is a bug in one of the packages in the Xenial repo that has been fixed by in versions in Zesty. --- Seth Reid On Fri, Mar 31, 2017 at 10:31 AM, Bob Peterson wrote: > - Original Message - > | I can confirm that doing an ifdown is not the source of my corosync > issues. > | My cluster is in another state, so I can't pull a cable, but I can down a > | port on a switch. That had the exact same affects as doing an ifdown. Two > | machines got fenced when it should have only been one. > | > | --- > | Seth Reid > > > Hi Seth, > > I don't know if your problem is the same thing I'm looking at BUT: > I'm currently working on a fix to the GFS2 file system for a > similar problem. The scenario is something like this: > > 1. Node X goes down for some reason. > 2. Node X gets fenced by one of the other nodes. > 3. As part of the recovery, GFS2 on all the other nodes have to >replay the journals for all the file systems mounted on X. > 4. GFS2 journal replay hogs the CPU, which causes corosync to be >starved for CPU on some node (say node Y). > 5. Since corosync on node Y was starved for CPU, it doesn't respond >in time to the other nodes (say node Z). > 6. Thus, node Z fences node Y. > > In my case, the solution is to fix GFS2 so that it does some > "cond_resched()" (conditional schedule) statements to allow corosync > (and dlm) to get some work done. Thus, corosync isn't starved for > CPU and does its work, and therefore, it doesn't get fenced. > > I don't know if that's what is happening in your case. > Do you have a lot of GFS2 mount points that would need recovery > when the first fence event occurs? > In my case, I can recreate the problem by having 60 GFS2 mount points. > > Hopefully I'll be sending a GFS2 patch to the cluster-devel > mailing list for this problem soon. > > In testing my fix, I've periodically experienced some weirdness > and other unexplained fencing, so maybe there's a second problem > lurking (or maybe there's just something weird in the experimental > kernel I'm using as a base). Hopefully testing will prove whether > my fix to GFS2 recovery is enough or if there's another problem. > > Regards, > > Bob Peterson > Red Hat File Systems > > ___ > Users mailing list: Users@clusterlabs.org > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
I can confirm that doing an ifdown is not the source of my corosync issues. My cluster is in another state, so I can't pull a cable, but I can down a port on a switch. That had the exact same affects as doing an ifdown. Two machines got fenced when it should have only been one. --- Seth Reid System Operations Engineer Vendini, Inc. 415.349.7736 sr...@vendini.com www.vendini.com On Fri, Mar 31, 2017 at 4:12 AM, Dejan Muhamedagic wrote: > Hi, > > On Fri, Mar 31, 2017 at 02:39:02AM -0400, Digimer wrote: > > On 31/03/17 02:32 AM, Jan Friesse wrote: > > >> The original message has the logs from nodes 1 and 3. Node 2, the one > > >> that > > >> got fenced in this test, doesn't really show much. Here are the logs > from > > >> it: > > >> > > >> Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #5 enp6s0f0, > > >> 192.168.100.14#123, interface stats: received=0, sent=0, dropped=0, > > >> active_time=3253 secs > > >> Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #7 enp6s0f0, > > >> fe80::a236:9fff:fe8a:6500%6#123, interface stats: received=0, sent=0, > > >> dropped=0, active_time=3253 secs > > >> Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] A processor > failed, > > >> forming new configuration. > > >> Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] A processor failed, > > >> forming > > >> new configuration. > > >> Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] The network > > >> interface > > >> is down. > > > > > > This is problem. Corosync handles ifdown really badly. If this was not > > > intentional it may be caused by NetworkManager. Then please install > > > equivalent of NetworkManager-config-server package (it's actually one > > > file called 00-server.conf so you can extract it from, for example, > > > Fedora package > > > https://www.rpmfind.net/linux/RPM/fedora/devel/rawhide/x86_ > 64/n/NetworkManager-config-server-1.8.0-0.1.fc27.noarch.html) > > > > ifdown'ing corosync's interface happens a lot, intentionally or > > otherwise. > > I'm not sure, but I think that it can happen only intentionally, > i.e. through a human intervention. If there's another problem > with the interface it doesn't disappear from the system. > > Thanks, > > Dejan > > > I think it is reasonable to expect corosync to handle this > > properly. How hard would it be to make corosync resilient to this fault > > case? > > > > -- > > Digimer > > Papers and Projects: https://alteeve.com/w/ > > "I am, somehow, less interested in the weight and convolutions of > > Einstein’s brain than in the near certainty that people of equal talent > > have lived and died in cotton fields and sweatshops." - Stephen Jay Gould > > > > ___ > > Users mailing list: Users@clusterlabs.org > > http://lists.clusterlabs.org/mailman/listinfo/users > > > > Project Home: http://www.clusterlabs.org > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > > Bugs: http://bugs.clusterlabs.org > > ___ > Users mailing list: Users@clusterlabs.org > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
The original message has the logs from nodes 1 and 3. Node 2, the one that
got fenced in this test, doesn't really show much. Here are the logs from
it:
Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #5 enp6s0f0,
192.168.100.14#123, interface stats: received=0, sent=0, dropped=0,
active_time=3253 secs
Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #7 enp6s0f0,
fe80::a236:9fff:fe8a:6500%6#123, interface stats: received=0, sent=0,
dropped=0, active_time=3253 secs
Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] A processor failed,
forming new configuration.
Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] A processor failed, forming
new configuration.
Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] The network interface
is down.
Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] adding new UDPU
member {192.168.100.13}
Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] adding new UDPU
member {192.168.100.14}
Mar 24 16:35:13 b014 corosync[2166]: notice [TOTEM ] adding new UDPU
member {192.168.100.15}
Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] The network interface is
down.
Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] adding new UDPU member
{192.168.100.13}
Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] adding new UDPU member
{192.168.100.14}
Mar 24 16:35:13 b014 corosync[2166]: [TOTEM ] adding new UDPU member
{192.168.100.15}
---
Seth Reid
On Wed, Mar 29, 2017 at 7:17 AM, Bob Peterson wrote:
> - Original Message -
> | I will try to install updated packages from ubuntu 16.10 or newer. It
> can't
> | get worse than not working.
> |
> | Can you think of any logs that might help? I've enabled debug on corosync
> | log, but it really doesn't show anything else other than corosync
> exiting.
> | Any diagnostic tools you can recommend?
> |
> | ---
> | Seth Reid
>
>
> Hi Seth,
>
> Can you post the pertinent messages from the consoles of all nodes in the
> cluster? Hopefully you were monitoring them.
>
> Regards,
>
> Bob Peterson
> Red Hat File Systems
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
I will try to install updated packages from ubuntu 16.10 or newer. It can't get worse than not working. Can you think of any logs that might help? I've enabled debug on corosync log, but it really doesn't show anything else other than corosync exiting. Any diagnostic tools you can recommend? --- Seth Reid System Operations Engineer Vendini, Inc. 415.349.7736 sr...@vendini.com www.vendini.com On Mon, Mar 27, 2017 at 3:10 PM, Ken Gaillot wrote: > On 03/27/2017 03:54 PM, Seth Reid wrote: > > > > > > > > On Fri, Mar 24, 2017 at 2:10 PM, Ken Gaillot > <mailto:kgail...@redhat.com>> wrote: > > > > On 03/24/2017 03:52 PM, Digimer wrote: > > > On 24/03/17 04:44 PM, Seth Reid wrote: > > >> I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its > not in > > >> production yet because I'm having a problem during fencing. When I > > >> disable the network interface of any one machine, the disabled > machines > > >> is properly fenced leaving me, briefly, with a two node cluster. A > > >> second node is then fenced off immediately, and the remaining node > > >> appears to try to fence itself off. This leave two nodes with > > >> corosync/pacemaker stopped, and the remaining machine still in the > > >> cluster but showing an offline node and an UNCLEAN node. What can > be > > >> causing this behavior? > > > > > > It looks like the fence attempt failed, leaving the cluster hung. > When > > > you say all nodes were fenced, did all nodes actually reboot? Or > did the > > > two surviving nodes just lock up? If the later, then that is the > proper > > > response to a failed fence (DLM stays blocked). > > > > See comments inline ... > > > > > > > >> Each machine has a dedicated network interface for the cluster, > and > > >> there is a vlan on the switch devoted to just these interfaces. > > >> In the following, I disabled the interface on node id 2 (b014). > > Node 1 > > >> (b013) is fenced as well. Node 2 (b015) is still up. > > >> > > >> Logs from b013: > > >> Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v > debian-sa1 > > > >> /dev/null && debian-sa1 1 1) > > >> Mar 24 16:35:13 b013 corosync[2134]: notice [TOTEM ] A processor > > >> failed, forming new configuration. > > >> Mar 24 16:35:13 b013 corosync[2134]: [TOTEM ] A processor failed, > > >> forming new configuration. > > >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] A new > > membership > > >> (192.168.100.13:576 <http://192.168.100.13:576> > > <http://192.168.100.13:576>) was formed. Members left: 2 > > >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] Failed to > > receive > > >> the leave message. failed: 2 > > >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] A new membership > > >> (192.168.100.13:576 <http://192.168.100.13:576> > > <http://192.168.100.13:576>) was formed. Members left: 2 > > >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] Failed to receive > the > > >> leave message. failed: 2 > > >> Mar 24 16:35:17 b013 attrd[2223]: notice: crm_update_peer_proc: > > Node > > >> b014-cl[2] - state is now lost (was member) > > >> Mar 24 16:35:17 b013 cib[2220]: notice: crm_update_peer_proc: > Node > > >> b014-cl[2] - state is now lost (was member) > > >> Mar 24 16:35:17 b013 cib[2220]: notice: Removing b014-cl/2 from > the > > >> membership list > > >> Mar 24 16:35:17 b013 cib[2220]: notice: Purged 1 peers with id=2 > > >> and/or uname=b014-cl from the membership cache > > >> Mar 24 16:35:17 b013 pacemakerd[2187]: notice: > > crm_reap_unseen_nodes: > > >> Node b014-cl[2] - state is now lost (was member) > > >> Mar 24 16:35:17 b013 attrd[2223]: notice: Removing b014-cl/2 > > from the > > >> membership list > > >> Mar 24 16:35:17 b013 attrd[2223]: notice: Purged 1 peers with > id=2 > > >> and/or uname=b014-cl from the membership cache > > >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: > > crm_update_peer_proc: > > >> Node b014-cl[2] - state is now
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
On Fri, Mar 24, 2017 at 2:10 PM, Ken Gaillot wrote: > On 03/24/2017 03:52 PM, Digimer wrote: > > On 24/03/17 04:44 PM, Seth Reid wrote: > >> I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in > >> production yet because I'm having a problem during fencing. When I > >> disable the network interface of any one machine, the disabled machines > >> is properly fenced leaving me, briefly, with a two node cluster. A > >> second node is then fenced off immediately, and the remaining node > >> appears to try to fence itself off. This leave two nodes with > >> corosync/pacemaker stopped, and the remaining machine still in the > >> cluster but showing an offline node and an UNCLEAN node. What can be > >> causing this behavior? > > > > It looks like the fence attempt failed, leaving the cluster hung. When > > you say all nodes were fenced, did all nodes actually reboot? Or did the > > two surviving nodes just lock up? If the later, then that is the proper > > response to a failed fence (DLM stays blocked). > > See comments inline ... > > > > >> Each machine has a dedicated network interface for the cluster, and > >> there is a vlan on the switch devoted to just these interfaces. > >> In the following, I disabled the interface on node id 2 (b014). Node 1 > >> (b013) is fenced as well. Node 2 (b015) is still up. > >> > >> Logs from b013: > >> Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 > > >> /dev/null && debian-sa1 1 1) > >> Mar 24 16:35:13 b013 corosync[2134]: notice [TOTEM ] A processor > >> failed, forming new configuration. > >> Mar 24 16:35:13 b013 corosync[2134]: [TOTEM ] A processor failed, > >> forming new configuration. > >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] A new membership > >> (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members > left: 2 > >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] Failed to receive > >> the leave message. failed: 2 > >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] A new membership > >> (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members > left: 2 > >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] Failed to receive the > >> leave message. failed: 2 > >> Mar 24 16:35:17 b013 attrd[2223]: notice: crm_update_peer_proc: Node > >> b014-cl[2] - state is now lost (was member) > >> Mar 24 16:35:17 b013 cib[2220]: notice: crm_update_peer_proc: Node > >> b014-cl[2] - state is now lost (was member) > >> Mar 24 16:35:17 b013 cib[2220]: notice: Removing b014-cl/2 from the > >> membership list > >> Mar 24 16:35:17 b013 cib[2220]: notice: Purged 1 peers with id=2 > >> and/or uname=b014-cl from the membership cache > >> Mar 24 16:35:17 b013 pacemakerd[2187]: notice: crm_reap_unseen_nodes: > >> Node b014-cl[2] - state is now lost (was member) > >> Mar 24 16:35:17 b013 attrd[2223]: notice: Removing b014-cl/2 from the > >> membership list > >> Mar 24 16:35:17 b013 attrd[2223]: notice: Purged 1 peers with id=2 > >> and/or uname=b014-cl from the membership cache > >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: crm_update_peer_proc: > >> Node b014-cl[2] - state is now lost (was member) > >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Removing b014-cl/2 from > >> the membership list > >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Purged 1 peers with > >> id=2 and/or uname=b014-cl from the membership cache > >> Mar 24 16:35:17 b013 dlm_controld[2727]: 3091 fence request 2 pid 19223 > >> nodedown time 1490387717 fence_all dlm_stonith > >> Mar 24 16:35:17 b013 kernel: [ 3091.800118] dlm: closing connection to > >> node 2 > >> Mar 24 16:35:17 b013 crmd[2227]: notice: crm_reap_unseen_nodes: Node > >> b014-cl[2] - state is now lost (was member) > >> Mar 24 16:35:17 b013 dlm_stonith: stonith_api_time: Found 0 entries for > >> 2/(null): 0 in progress, 0 completed > >> Mar 24 16:35:18 b013 stonith-ng[2221]: notice: Operation reboot of > >> b014-cl by b015-cl for stonith-api.19223@b013-cl.7aeb2ffb: OK > >> Mar 24 16:35:18 b013 stonith-api[19223]: stonith_api_kick: Node 2/(null) > >> kicked: reboot > > It looks like the fencing of b014-cl is reported as successful above ... > > >> Mar 24 16:35:18 b013 kernel: [ 3092.421495] dlm: closing connection to > >> node 3 > >> Mar 24 16:35:18 b013 kernel: [ 3092.422
Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves
> On 24/03/17 04:44 PM, Seth Reid wrote: > > I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in > > production yet because I'm having a problem during fencing. When I > > disable the network interface of any one machine, the disabled machines > > is properly fenced leaving me, briefly, with a two node cluster. A > > second node is then fenced off immediately, and the remaining node > > appears to try to fence itself off. This leave two nodes with > > corosync/pacemaker stopped, and the remaining machine still in the > > cluster but showing an offline node and an UNCLEAN node. What can be > > causing this behavior? > > It looks like the fence attempt failed, leaving the cluster hung. When > you say all nodes were fenced, did all nodes actually reboot? Or did the > two surviving nodes just lock up? If the later, then that is the proper > response to a failed fence (DLM stays blocked). > The action if "off", so we aren't rebooting. The logs do still say reboot though. In terms of actual fencing, only node 2 gets fenced, in that its keys get removed from the shared volume. Node 1's keys don't get removed so that is the failed fence. Node2 fence succeeds. Of the remaining nodes, node 1 is offline in that corosync and pacemaker are no longer running, so it can't access cluster resources. Node 3 shows node 1 as Online but in a clean state. Neither node 1 or node 3 can write to the cluster, but node 3 still has corosync and pacemaker running. Here are the commands I used to build the cluster. I meant to put these in the original post. (single machine)$> pcs property set no-quorum-policy=freeze (single machine)$> pcs property set stonith-enabled=true (single machine)$> pcs property set symmetric-cluster=true (single machine)$> pcs cluster enable --all (single machine)$> pcs stonith create fence_wh fence_scsi debug="/var/log/cluster/fence-debug.log" vgs_path="/sbin/vgs" sg_persist_path="/usr/bin/sg_persist" sg_turs_path="/usr/bin/sg_turs" pcmk_reboot_action="off" pcmk_host_list="b013-cl b014-cl b015-cl" pcmk_monitor_action="metadata" meta provides="unfencing" --force (single machine)$> pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence clone interleave=true ordered=true (single machine)$> pcs resource create clvmd ocf:heartbeat:clvm op monitor interval=30s on-fail=fence clone interleave=true ordered=true (single machine)$> pcs constraint order start dlm-clone then clvmd-clone (single machine)$> pcs constraint colocation add clvmd-clone with dlm-clone (single machine)$> mkfs.gfs2 -p lock_dlm -t webhosts:share_data -j 3 /dev/mapper/share-data (single machine)$> pcs resource create gfs2share Filesystem device="/dev/mapper/share-data" directory="/share" fstype="gfs2" options="noatime,nodiratime" op monitor interval=10s on-fail=fence clone interleave=true (single machine)$> pcs constraint order start clvmd-clone then gfs2share-clone (single machine)$> pcs constraint colocation add gfs2share-clone with clvmd-clone > > > Each machine has a dedicated network interface for the cluster, and > > there is a vlan on the switch devoted to just these interfaces. > > In the following, I disabled the interface on node id 2 (b014). Node 1 > > (b013) is fenced as well. Node 2 (b015) is still up. > > > > Logs from b013: > > Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 > > > /dev/null && debian-sa1 1 1) > > Mar 24 16:35:13 b013 corosync[2134]: notice [TOTEM ] A processor > > failed, forming new configuration. > > Mar 24 16:35:13 b013 corosync[2134]: [TOTEM ] A processor failed, > > forming new configuration. > > Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] A new membership > > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members > left: 2 > > Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] Failed to receive > > the leave message. failed: 2 > > Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] A new membership > > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members > left: 2 > > Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] Failed to receive the > > leave message. failed: 2 > > Mar 24 16:35:17 b013 attrd[2223]: notice: crm_update_peer_proc: Node > > b014-cl[2] - state is now lost (was member) > > Mar 24 16:35:17 b013 cib[2220]: notice: crm_update_peer_proc: Node > > b014-cl[2] - state is now lost (was member) > > Mar 24 16:35:17 b013 cib[2220]: notice: Removing b014-cl/2 from the > > membership list > > Mar 24 16:35:17 b013 cib[2220]: notice: Pu
[ClusterLabs] Three node cluster becomes completely fenced if one node leaves
: notice: crm_update_peer_proc: Node b013-cl[1] - state is now lost (was member) Mar 24 16:35:26 b015 attrd[2253]: notice: Purged 1 peers with id=1 and/or uname=b013-cl from the membership cache Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Removing b013-cl/1 from the membership list Mar 24 16:35:26 b015 pacemakerd[2159]: notice: Membership 580: quorum lost (1) Mar 24 16:35:26 b015 cib[2249]: notice: crm_update_peer_proc: Node b013-cl[1] - state is now lost (was member) Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Purged 1 peers with id=1 and/or uname=b013-cl from the membership cache Mar 24 16:35:26 b015 pacemakerd[2159]: notice: crm_reap_unseen_nodes: Node b013-cl[1] - state is now lost (was member) Mar 24 16:35:26 b015 cib[2249]: notice: Removing b013-cl/1 from the membership list Mar 24 16:35:26 b015 cib[2249]: notice: Purged 1 peers with id=1 and/or uname=b013-cl from the membership cache Mar 24 16:35:26 b015 crmd[2255]: notice: Membership 580: quorum lost (1) Mar 24 16:35:26 b015 crmd[2255]: notice: crm_reap_unseen_nodes: Node b013-cl[1] - state is now lost (was member) Mar 24 16:35:26 b015 pengine[2254]: notice: We do not have quorum - fencing and resource management disabled Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean because the node is no longer part of the cluster Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Action clvmd:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Action clvmd:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Action gfs2share:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Action gfs2share:1_stop_0 on b013-cl is unrunnable (offline) Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean! Mar 24 16:35:26 b015 pengine[2254]: notice: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Mar 24 16:35:26 b015 pengine[2254]: notice: Start fence_wh#011(b015-cl - blocked) Mar 24 16:35:26 b015 pengine[2254]: notice: Stopdlm:1#011(b013-cl - blocked) Mar 24 16:35:26 b015 pengine[2254]: notice: Stopclvmd:1#011(b013-cl - blocked) Mar 24 16:35:26 b015 pengine[2254]: notice: Stop gfs2share:1#011(b013-cl - blocked) Mar 24 16:35:26 b015 pengine[2254]: warning: Calculated Transition 9: /var/lib/pacemaker/pengine/pe-warn-2669.bz2 Mar 24 16:35:26 b015 crmd[2255]: notice: Transition 9 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-2669.bz2): Complete Mar 24 16:35:26 b015 crmd[2255]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Mar 24 16:35:31 b015 controld(dlm)[2]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing Mar 24 16:35:31 b015 fence_scsi: Failed: keys cannot be same. You can not fence yourself. Mar 24 16:35:31 b015 fence_scsi: Please use '-h' for usage Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=b015-cl' ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ Failed: keys cannot be same. You can not fence yourself. ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ERROR:root:Please use '-h' for usage ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ Please use '-h' for usage ] Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020] stderr: [ ] Software versions: corosync 2.3.5-3ubuntu1 pacemaker-common 1.1.14-2ubuntu1.1 pcs 0.9.149-1ubuntu1 libqb0:amd641.0-1ubuntu1 gfs2-utils3.1.6-0ubuntu3 --- Seth Reid System Operations Engineer Vendini, Inc. 415.349.7736 sr...@vendini.com www.vendini.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
