Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves

2017-03-27 Thread Ken Gaillot 
On 03/27/2017 03:54 PM, Seth Reid wrote:
> 
> 
> 
> On Fri, Mar 24, 2017 at 2:10 PM, Ken Gaillot  > wrote:
> 
> On 03/24/2017 03:52 PM, Digimer wrote:
> > On 24/03/17 04:44 PM, Seth Reid wrote:
> >> I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in
> >> production yet because I'm having a problem during fencing. When I
> >> disable the network interface of any one machine, the disabled machines
> >> is properly fenced leaving me, briefly, with a two node cluster. A
> >> second node is then fenced off immediately, and the remaining node
> >> appears to try to fence itself off. This leave two nodes with
> >> corosync/pacemaker stopped, and the remaining machine still in the
> >> cluster but showing an offline node and an UNCLEAN node. What can be
> >> causing this behavior?
> >
> > It looks like the fence attempt failed, leaving the cluster hung. When
> > you say all nodes were fenced, did all nodes actually reboot? Or did the
> > two surviving nodes just lock up? If the later, then that is the proper
> > response to a failed fence (DLM stays blocked).
> 
> See comments inline ...
> 
> >
> >> Each machine has a dedicated network interface for the cluster, and
> >> there is a vlan on the switch devoted to just these interfaces.
> >> In the following, I disabled the interface on node id 2 (b014).
> Node 1
> >> (b013) is fenced as well. Node 2 (b015) is still up.
> >>
> >> Logs from b013:
> >> Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 >
> >> /dev/null && debian-sa1 1 1)
> >> Mar 24 16:35:13 b013 corosync[2134]: notice  [TOTEM ] A processor
> >> failed, forming new configuration.
> >> Mar 24 16:35:13 b013 corosync[2134]:  [TOTEM ] A processor failed,
> >> forming new configuration.
> >> Mar 24 16:35:17 b013 corosync[2134]: notice  [TOTEM ] A new
> membership
> >> (192.168.100.13:576 
> ) was formed. Members left: 2
> >> Mar 24 16:35:17 b013 corosync[2134]: notice  [TOTEM ] Failed to
> receive
> >> the leave message. failed: 2
> >> Mar 24 16:35:17 b013 corosync[2134]:  [TOTEM ] A new membership
> >> (192.168.100.13:576 
> ) was formed. Members left: 2
> >> Mar 24 16:35:17 b013 corosync[2134]:  [TOTEM ] Failed to receive the
> >> leave message. failed: 2
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: crm_update_peer_proc:
> Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: crm_update_peer_proc: Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: Removing b014-cl/2 from the
> >> membership list
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: Purged 1 peers with id=2
> >> and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 pacemakerd[2187]:   notice:
> crm_reap_unseen_nodes:
> >> Node b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: Removing b014-cl/2
> from the
> >> membership list
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: Purged 1 peers with id=2
> >> and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice:
> crm_update_peer_proc:
> >> Node b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice: Removing
> b014-cl/2 from
> >> the membership list
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice: Purged 1 peers with
> >> id=2 and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 dlm_controld[2727]: 3091 fence request 2 pid
> 19223
> >> nodedown time 1490387717 fence_all dlm_stonith
> >> Mar 24 16:35:17 b013 kernel: [ 3091.800118] dlm: closing
> connection to
> >> node 2
> >> Mar 24 16:35:17 b013 crmd[2227]:   notice: crm_reap_unseen_nodes:
> Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 dlm_stonith: stonith_api_time: Found 0
> entries for
> >> 2/(null): 0 in progress, 0 completed
> >> Mar 24 16:35:18 b013 stonith-ng[2221]:   notice: Operation reboot of
> >> b014-cl by b015-cl for stonith-api.19223@b013-cl.7aeb2ffb: OK
> >> Mar 24 16:35:18 b013 stonith-api[19223]: stonith_api_kick: Node
> 2/(null)
> >> kicked: reboot
> 
> It looks like the fencing of b014-cl is reported as successful above ...
> 
> >> Mar 24 16:35:18 b013 kernel: [ 3092.421495] dlm: closing connection to
> >> node 3
> >> Mar 24 16:35:18 b013 kernel: [ 3092.422246] dlm: closing connection to
> >> node 1
> >> Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace 
>

Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves

2017-03-27 Thread Seth Reid 
On Fri, Mar 24, 2017 at 2:10 PM, Ken Gaillot  wrote:

> On 03/24/2017 03:52 PM, Digimer wrote:
> > On 24/03/17 04:44 PM, Seth Reid wrote:
> >> I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in
> >> production yet because I'm having a problem during fencing. When I
> >> disable the network interface of any one machine, the disabled machines
> >> is properly fenced leaving me, briefly, with a two node cluster. A
> >> second node is then fenced off immediately, and the remaining node
> >> appears to try to fence itself off. This leave two nodes with
> >> corosync/pacemaker stopped, and the remaining machine still in the
> >> cluster but showing an offline node and an UNCLEAN node. What can be
> >> causing this behavior?
> >
> > It looks like the fence attempt failed, leaving the cluster hung. When
> > you say all nodes were fenced, did all nodes actually reboot? Or did the
> > two surviving nodes just lock up? If the later, then that is the proper
> > response to a failed fence (DLM stays blocked).
>
> See comments inline ...
>
> >
> >> Each machine has a dedicated network interface for the cluster, and
> >> there is a vlan on the switch devoted to just these interfaces.
> >> In the following, I disabled the interface on node id 2 (b014). Node 1
> >> (b013) is fenced as well. Node 2 (b015) is still up.
> >>
> >> Logs from b013:
> >> Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 >
> >> /dev/null && debian-sa1 1 1)
> >> Mar 24 16:35:13 b013 corosync[2134]: notice  [TOTEM ] A processor
> >> failed, forming new configuration.
> >> Mar 24 16:35:13 b013 corosync[2134]:  [TOTEM ] A processor failed,
> >> forming new configuration.
> >> Mar 24 16:35:17 b013 corosync[2134]: notice  [TOTEM ] A new membership
> >> (192.168.100.13:576 ) was formed. Members
> left: 2
> >> Mar 24 16:35:17 b013 corosync[2134]: notice  [TOTEM ] Failed to receive
> >> the leave message. failed: 2
> >> Mar 24 16:35:17 b013 corosync[2134]:  [TOTEM ] A new membership
> >> (192.168.100.13:576 ) was formed. Members
> left: 2
> >> Mar 24 16:35:17 b013 corosync[2134]:  [TOTEM ] Failed to receive the
> >> leave message. failed: 2
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: crm_update_peer_proc: Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: crm_update_peer_proc: Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: Removing b014-cl/2 from the
> >> membership list
> >> Mar 24 16:35:17 b013 cib[2220]:   notice: Purged 1 peers with id=2
> >> and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 pacemakerd[2187]:   notice: crm_reap_unseen_nodes:
> >> Node b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: Removing b014-cl/2 from the
> >> membership list
> >> Mar 24 16:35:17 b013 attrd[2223]:   notice: Purged 1 peers with id=2
> >> and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice: crm_update_peer_proc:
> >> Node b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice: Removing b014-cl/2 from
> >> the membership list
> >> Mar 24 16:35:17 b013 stonith-ng[2221]:   notice: Purged 1 peers with
> >> id=2 and/or uname=b014-cl from the membership cache
> >> Mar 24 16:35:17 b013 dlm_controld[2727]: 3091 fence request 2 pid 19223
> >> nodedown time 1490387717 fence_all dlm_stonith
> >> Mar 24 16:35:17 b013 kernel: [ 3091.800118] dlm: closing connection to
> >> node 2
> >> Mar 24 16:35:17 b013 crmd[2227]:   notice: crm_reap_unseen_nodes: Node
> >> b014-cl[2] - state is now lost (was member)
> >> Mar 24 16:35:17 b013 dlm_stonith: stonith_api_time: Found 0 entries for
> >> 2/(null): 0 in progress, 0 completed
> >> Mar 24 16:35:18 b013 stonith-ng[2221]:   notice: Operation reboot of
> >> b014-cl by b015-cl for stonith-api.19223@b013-cl.7aeb2ffb: OK
> >> Mar 24 16:35:18 b013 stonith-api[19223]: stonith_api_kick: Node 2/(null)
> >> kicked: reboot
>
> It looks like the fencing of b014-cl is reported as successful above ...
>
> >> Mar 24 16:35:18 b013 kernel: [ 3092.421495] dlm: closing connection to
> >> node 3
> >> Mar 24 16:35:18 b013 kernel: [ 3092.422246] dlm: closing connection to
> >> node 1
> >> Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace
> share_data
> >> Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace clvmd
> >> Mar 24 16:35:18 b013 kernel: [ 3092.426545] dlm: dlm user daemon left 2
> >> lockspaces
> >> Mar 24 16:35:18 b013 systemd[1]: corosync.service: Main process exited,
> >> code=exited, status=255/n/a
>
> ... but then DLM and corosync exit on this node. Pacemaker can only
> exit, and the node gets fenced.
>
> What does your fencing configuration look like?
>

This is the command I used. b013-cl, for example is a hosts file entry so
that the cluster

Re: [ClusterLabs] stonith in dual HMC environment

2017-03-27 Thread Alexander Markov 

Hello, Dejan,



The first thing I'd try is making sure you can fence each node from the
command line by manually running the fence agent. I'm not sure how to 
do

that for the "stonith:" type agents.

There's a program stonith(8). It's easy to replicate the
configuration on the command line.


Unfortunately, it is not.

The landscape I refer to is similar to VMWare. We use cluster for 
virtual machines (LPARs) and everything works OK but the real pain 
occurs when whole host system is down. Keeping in mind that it's 
actually used now in production, I just can't afford to turn it off for 
test reason.




Stonith agents are to be queried for the list of nodes they can
manage. It's part of the interface. Some agents can figure that
out by themself and some need a parameter defining the node list.


And this is just the place I'm stuck. I've got two stonith devices 
(ibmhmc) for redundancy. Both of them are capable to manage every node. 
The problem starts when


1) one stonith device is completely lost and inaccessible (due to power 
outage in datacenter)
2) survived stonith device cannot access nor cluster node neither 
hosting system (in VMWare terms) for this cluster node, for both of them 
are also lost due to power outage.


What is the correct solution for this situation?


Well, this used to be a standard way to configure one kind of
stonith resources, one common representative being ipmi, and
served exactly the purpose of restricting the stonith resource
from being enabled ("running") on a node which this resource
manages.


Unfortunately, there's no such thing as ipmi in IBM Power boxes. But it 
triggers interesting question for me: if both one node and its 
complementary ipmi device are lost (due to power outage) - what's 
happening with a cluster? Survived node, running stonith resource for 
dead node tries to contact ipmi device (which is also dead). How does 
cluster understand that lost node is really dead and it's not just a 
network issue?


Thank you.

--
Regards,
Alexander Markov
+79104531955

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org