Re: [ClusterLabs] Unable to add 'NodeX' to cluster: node is already in a cluster
Tomas,
Yes, that was it.
[root@zs95KL corosync]# pcs cluster destroy
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
[root@zs95KL corosync]#
[root@zs93kl corosync]# pcs cluster node add zs95KLpcs1,zs95KLpcs2
zs95kjpcs1: Corosync updated
zs93KLpcs1: Corosync updated
zs95KLpcs1: Succeeded
Synchronizing pcsd certificates on nodes zs95KLpcs1...
zs95KLpcs1: Success
Restaring pcsd on the nodes in order to reload the certificates...
zs95KLpcs1: Success
[root@zs93kl corosync]#
Thank you very much for this quick fix.
- Scott
Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie,
N.Y.
INTERNET: swgre...@us.ibm.com
From: Tomas Jelinek
To: users@clusterlabs.org
Date: 06/29/2017 12:13 PM
Subject:Re: [ClusterLabs] Unable to add 'NodeX' to cluster: node is
already in a cluster
Hi Scott,
It looks like some of cluster configuration files still exist on your
node 'zs95KLpcs1'. Try running "pcs cluster destroy" on that node. This
will delete all cluster config files on the node. So make sure it is the
right node before running the command. Then you should be able to add
the node to your cluster.
Regards,
Tomas
Dne 29.6.2017 v 17:32 Scott Greenlese napsal(a):
> Hi all...
>
> When I try to add a previously removed cluster node back into my
> pacemaker cluster, I get the following error:
>
> [root@zs93kl]# pcs cluster node add zs95KLpcs1,zs95KLpcs2
> Error: Unable to add 'zs95KLpcs1' to cluster: node is already in a
cluster
>
> The node I am adding was recently removed from the cluster, but
> apparently the removal
> was incomplete.
>
> I am looking for some help to thoroughly remove zs95KLpcs1 from this (or
> any other)
> cluster that this host may be a part of.
>
>
> Background:
>
> I had removed node ( zs95KLpcs1) from my 3 node, single ring protocol
> pacemaker cluster while that node
> (which happens to be a KVM on System Z Linux host), was deactivated /
> shut down due to
> relentless, unsolicited STONITH events. My thought was that there was
> some issue with the ring0
> interface (on vlan1293) causing the cluster to initiate fence (power
> off) actions, just minutes after
> joining the cluster. That's why I went ahead and deactivated that node.
>
> The first procedure I used to remove zs95KLpcs1 was flawed, because I
> forgot that there's an issue with
> attempting to remove an unreachable cluster node on the older pacemaker
> code:
>
> [root@zs95kj ]# date;pcs cluster node remove zs95KLpcs1
> Tue Jun 27 18:28:23 EDT 2017
> Error: pcsd is not running on zs95KLpcs1
>
> I then followed this procedure (courtesy of Tomasand Ken inthis user
> group):
>
> 1. run 'pcs cluster localnode remove ' on all remaining nodes
> 2. run 'pcs cluster reload corosync' on one node
> 3. run 'crm_node -R --force' on one node
>
> My execution:
>
> I made the mistake of manually removing the target node (zs95KLpcs1)
> stanza from corosync.conf file before
> executing the above procedure:
>
> [root@zs95kj ]# vi /etc/corosync/corosync.conf
>
> Removed this stanza:
>
> node {
> ring0_addr: zs95KLpcs1
> nodeid: 3
> }
>
> I then followed the recommended steps ...
>
> [root@zs95kj ]# pcs cluster localnode remove zs95KLpcs1
> Error: unable to remove zs95KLpcs1 ### I assume this was because I
> manually removed the stanza (above)
>
> [root@zs93kl ]# pcs cluster localnode remove zs95KLpcs1
> zs95KLpcs1: successfully removed!
> [root@zs93kl ]#
>
> [root@zs95kj ]# pcs cluster reload corosync
> Corosync reloaded
> [root@zs95kj ]#
>
> [root@zs95kj ]# crm_node -R zs95KLpcs1 --force
> [root@zs95kj ]#
>
>
> [root@zs95kj ]# pcs status |less
> Cluster name: test_cluster_2
> Last updated: Tue Jun 27 18:39:14 2017 Last change: Tue Jun 27 18:38:56
> 2017 by root via crm_node on zs95kjpcs1
> Stack: corosync
> Current DC: zs93KLpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) -
> partition with quorum
> 45 nodes and 227 resources configured
>
> *Online: [ zs93KLpcs1 zs95kjpcs1 ]*
>
>
> This seemed to work well, at least I'm showing only the two cluster
nodes.
>
> Later on, once I was able to activate zs95KLpcs1 (former cluster member)
> ... I did what I thought
> I should do to tell that node that it's no longer a member of the
cluster:
>
> [root@zs95kj ]# cat neuter.sh
> ssh root@zs95KL "/usr/sbin/pcs cluster localnode *remove *zs95KLpcs1"
> ssh root@zs95KL "/usr/sbin/pcs cluster reload corosync"
> ssh root@zs
[ClusterLabs] Unable to add 'NodeX' to cluster: node is already in a cluster
Hi all...
When I try to add a previously removed cluster node back into my pacemaker
cluster, I get the following error:
[root@zs93kl]# pcs cluster node add zs95KLpcs1,zs95KLpcs2
Error: Unable to add 'zs95KLpcs1' to cluster: node is already in a cluster
The node I am adding was recently removed from the cluster, but apparently
the removal
was incomplete.
I am looking for some help to thoroughly remove zs95KLpcs1 from this (or
any other)
cluster that this host may be a part of.
Background:
I had removed node ( zs95KLpcs1) from my 3 node, single ring protocol
pacemaker cluster while that node
(which happens to be a KVM on System Z Linux host), was deactivated / shut
down due to
relentless, unsolicited STONITH events. My thought was that there was some
issue with the ring0
interface (on vlan1293) causing the cluster to initiate fence (power off)
actions, just minutes after
joining the cluster. That's why I went ahead and deactivated that node.
The first procedure I used to remove zs95KLpcs1 was flawed, because I
forgot that there's an issue with
attempting to remove an unreachable cluster node on the older pacemaker
code:
[root@zs95kj ]# date;pcs cluster node remove zs95KLpcs1
Tue Jun 27 18:28:23 EDT 2017
Error: pcsd is not running on zs95KLpcs1
I then followed this procedure (courtesy of Tomas and Ken in this user
group):
1. run 'pcs cluster localnode remove ' on all remaining nodes
2. run 'pcs cluster reload corosync' on one node
3. run 'crm_node -R --force' on one node
My execution:
I made the mistake of manually removing the target node (zs95KLpcs1) stanza
from corosync.conf file before
executing the above procedure:
[root@zs95kj ]# vi /etc/corosync/corosync.conf
Removed this stanza:
node {
ring0_addr: zs95KLpcs1
nodeid: 3
}
I then followed the recommended steps ...
[root@zs95kj ]# pcs cluster localnode remove zs95KLpcs1
Error: unable to remove zs95KLpcs1### I assume this was because I
manually removed the stanza (above)
[root@zs93kl ]# pcs cluster localnode remove zs95KLpcs1
zs95KLpcs1: successfully removed!
[root@zs93kl ]#
[root@zs95kj ]# pcs cluster reload corosync
Corosync reloaded
[root@zs95kj ]#
[root@zs95kj ]# crm_node -R zs95KLpcs1 --force
[root@zs95kj ]#
[root@zs95kj ]# pcs status |less
Cluster name: test_cluster_2
Last updated: Tue Jun 27 18:39:14 2017 Last change: Tue Jun 27
18:38:56 2017 by root via crm_node on zs95kjpcs1
Stack: corosync
Current DC: zs93KLpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition
with quorum
45 nodes and 227 resources configured
Online: [ zs93KLpcs1 zs95kjpcs1 ]
This seemed to work well, at least I'm showing only the two cluster nodes.
Later on, once I was able to activate zs95KLpcs1 (former cluster
member) ... I did what I thought
I should do to tell that node that it's no longer a member of the cluster:
[root@zs95kj ]# cat neuter.sh
ssh root@zs95KL "/usr/sbin/pcs cluster localnode remove zs95KLpcs1"
ssh root@zs95KL "/usr/sbin/pcs cluster reload corosync"
ssh root@zs95KL "/usr/sbin/crm_node -R zs95KLpcs1 --force"
[root@zs95kj ]# ./neuter.sh
zs95KLpcs1: successfully removed!
Corosync reloaded
[root@zs95kj ]#
Next, I followed a procedure to convert my current 2-node, single ring
cluster to RRP ... which seems to be running
well, and the corosync config looks like this:
[root@zs93kl ]# for host in zs95kjpcs1 zs93KLpcs1 ; do ssh $host
"hostname;corosync-cfgtool -s"; done
zs95kj
Printing ring status.
Local node ID 2
RING ID 0
id = 10.20.93.12
status = ring 0 active with no faults
RING ID 1
id = 10.20.94.212
status = ring 1 active with no faults
zs93kl
Printing ring status.
Local node ID 5
RING ID 0
id = 10.20.93.13
status = ring 0 active with no faults
RING ID 1
id = 10.20.94.213
status = ring 1 active with no faults
[root@zs93kl ]#
So now, when I try to add zs95KLpcs1 (and the second ring interface,
zs95KLpcs2) to the RRP config,
I get the error:
[root@zs93kl]# pcs cluster node add zs95KLpcs1,zs95KLpcs2
Error: Unable to add 'zs95KLpcs1' to cluster: node is already in a cluster
I re-ran the node removal procedures, and also
deleted /etc/corosync/corosync.conf
on the target node zs95KLpcs1, and nothing I've tried resolves my problem.
I checked to see if zs95KLpcs1 exists in any "corosync.conf" file on the 3
nodes, and it does not.
[root@zs95kj corosync]# grep zs95KLpcs1 *
[root@zs95kj corosync]#
[root@zs93kl corosync]# grep zs95KLpcs1 *
[root@zs95kj corosync]#
[root@zs95KL corosync]# grep zs95KLpcs1 *
[root@zs95kj corosync]#
Thanks in advance ..
Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie,
N.Y.
INTERNET: swgre...@us.ibm.com
___
Users mailing list: Users@clusterlabs.org
http://lists.clu
Re: [ClusterLabs] How to force remove a cluster node?
Tomas,
Yes, I have an IBM internal build we're using for KVM on System Z.I
tried the --force option and, while it didn't complain,
it didn't work either (as expected, as per bug
https://bugzilla.redhat.com/show_bug.cgi?id=1225423), so
maybe it is a valid option.
[root@zs95kj VD]# date; pcs cluster node remove zs95KLpcs1 --force
Wed Apr 19 10:14:10 EDT 2017
Error: pcsd is not running on zs95KLpcs1
[root@zs95kj VD]#
Hopefully we'll roll in pcs-0.9.143-15.el7_2.1 with our next release of
KVM.
In the mean time, thanks very much for all the valuable feedback. I'm good
to go for now with the workaround.
Scott
Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie,
N.Y.
INTERNET: swgre...@us.ibm.com
From: Tomas Jelinek
To: users@clusterlabs.org
Date: 04/19/2017 03:25 AM
Subject:Re: [ClusterLabs] How to force remove a cluster node?
Dne 18.4.2017 v 19:52 Scott Greenlese napsal(a):
> My thanks to both Ken Gaillot and Tomas Jelinek for the workaround. The
> procedure(s) worked like a champ.
>
> I just have a few side comments / observations ...
>
> First - Tomas, in the bugzilla you show this error message on your
> cluster remove command, directing you to use the --force option:
>
> [root@rh72-node1:~]# pcs cluster node remove rh72-node3
> Error: pcsd is not running on rh72-node3, use --force to override
>
> When I issue the cluster remove, I do not get and reference to the
> --force option in the error message:
>
> [root@zs93kl ]# pcs cluster node remove zs95KLpcs1
> Error: pcsd is not running on zs95KLpcs1
> [root@zs93kl ]#
>
> The man page doesn't mention --force at my level.
The man page doesn't mention --force for most commands in which --force
can be used. One shouldn't really make any conclusions from that.
> Is this a feature added after pcs-0.9.143-15.el7_2.ibm.2.s390x ?
The feature has been backported to pcs-0.9.143-15.el7_2.1. I cannot
really check if it is present in pcs-0.9.143-15.el7_2.ibm.2.s390x
because I don't have access to that particular build. Based on the name
I would say it was build internally at IBM. However, if the error
message doesn't suggest using --force, than the feature is most likely
not present in that build.
>
> Also, in your workaround procedure, you have me do: 'pcs cluster
> *localnode*remove '.
> However, wondering why the 'localnode' option is not in the pcs man page
> for the pcs cluster command?
> The command / option worked great, just curious why it's not
documented ...
It's an internal pcs command which is not meant to be run by users. It
exists mostly for the sake of the current pcs/pcsd architecture ("pcs
cluster node" calls pcsd instance on all nodes over network and pcsd
instance runs "pcs cluster localnode" to do the actual job) and is
likely to be removed in the future. It is useful for the workaround as
the check whether all nodes are running is done in the "pcs cluster
node" command.
Regards,
Tomas
>
> [root@zs93kl # pcs cluster localnode remove zs93kjpcs1
> zs93kjpcs1: successfully removed!
>
> My man page level:
>
> [root@zs93kl VD]# rpm -q --whatprovides /usr/share/man/man8/pcs.8.gz
> pcs-0.9.143-15.el7_2.ibm.2.s390x
> [root@zs93kl VD]#
>
> Thanks again,
>
> Scott G.
>
> Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie,
N.Y.
> INTERNET: swgre...@us.ibm.com
>
>
> Inactive hide details for Tomas Jelinek ---04/18/2017 09:04:59 AM---Dne
> 17.4.2017 v 17:28 Ken Gaillot napsal(a): > On 04/13/201Tomas Jelinek
> ---04/18/2017 09:04:59 AM---Dne 17.4.2017 v 17:28 Ken Gaillot napsal(a):
>> On 04/13/2017 01:11 PM, Scott Greenlese wrote:
>
> From: Tomas Jelinek
> To: users@clusterlabs.org
> Date: 04/18/2017 09:04 AM
> Subject: Re: [ClusterLabs] How to force remove a cluster node?
>
>
>
>
>
> Dne 17.4.2017 v 17:28 Ken Gaillot napsal(a):
>> On 04/13/2017 01:11 PM, Scott Greenlese wrote:
>>> Hi,
>>>
>>> I need to remove some nodes from my existing pacemaker cluster which
are
>>> currently unbootable / unreachable.
>>>
>>> Referenced
>>>
>
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-clusternodemanage-HAAR.html#s2-noderemove-HAAR
>>>
>>> *4.4.4. Removing Cluster Nodes*
>>> The following command shuts down the specified node and removes it from
>>> the cluster configuration file, corosync.conf, on all of the other
nodes
>>> in the cluster. For information on removing all information about the
>>> cluster from the cluster nodes entirely,
Re: [ClusterLabs] How to force remove a cluster node?
My thanks to both Ken Gaillot and Tomas Jelinek for the workaround. The procedure(s) worked like a champ. I just have a few side comments / observations ... First - Tomas, in the bugzilla you show this error message on your cluster remove command, directing you to use the --force option: [root@rh72-node1:~]# pcs cluster node remove rh72-node3 Error: pcsd is not running on rh72-node3, use --force to override When I issue the cluster remove, I do not get and reference to the --force option in the error message: [root@zs93kl ]# pcs cluster node remove zs95KLpcs1 Error: pcsd is not running on zs95KLpcs1 [root@zs93kl ]# The man page doesn't mention --force at my level. Is this a feature added after pcs-0.9.143-15.el7_2.ibm.2.s390x ? Also, in your workaround procedure, you have me do: 'pcs cluster localnode remove '. However, wondering why the 'localnode' option is not in the pcs man page for the pcs cluster command? The command / option worked great, just curious why it's not documented ... [root@zs93kl # pcs cluster localnode remove zs93kjpcs1 zs93kjpcs1: successfully removed! My man page level: [root@zs93kl VD]# rpm -q --whatprovides /usr/share/man/man8/pcs.8.gz pcs-0.9.143-15.el7_2.ibm.2.s390x [root@zs93kl VD]# Thanks again, Scott G. Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Tomas Jelinek To: users@clusterlabs.org Date: 04/18/2017 09:04 AM Subject:Re: [ClusterLabs] How to force remove a cluster node? Dne 17.4.2017 v 17:28 Ken Gaillot napsal(a): > On 04/13/2017 01:11 PM, Scott Greenlese wrote: >> Hi, >> >> I need to remove some nodes from my existing pacemaker cluster which are >> currently unbootable / unreachable. >> >> Referenced >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-clusternodemanage-HAAR.html#s2-noderemove-HAAR >> >> *4.4.4. Removing Cluster Nodes* >> The following command shuts down the specified node and removes it from >> the cluster configuration file, corosync.conf, on all of the other nodes >> in the cluster. For information on removing all information about the >> cluster from the cluster nodes entirely, thereby destroying the cluster >> permanently, refer to _Section 4.6, “Removing the Cluster >> Configuration”_ >> < https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-clusterremove-HAAR.html#s2-noderemove-HAAR >. >> >> pcs cluster node remove /node/ >> >> I ran the command with the cluster active on 3 of the 5 available >> cluster nodes (with quorum). The command fails with: >> >> [root@zs90KP VD]# date;*pcs cluster node remove zs93kjpcs1* >> Thu Apr 13 13:40:59 EDT 2017 >> *Error: pcsd is not running on zs93kjpcs1* >> >> >> The node was not removed: >> >> [root@zs90KP VD]# pcs status |less >> Cluster name: test_cluster_2 >> Last updated: Thu Apr 13 14:08:15 2017 Last change: Wed Apr 12 16:40:26 >> 2017 by root via cibadmin on zs93KLpcs1 >> Stack: corosync >> Current DC: zs90kppcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - >> partition with quorum >> 45 nodes and 180 resources configured >> >> Node zs95KLpcs1: UNCLEAN (offline) >> Online: [ zs90kppcs1 zs93KLpcs1 zs95kjpcs1 ] >> *OFFLINE: [ zs93kjpcs1 ]* >> >> >> Is there a way to force remove a node that's no longer bootable? If not, >> what's the procedure for removing a rogue cluster node? >> >> Thank you... >> >> Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. >> INTERNET: swgre...@us.ibm.com > > Yes, the pcs command is just a convenient shorthand for a series of > commands. You want to ensure pacemaker and corosync are stopped on the > node to be removed (in the general case, obviously already done in this > case), remove the node from corosync.conf and restart corosync on all > other nodes, then run "crm_node -R " on any one active node. Hi Scott, It is possible to remove an offline node from a cluster with upstream pcs 0.9.154 or RHEL pcs-0.9.152-5 (available in RHEL7.3) or newer. If you have an older version, here's a workaround: 1. run 'pcs cluster localnode remove ' on all remaining nodes 2. run 'pcs cluster reload corosync' on one node 3. run 'crm_node -R --force' on one node It's basically the same procedure Ken described. See https://bugzilla.redhat.com/show_bug.cgi?id=1225423 for more details. Regards, Tomas ___ Users mailing list: Users@clusterlabs.org http://lists.clust
[ClusterLabs] How to force remove a cluster node?
Hi, I need to remove some nodes from my existing pacemaker cluster which are currently unbootable / unreachable. Referenced https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-clusternodemanage-HAAR.html#s2-noderemove-HAAR 4.4.4. Removing Cluster Nodes The following command shuts down the specified node and removes it from the cluster configuration file, corosync.conf, on all of the other nodes in the cluster. For information on removing all information about the cluster from the cluster nodes entirely, thereby destroying the cluster permanently, refer to Section 4.6, “Removing the Cluster Configuration”. pcs cluster node remove node I ran the command with the cluster active on 3 of the 5 available cluster nodes (with quorum). The command fails with: [root@zs90KP VD]# date;pcs cluster node remove zs93kjpcs1 Thu Apr 13 13:40:59 EDT 2017 Error: pcsd is not running on zs93kjpcs1 The node was not removed: [root@zs90KP VD]# pcs status |less Cluster name: test_cluster_2 Last updated: Thu Apr 13 14:08:15 2017 Last change: Wed Apr 12 16:40:26 2017 by root via cibadmin on zs93KLpcs1 Stack: corosync Current DC: zs90kppcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition with quorum 45 nodes and 180 resources configured Node zs95KLpcs1: UNCLEAN (offline) Online: [ zs90kppcs1 zs93KLpcs1 zs95kjpcs1 ] OFFLINE: [ zs93kjpcs1 ] Is there a way to force remove a node that's no longer bootable?If not, what's the procedure for removing a rogue cluster node? Thank you... Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: Antw: Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config?
Ulrich, Can you share with me some details how your pacemaker, corosync and LGM is configured? I should consider implementing a similar approach and this would be very helpful to me. Thanks... Scott Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301) From: "Ulrich Windl" To: Date: 03/08/2017 03:20 AM Subject:[ClusterLabs] Antw: Re: Antw: Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config? >>> "Scott Greenlese" schrieb am 07.03.2017 um 17:28 in Nachricht : [...] > > Maybe my question is... is there any way to facilitate an alternate Live > Guest Migration path in the event of a ring0_addr failure? > This might also apply to a single ring protocol as well. [...] I don't know the answer, but what we have here is that every network is connected using two NICs in a bonding interface, and we have two of these networks for cluster communication, one being exclusively for cluster communication. And the network being used for migration is separate from the two networks used for cluster communication. As said, we are USING SLES11 here, but we never had problems that you have. One problem we had was that massive parallel VM migrations would overload the network, so we limited the number of parallel migrations. Maybe this is still helpful... Regards, Ulrich ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config?
Ulrich, Thank you very much for your feedback. You wrote, "Could it be you forgot "allow-migrate=true" at the resource level or some migration IP address at the node level? I only have SLES11 here..." I know for sure that the pacemaker remote node (zs95kjg110102) I mentioned below is configured correctly for pacemaker Live Guest Migration. I can demonstrate this using the 'pcs resource move' CLI : I will migrate this "remote node" guest (zs95kjg110102) and resource "zs95kjg110102_res" to another cluster node (e.g. zs95kjpcs1 / 10.20.93.12) , using the 'pcs1' hostname / IP which is currently running on zs93kjpcs1 (10.20.93.11): [root@zs95kj ~]# pcs resource show |grep zs95kjg110102_res zs95kjg110102_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1 [root@zs93kj ~]# pcs resource move zs95kjg110102_res zs95kjpcs1 [root@zs93kj ~]# pcs resource show |grep zs95kjg110102_res zs95kjg110102_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1 ## On zs95kjpcs1, you can see that the guest is actually running there... [root@zs95kj ~]# virsh list |grep zs95kjg110102 63zs95kjg110102 running [root@zs95kj ~]# ping 10.20.110.102 PING 10.20.110.102 (10.20.110.102) 56(84) bytes of data. 64 bytes from 10.20.110.102: icmp_seq=1 ttl=63 time=0.775 ms So, everything seems set up correctly for live guest migration of this VirtualDomain resource. What I am really looking for is a way to ensure 100% availability of a "live guest migratable" pacemaker remote node guest in a situation where the interface (in this case vlan1293) ring0_addr goes down. I thought that maybe configuring Redundant Ring Protocol (RRP) for corosync would provide this, but from what I've seen so far it doesn't look that way.If the ring0_addr interface is lost in an RRP configuration while the remote guest is connected to the host using that ring0_addr, the guest gets rebooted and reestablishes the "remote-node-to-host" connection over the ring1_addr, which is great as long as you don't care if the guest gets rebooted. Corosync is doing its job of preventing the cluster node from being fenced by failing over its heartbeat messaging to ring1, however the remote_node guests take a short term hit due to the remote-node-to-host reconnect. In the event of a ring0_addr failure, I don't see any attempt by pacemaker to migrate the remote_node to another cluster node, but maybe this is by design, since there is no alternate path for the guest to use for LGM (i.e. ring0 is a single point of failure). If the guest could be migrated over an alternate route, it would prevent the guest outage. Maybe my question is... is there any way to facilitate an alternate Live Guest Migration path in the event of a ring0_addr failure? This might also apply to a single ring protocol as well. Thanks, Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: "Ulrich Windl" To: Date: 03/02/2017 02:39 AM Subject:[ClusterLabs] Antw: Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config? >>> "Scott Greenlese" schrieb am 01.03.2017 um 22:07 in Nachricht : > Hi.. > > I am running a few corosync "passive mode" Redundant Ring Protocol (RRP) > failure scenarios, where > my cluster has several remote-node VirtualDomain resources running on each > node in the cluster, > which have been configured to allow Live Guest Migration (LGM) operations. > > While both corosync rings are active, if I drop ring0 on a given node where > I have remote node (guests) running, > I noticed that the guest will be shutdown / re-started on the same host, > after which the connection is re-established > and the guest proceeds to run on that same cluster node. Could it be you forgot "allow-migrate=true" at the resource level or some migration IP address at the node level? I only have SLES11 here... > > I am wondering why pacemaker doesn't try to "live" migrate the remote node > (guest) to a different node, instead > of rebooting the guest? Is there some way to configure the remote nodes > such that the recovery action is > LGM instead of reboot when the host-to-remote_node connect is lost in an > RRP situation? I guess the > next question is, is it even possible to LGM a remote node guest if the > corosync ring fails over from ring0 to ring1 > (or vise-versa)? > > # For example, here's a remote node's VirtualDomain resource definition. > > [root@zs95kj]# pcs resource show zs95kjg110102_res > Resource: zs95kjg110102_res (class=ocf provider=heartbeat > type=VirtualDomain) > Attributes: config=/guestxml/nfs1/zs95kjg110102.xml > hy
[ClusterLabs] Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config?
ame: stonith-ng Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info: qb_ipcs_us_publish:server name: crmd Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info: main: Starting Feb 28 15:55:38 [942] zs95kjg110102 pacemaker_remoted: notice: lrmd_remote_listen:LRMD client connection established. 0xbed1ab50 id: b19ed532-6f61-4d9c-9439-ffb836eea34f zs95kjg110102:~ # zs95kjg110102:~ # netstat -anp |less Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 961/sshd tcp0 0 127.0.0.1:250.0.0.0:* LISTEN 1065/master tcp0 0 0.0.0.0:56660.0.0.0:* LISTEN 946/xinetd tcp0 0 0.0.0.0:58010.0.0.0:* LISTEN 946/xinetd tcp0 0 0.0.0.0:59010.0.0.0:* LISTEN 946/xinetd tcp0 0 10.20.110.102:2210.20.94.32:57749 ESTABLISHED 1134/0 tcp0 0 :::21 :::*LISTEN 941/vsftpd tcp0 0 :::22 :::*LISTEN 961/sshd tcp0 0 ::1:25 :::*LISTEN 1065/master tcp0 0 :::80 :::*LISTEN 944/httpd-prefork tcp0 0 :::3121 :::*LISTEN 942/pacemaker_remot tcp0 0 :::34836:::*LISTEN 1070/xdm tcp0 0 10.20.110.102:3121 10.20.94.212:49666 ESTABLISHED 942/pacemaker_remot udp0 0 :::177 :::* 1070/xdm ## On host node, zs95kj show system messages indicating remote node (guest) shutdown / start ... (but no attempt to LGM). [root@zs95kj ~]# grep "Feb 28" /var/log/messages |grep zs95kjg110102 Feb 28 15:55:07 zs95kj crmd[121380]: error: Operation zs95kjg110102_monitor_3: Timed Out (node=zs95kjpcs1, call=2, timeout=3ms) Feb 28 15:55:07 zs95kj crmd[121380]: error: Unexpected disconnect on remote-node zs95kjg110102 Feb 28 15:55:17 zs95kj crmd[121380]: notice: Operation zs95kjg110102_stop_0: ok (node=zs95kjpcs1, call=38, rc=0, cib-update=370, confirmed=true) Feb 28 15:55:17 zs95kj attrd[121378]: notice: Removing all zs95kjg110102 attributes for zs95kjpcs1 Feb 28 15:55:17 zs95kj VirtualDomain(zs95kjg110102_res)[173127]: INFO: Issuing graceful shutdown request for domain zs95kjg110102. Feb 28 15:55:23 zs95kj systemd-machined: Machine qemu-38-zs95kjg110102 terminated. Feb 28 15:55:23 zs95kj crmd[121380]: notice: Operation zs95kjg110102_res_stop_0: ok (node=zs95kjpcs1, call=858, rc=0, cib-update=378, confirmed=true) Feb 28 15:55:24 zs95kj systemd-machined: New machine qemu-64-zs95kjg110102. Feb 28 15:55:24 zs95kj systemd: Started Virtual Machine qemu-64-zs95kjg110102. Feb 28 15:55:24 zs95kj systemd: Starting Virtual Machine qemu-64-zs95kjg110102. Feb 28 15:55:25 zs95kj crmd[121380]: notice: Operation zs95kjg110102_res_start_0: ok (node=zs95kjpcs1, call=859, rc=0, cib-update=385, confirmed=true) Feb 28 15:55:38 zs95kj crmd[121380]: notice: Operation zs95kjg110102_start_0: ok (node=zs95kjpcs1, call=44, rc=0, cib-update=387, confirmed=true) [root@zs95kj ~]# Once the remote node established re-connection, there was no further remote node / resource instability. Anyway, just wondering why there was no attempt to migrate this remote node guest as opposed to a reboot? Is it necessary to reboot the guest in order to be managed by pacemaker and corosync over the ring1 interface if ring0 goes down? Is live guest migration even possible if ring0 goes away and ring1 takes over? Thanks in advance.. Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] How to configure my ocf_heartbeat_iscsi resource(s) such that I have both paths to the LUN?
Refreshing this post in case anyone might have missed it... still trying to figure out some way to have multiple iscsi paths managed by a single ocf_heartbeat_iscsi resource. Any ideas? Thanks! Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Scott Greenlese/Poughkeepsie/IBM@IBMUS To: users@clusterlabs.org Cc: Si Bo Niu , Michael Tebolt/Poughkeepsie/IBM@IBMUS Date: 02/15/2017 12:18 PM Subject:[ClusterLabs] How to configure my ocf_heartbeat_iscsi resource (s) such that I have both paths to the LUN? Hi folks, I'm running some test scenarios with an ocf_heartbeat_iscsi pacemaker resource, using the following XIV multipath'ed configuration: I created a single XIV iscsi host definition containing all the pacemaker host (cluster node) 'Initiator's: XIV 7812475>>host_list_ports host=pacemaker_iscsi Host Type Port Name pacemaker_iscsi iSCSI iqn.2005-03.org.open-iscsi:6539c3daf095 pacemaker_iscsi iSCSI iqn.2005-03.org.open-iscsi:11d639c0976c pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:74ca24d6476 pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:ea17bebd09a pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:b852a67852c Here is the XIV Target IP:port / IQN info: 10.20.92.108:3260 Target: iqn.2005-10.com.xivstorage:012475 10.20.92.109:3260 Target: iqn.2005-10.com.xivstorage:012475 I mapped a single 17Gb Lun to the XIV host: 20017380030BB110F >From ocf_heartbeat_iscsi man page, it didn't seem obvious to me how I might specify multiple IP Portals (i.e. both the .108 and .109 IPs) to a single ocf_heartbeat_iscsi resource. So, I went ahead and configured my resource using just the .108 IP path, as follows: Using the above XIV definitions, I created a single iscsi pacemaker resource using only one of the two IP paths to the XIV LUN: [root@zs95kj VD]# pcs resource show iscsi_r4 Resource: iscsi_r4 (class=ocf provider=heartbeat type=iscsi) Attributes: portal=10.20.92.108:3260 target=iqn.2005-10.com.xivstorage:012475 Operations: start interval=0s timeout=120 (iscsi_r4-start-interval-0s) stop interval=0s timeout=120 (iscsi_r4-stop-interval-0s) monitor interval=120 timeout=30 (iscsi_r4-monitor-interval-120) I'm looking for suggestions as to how I should configure my iscsi resource (s) such that I have both paths (.108 and .109) to the LUN available to my application? Do I need to create a second iscsi resource for the .109 path, and colocate them so that they move about the cluster together? As an aside, I have run into situations where the second IP (.109) comes online to where my iscsi_r4 resource is running (how or why, I'm not sure), which introduces problems because iscsi_r4 only manages the .108 connection. Thanks in advance... Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] How to configure my ocf_heartbeat_iscsi resource(s) such that I have both paths to the LUN?
Hi folks, I'm running some test scenarios with an ocf_heartbeat_iscsi pacemaker resource, using the following XIV multipath'ed configuration: I created a single XIV iscsi host definition containing all the pacemaker host (cluster node) 'Initiator's: XIV 7812475>>host_list_ports host=pacemaker_iscsi Host TypePort Name pacemaker_iscsi iSCSI iqn.2005-03.org.open-iscsi:6539c3daf095 pacemaker_iscsi iSCSI iqn.2005-03.org.open-iscsi:11d639c0976c pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:74ca24d6476 pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:ea17bebd09a pacemaker_iscsi iSCSI iqn.1994-05.com.redhat:b852a67852c Here is the XIV Target IP:port / IQN info: 10.20.92.108:3260 Target: iqn.2005-10.com.xivstorage:012475 10.20.92.109:3260 Target: iqn.2005-10.com.xivstorage:012475 I mapped a single 17Gb Lun to the XIV host: 20017380030BB110F >From ocf_heartbeat_iscsi man page, it didn't seem obvious to me how I might specify multiple IP Portals (i.e. both the .108 and .109 IPs) to a single ocf_heartbeat_iscsi resource. So, I went ahead and configured my resource using just the .108 IP path, as follows: Using the above XIV definitions, I created a single iscsi pacemaker resource using only one of the two IP paths to the XIV LUN: [root@zs95kj VD]# pcs resource show iscsi_r4 Resource: iscsi_r4 (class=ocf provider=heartbeat type=iscsi) Attributes: portal=10.20.92.108:3260 target=iqn.2005-10.com.xivstorage:012475 Operations: start interval=0s timeout=120 (iscsi_r4-start-interval-0s) stop interval=0s timeout=120 (iscsi_r4-stop-interval-0s) monitor interval=120 timeout=30 (iscsi_r4-monitor-interval-120) I'm looking for suggestions as to how I should configure my iscsi resource (s) such that I have both paths (.108 and .109) to the LUN available to my application? Do I need to create a second iscsi resource for the .109 path, and colocate them so that they move about the cluster together? As an aside, I have run into situations where the second IP (.109) comes online to where my iscsi_r4 resource is running (how or why, I'm not sure), which introduces problems because iscsi_r4 only manages the .108 connection. Thanks in advance... Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] pcsd 99% CPU
/lib64/libbz2.so.1 Saved corefile /guestimages/nfs1/zs95kj_cib_core_Feb6.27225 In the pcsd log, I see "pcs status nodes corosync" requests (approximately) once per 64 seconds. [root@zs95kj cluster]# grep "pcs status nodes corosync" /var/log/pcsd/pcsd.log |wc -l 1861 [root@zs95kj ~]# grep "pcs status nodes corosync" /var/log/pcsd/pcsd.log | tail I, [2017-02-06T16:18:53.468885 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:19:57.139796 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:21:00.833517 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:22:04.151613 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:23:07.803426 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:24:11.360107 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:25:14.828617 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:26:18.553637 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:27:22.363723 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:28:26.330005 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync A typical, repeating set of entries in the pcsd.log look like this: I, [2017-02-06T16:29:26.367339 #23277] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name I, [2017-02-06T16:29:26.367458 #23277] INFO -- : CIB USER: hacluster, groups: I, [2017-02-06T16:29:27.438925 #23277] INFO -- : Return Value: 0 :::10.20.93.13 - - [06/Feb/2017:16:29:27 -0500] "GET /remote/get_configs HTTP/1.1" 200 1077 1.0768 :::10.20.93.13 - - [06/Feb/2017:16:29:27 -0500] "GET /remote/get_configs HTTP/1.1" 200 1077 1.0770 zs93KLpcs1.pokprv.stglabs.ibm.com - - [06/Feb/2017:16:29:26 EST] "GET /remote/get_configs HTTP/1.1" 200 1077 - -> /remote/get_configs I, [2017-02-06T16:29:28.816100 #23277] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name I, [2017-02-06T16:29:28.816288 #23277] INFO -- : CIB USER: hacluster, groups: I, [2017-02-06T16:29:29.890788 #23277] INFO -- : Return Value: 0 I, [2017-02-06T16:29:29.890958 #23277] INFO -- : Running: /usr/sbin/pcs status nodes corosync I, [2017-02-06T16:29:29.890990 #23277] INFO -- : CIB USER: hacluster, groups: I, [2017-02-06T16:29:31.082885 #23277] INFO -- : Return Value: 0 I, [2017-02-06T16:29:31.084041 #23277] INFO -- : SRWT Node: zs90kppcs1 Request: get_configs I, [2017-02-06T16:29:31.085012 #23277] INFO -- : SRWT Node: zs93kjpcs1 Request: get_configs I, [2017-02-06T16:29:31.086098 #23277] INFO -- : SRWT Node: zs93KLpcs1 Request: get_configs I, [2017-02-06T16:29:31.088295 #23277] INFO -- : SRWT Node: zs95kjpcs1 Request: get_configs I, [2017-02-06T16:29:31.088515 #23277] INFO -- : SRWT Node: zs95KLpcs1 Request: get_configs I, [2017-02-06T16:29:31.157357 #23277] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name I, [2017-02-06T16:29:31.157480 #23277] INFO -- : CIB USER: hacluster, groups: I, [2017-02-06T16:29:32.271072 #23277] INFO -- : Return Value: 0 :::10.20.93.12 - - [06/Feb/2017:16:29:32 -0500] "GET /remote/get_configs HTTP/1.1" 200 1077 1.1190 :::10.20.93.12 - - [06/Feb/2017:16:29:32 -0500] "GET /remote/get_configs HTTP/1.1" 200 1077 1.1193 zs95kjpcs1.pokprv.stglabs.ibm.com - - [06/Feb/2017:16:29:31 EST] "GET /remote/get_configs HTTP/1.1" 200 1077 - -> /remote/get_configs Again, the pcsd query commands are returning quickly now, so all this data may not be of any use now. Comments / suggestions welcome. Thanks... Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Tomas Jelinek To: users@clusterlabs.org Date: 02/06/2017 04:55 AM Subject:Re: [ClusterLabs] pcsd 99% CPU Dne 3.2.2017 v 22:08 Scott Greenlese napsal(a): > Hi all.. > > Over the past few days, I noticed that pcsd and ruby process is pegged > at 99% CPU, and commands such as > pcs status pcsd take up to 5 minutes to complete. On all active cluster > nodes, top shows: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 27225 haclust+ 20 0 116324 91600 23136 R 99.3 0.1 1943:40 cib > 23277 root 20 0 12.868g 8.176g 8460 S 99.7 13.0 407:44.18 ruby > > The system log indicates High CIB load detected over the past 2 days: > > [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep > "Feb 3" |wc -l > 1655 > [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep > "Feb 2" |wc -l > 1658 > [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep > "Feb 1" |wc -l > 147 > [root@zs95kj ~]# grep "High CIB load detected
Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action.
Further explanation for my concern about --disabled not taking effect until after the iface-bridge was configured ... The reason I wanted to create the iface-bridge resource "disabled", was to allow me the opportunity to impose a location constraint / rule on the resource to prevent it from being started on certain cluster nodes, where the specified slave vlan did not exist. In my case, pacemaker assigned the resource to a cluster node where the specified slave vlan did not exist, which in turn triggered a fenced (off) action against that node (apparently, because the device could not be stopped, per Ken's reply earlier). Again, my cluster is configured as "symmetric" , so I would have to "opt out" my new resource from certain cluster nodes via location constraint. So, if this really is how --disable is designed to work, is there any way to impose a location constraint rule BEFORE the iface-bridge resource gets assigned. configured and started on a cluster node in a symmetrical cluster? Thanks, Scott Greenlese ... IBM KVM on System Z - Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Scott Greenlese/Poughkeepsie/IBM@IBMUS To: kgail...@redhat.com, Cluster Labs - All topics related to open-source clustering welcomed Date: 02/03/2017 03:23 PM Subject:Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action. Ken, Thanks for the explanation. One other thing, relating to the iface-bridge resource creation. I specified --disabled flag: > [root@zs95kj VD]# date;pcs resource create br0_r1 > ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op > monitor timeout="20s" interval="10s" --disabled Does the bridge device have to be successfully configured by pacemaker before disabling the resource? It seems that that was the behavior, since it failed the resource and fenced the node instead of disabling the resource. Just checking with you to be sure. Thanks again.. Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com Inactive hide details for Ken Gaillot ---02/02/2017 03:29:12 PM---On 02/02/2017 02:14 PM, Scott Greenlese wrote: > Hi folks,Ken Gaillot ---02/02/2017 03:29:12 PM---On 02/02/2017 02:14 PM, Scott Greenlese wrote: > Hi folks, From: Ken Gaillot To: users@clusterlabs.org Date: 02/02/2017 03:29 PM Subject: Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action. On 02/02/2017 02:14 PM, Scott Greenlese wrote: > Hi folks, > > I'm testing iface-bridge resource support on a Linux KVM on System Z > pacemaker cluster. > > pacemaker-1.1.13-10.el7_2.ibm.1.s390x > corosync-2.3.4-7.el7_2.ibm.1.s390x > > I created an iface-bridge resource, but specified a non-existent > bridge_slaves value, vlan1292 (i.e. vlan1292 doesn't exist). > > [root@zs95kj VD]# date;pcs resource create br0_r1 > ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op > monitor timeout="20s" interval="10s" --disabled > Wed Feb 1 17:49:16 EST 2017 > [root@zs95kj VD]# > > [root@zs95kj VD]# pcs resource show |grep br0 > br0_r1 (ocf::heartbeat:iface-bridge): FAILED zs93kjpcs1 > [root@zs95kj VD]# > > As you can see, the resource was created, but failed to start on the > target node zs93kppcs1. > > To my surprise, the target node zs93kppcs1 was unceremoniously fenced. > > pacemaker.log shows a fence (off) action initiated against that target > node, "because of resource failure(s)" : > > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug: > determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not > configured' (6) instead of the expected value: 'ok' (0) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning: > unpack_rsc_op_failure: Processing failed op stop for br0_r1 on > zs93kjpcs1: not configured (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error: > unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation > stop failed 'not configured' (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug: > determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not > configured' (6) instead of the expected value: 'ok' (0) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning: > unpack_rsc_op_failure: Processing failed op stop for br0_r1 on > zs93kjpcs1: not configured (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error: > unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation > stop failed 'not configured' (6) > Feb 01 17:55:56 [52941] zs95kj crm_resourc
[ClusterLabs] pcsd 99% CPU
Hi all.. Over the past few days, I noticed that pcsd and ruby process is pegged at 99% CPU, and commands such as pcs status pcsd take up to 5 minutes to complete. On all active cluster nodes, top shows: PID USER PR NI VIRT RES SHRS %CPU %MEM TIME+ COMMAND 27225 haclust+ 20 0 116324 91600 23136 R 99.3 0.1 1943:40cib 23277 root 200 12.868g 8.176g 8460 S 99.7 13.0407:44.18 ruby The system log indicates High CIB load detected over the past 2 days: [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep "Feb 3" |wc -l 1655 [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep "Feb 2" |wc -l 1658 [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep "Feb 1" |wc -l 147 [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep "Jan 31" |wc -l 444 [root@zs95kj ~]# grep "High CIB load detected" /var/log/messages |grep "Jan 30" |wc -l 352 The first entries logged on Feb 2 started around 8:42am ... Feb 2 08:42:12 zs95kj crmd[27233]: notice: High CIB load detected: 0.974333 This happens to coincide with the time that I had caused a node fence (off) action by creating a iface-bridge resources and specified a non-existent vlan slave interface (reported to the group yesterday in a separate email thread). It also happened to cause me to lose quorum in the cluster, because 2 of my 5 cluster nodes were already offline. My cluster currently has just over 200 VirtualDomain resources to manage, plus one iface-bridge resource and one iface-vlan resource. Both of which are currently configured properly and operational. I would appreciate some guidance how to proceed with debugging this issue. I have not taken any recovery actions yet. I considered stopping the cluster, recycling pcsd.service on all nodes, restarting cluster... and also, reboot the nodes, if necessary. But, didn't want to clear it yet in case there's anything I can capture while in this state. Thanks.. Scott Greenlese ... KVM on System Z - Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action.
Ken, Thanks for the explanation. One other thing, relating to the iface-bridge resource creation. I specified --disabled flag: > [root@zs95kj VD]# date;pcs resource create br0_r1 > ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op > monitor timeout="20s" interval="10s" --disabled Does the bridge device have to be successfully configured by pacemaker before disabling the resource? It seems that that was the behavior, since it failed the resource and fenced the node instead of disabling the resource. Just checking with you to be sure. Thanks again.. Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Ken Gaillot To: users@clusterlabs.org Date: 02/02/2017 03:29 PM Subject:Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action. On 02/02/2017 02:14 PM, Scott Greenlese wrote: > Hi folks, > > I'm testing iface-bridge resource support on a Linux KVM on System Z > pacemaker cluster. > > pacemaker-1.1.13-10.el7_2.ibm.1.s390x > corosync-2.3.4-7.el7_2.ibm.1.s390x > > I created an iface-bridge resource, but specified a non-existent > bridge_slaves value, vlan1292 (i.e. vlan1292 doesn't exist). > > [root@zs95kj VD]# date;pcs resource create br0_r1 > ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op > monitor timeout="20s" interval="10s" --disabled > Wed Feb 1 17:49:16 EST 2017 > [root@zs95kj VD]# > > [root@zs95kj VD]# pcs resource show |grep br0 > br0_r1 (ocf::heartbeat:iface-bridge): FAILED zs93kjpcs1 > [root@zs95kj VD]# > > As you can see, the resource was created, but failed to start on the > target node zs93kppcs1. > > To my surprise, the target node zs93kppcs1 was unceremoniously fenced. > > pacemaker.log shows a fence (off) action initiated against that target > node, "because of resource failure(s)" : > > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug: > determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not > configured' (6) instead of the expected value: 'ok' (0) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning: > unpack_rsc_op_failure: Processing failed op stop for br0_r1 on > zs93kjpcs1: not configured (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error: > unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation > stop failed 'not configured' (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug: > determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not > configured' (6) instead of the expected value: 'ok' (0) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning: > unpack_rsc_op_failure: Processing failed op stop for br0_r1 on > zs93kjpcs1: not configured (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error: > unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation > stop failed 'not configured' (6) > Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:96 ) warning: > pe_fence_node: Node zs93kjpcs1 will be fenced because of resource failure (s) > > > Thankfully, I was able to successfully create a iface-bridge resource > when I changed the bridge_slaves value to an existent vlan interface. > > My main concern is, why would the response to a failed bridge config > operation warrant a node fence (off) action? Isn't it enough to just > fail the resource and try another cluster node, > or at most, give up if it can't be started / configured on any node? > > Is there any way to control this harsh recovery action in the cluster? > > Thanks much.. > > > Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y. > INTERNET: swgre...@us.ibm.com It's actually the stop operation failure that leads to the fence. If a resource fails to stop, fencing is the only way pacemaker can recover the resource elsewhere. Consider a database master -- if it doesn't stop, starting the master elsewhere could lead to severe data inconsistency. You can tell pacemaker to not attempt recovery, by setting on-fail=block on the stop operation, so it doesn't need to fence. Obviously, that prevents high availability, as manual intervention is required to do anything further with the service. ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action.
Hi folks, I'm testing iface-bridge resource support on a Linux KVM on System Z pacemaker cluster. pacemaker-1.1.13-10.el7_2.ibm.1.s390x corosync-2.3.4-7.el7_2.ibm.1.s390x I created an iface-bridge resource, but specified a non-existent bridge_slaves value, vlan1292 (i.e. vlan1292 doesn't exist). [root@zs95kj VD]# date;pcs resource create br0_r1 ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op monitor timeout="20s" interval="10s" --disabled Wed Feb 1 17:49:16 EST 2017 [root@zs95kj VD]# [root@zs95kj VD]# pcs resource show |grep br0 br0_r1 (ocf::heartbeat:iface-bridge): FAILED zs93kjpcs1 [root@zs95kj VD]# As you can see, the resource was created, but failed to start on the target node zs93kppcs1. To my surprise, the target node zs93kppcs1 was unceremoniously fenced. pacemaker.log shows a fence (off) action initiated against that target node, "because of resource failure(s)" : Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:2719 ) debug: determine_op_status:br0_r1_stop_0 on zs93kjpcs1 returned 'not configured' (6) instead of the expected value: 'ok' (0) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:2602 ) warning: unpack_rsc_op_failure: Processing failed op stop for br0_r1 on zs93kjpcs1: not configured (6) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:3244 ) error: unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation stop failed 'not configured' (6) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:2719 ) debug: determine_op_status:br0_r1_stop_0 on zs93kjpcs1 returned 'not configured' (6) instead of the expected value: 'ok' (0) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:2602 ) warning: unpack_rsc_op_failure: Processing failed op stop for br0_r1 on zs93kjpcs1: not configured (6) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:3244 ) error: unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation stop failed 'not configured' (6) Feb 01 17:55:56 [52941] zs95kj crm_resource: (unpack.c:96) warning: pe_fence_node: Node zs93kjpcs1 will be fenced because of resource failure(s) Thankfully, I was able to successfully create a iface-bridge resource when I changed the bridge_slaves value to an existent vlan interface. My main concern is, why would the response to a failed bridge config operation warrant a node fence (off) action? Isn't it enough to just fail the resource and try another cluster node, or at most, give up if it can't be started / configured on any node? Is there any way to control this harsh recovery action in the cluster? Thanks much.. Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources
Ken (and Ulrich), Found it! You're right, we do deliver a man page... man]# find . -name *Virtual* -print ./man7/ocf_heartbeat_VirtualDomain.7.gz # rpm -q --whatprovides /usr/share/man/man7/ocf_heartbeat_VirtualDomain.7.gz resource-agents-3.9.7-4.el7_2.kvmibm1_1_3.1.s390x Much obliged, sir(s). Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966 From: Ken Gaillot To: users@clusterlabs.org Date: 02/01/2017 10:33 AM Subject:Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources On 02/01/2017 09:15 AM, Scott Greenlese wrote: > Hi all... > > Just a quick follow-up. > > Thought I should come clean and share with you that the incorrect > "migrate-to" operation name defined in my VirtualDomain > resource was my mistake. It was mis-coded in the virtual guest > provisioning script. I have since changed it to "migrate_to" > and of course, the specified live migration timeout value is working > effectively now. (For some reason, I assumed we were letting that > operation meta value default). > > I was wondering if someone could refer me to the definitive online link > for pacemaker resource man pages? I don't see any resource man pages > installed > on my system anywhere. I found this one online: > https://www.mankier.com/7/ocf_heartbeat_VirtualDomain but is there a > more 'official' page I should refer our > Linux KVM on System z customers to? All distributions that I know of include the man pages with the packages they distribute. Are you building from source? They are named like "man ocf_heartbeat_IPaddr2". FYI after following this thread, the pcs developers are making a change so that pcs refuses to add an unrecognized operation unless the user uses --force. Thanks for being involved in the community; this is how we learn to improve! > Thanks again for your assistance. > > Scott Greenlese ...IBM KVM on System Z Solution Test Poughkeepsie, N.Y. > INTERNET: swgre...@us.ibm.com > > > Inactive hide details for "Ulrich Windl" ---01/27/2017 02:32:43 AM--->>> > "Scott Greenlese" schrieb am 27."Ulrich Windl" > ---01/27/2017 02:32:43 AM--->>> "Scott Greenlese" > schrieb am 27.01.2017 um 02:47 in Nachricht > > From: "Ulrich Windl" > To: , Scott Greenlese/Poughkeepsie/IBM@IBMUS > Cc: "Si Bo Niu" , Michael Tebolt/Poughkeepsie/IBM@IBMUS > Date: 01/27/2017 02:32 AM > Subject: Antw: Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts > for VirtualDomain resources > > > > > >>>> "Scott Greenlese" schrieb am 27.01.2017 um > 02:47 in > Nachricht > > m>: > >> Hi guys.. >> >> Well, today I confirmed that what Ulrich said is correct. If I update the >> VirtualDomain resource with the operation name "migrate_to" instead of >> "migrate-to", it effectively overrides and enforces the 1200ms default >> value to the new value. >> >> I am wondering how I would have known that I was using the wrong operation >> name, when the initial operation name is already incorrect >> when the resource is created? > > For SLES 11, I made a quick (portable non-portable unstable) try (print > the operations known to an RA): > # crm ra info VirtualDomain |sed -n -e "/Operations' defaults/,\$p" > Operations' defaults (advisory minimum): > >start timeout=90 >stop timeout=90 >statustimeout=30 interval=10 >monitor timeout=30 interval=10 >migrate_from timeout=60 >migrate_totimeout=120 > > Regards, > Ulrich > >> >> This is what the meta data for my resource looked like after making the >> update: >> >> [root@zs95kj VD]# date;pcs resource update zs95kjg110065_res op migrate_to >> timeout="360s" >> Thu Jan 26 16:43:11 EST 2017 >> You have new mail in /var/spool/mail/root >> >> [root@zs95kj VD]# date;pcs resource show zs95kjg110065_res >> Thu Jan 26 16:43:46 EST 2017 >> Resource: zs95kjg110065_res (class=ocf provider=heartbeat >> type=VirtualDomain) >> Attributes: config=/guestxml/nfs1/zs95kjg110065.xml >> hypervisor=qemu:///system migration_transport=ssh >> Meta Attrs: allow-migrate=true >> Operations: start interval=0s timeout=120 >> (zs95kjg110065_res-start-interval-0s) >> stop interval=0s timeout=120 >> (zs95kjg110065_res-stop-interval-0s) >> monitor
Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources
Hi all... Just a quick follow-up. Thought I should come clean and share with you that the incorrect "migrate-to" operation name defined in my VirtualDomain resource was my mistake. It was mis-coded in the virtual guest provisioning script. I have since changed it to "migrate_to" and of course, the specified live migration timeout value is working effectively now. (For some reason, I assumed we were letting that operation meta value default). I was wondering if someone could refer me to the definitive online link for pacemaker resource man pages? I don't see any resource man pages installed on my system anywhere. I found this one online: https://www.mankier.com/7/ocf_heartbeat_VirtualDomain but is there a more 'official' page I should refer our Linux KVM on System z customers to? Thanks again for your assistance. Scott Greenlese ...IBM KVM on System Z Solution Test Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: "Ulrich Windl" To: , Scott Greenlese/Poughkeepsie/IBM@IBMUS Cc: "Si Bo Niu" , Michael Tebolt/Poughkeepsie/IBM@IBMUS Date: 01/27/2017 02:32 AM Subject:Antw: Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts for VirtualDomain resources >>> "Scott Greenlese" schrieb am 27.01.2017 um 02:47 in Nachricht : > Hi guys.. > > Well, today I confirmed that what Ulrich said is correct. If I update the > VirtualDomain resource with the operation name "migrate_to" instead of > "migrate-to", it effectively overrides and enforces the 1200ms default > value to the new value. > > I am wondering how I would have known that I was using the wrong operation > name, when the initial operation name is already incorrect > when the resource is created? For SLES 11, I made a quick (portable non-portable unstable) try (print the operations known to an RA): # crm ra info VirtualDomain |sed -n -e "/Operations' defaults/,\$p" Operations' defaults (advisory minimum): start timeout=90 stop timeout=90 statustimeout=30 interval=10 monitor timeout=30 interval=10 migrate_from timeout=60 migrate_totimeout=120 Regards, Ulrich > > This is what the meta data for my resource looked like after making the > update: > > [root@zs95kj VD]# date;pcs resource update zs95kjg110065_res op migrate_to > timeout="360s" > Thu Jan 26 16:43:11 EST 2017 > You have new mail in /var/spool/mail/root > > [root@zs95kj VD]# date;pcs resource show zs95kjg110065_res > Thu Jan 26 16:43:46 EST 2017 > Resource: zs95kjg110065_res (class=ocf provider=heartbeat > type=VirtualDomain) > Attributes: config=/guestxml/nfs1/zs95kjg110065.xml > hypervisor=qemu:///system migration_transport=ssh > Meta Attrs: allow-migrate=true > Operations: start interval=0s timeout=120 > (zs95kjg110065_res-start-interval-0s) > stop interval=0s timeout=120 > (zs95kjg110065_res-stop-interval-0s) > monitor interval=30s (zs95kjg110065_res-monitor-interval-30s) > migrate-from interval=0s timeout=1200 > (zs95kjg110065_res-migrate-from-interval-0s) > migrate-to interval=0s timeout=1200 > (zs95kjg110065_res-migrate-to-interval-0s) <<< Original op name / value > migrate_to interval=0s timeout=360s > (zs95kjg110065_res-migrate_to-interval-0s) <<< New op name / value > > > Where does that original op name come from in the VirtualDomain resource > definition? How can we get the initial meta value changed and shipped with > a valid operation name (i.e. migrate_to), and > maybe a more reasonable migrate_to timeout value... something significantly > higher than 1200ms , i.e. 1.2 seconds ? Can I report this request as a > bugzilla on the RHEL side, or should this go to my internal IBM bugzilla > for KVM on System Z development? > > Anyway, thanks so much for identifying my issue. I can reconfigure my > resources to make them tolerate longer migration execution times. > > > Scott Greenlese ... IBM KVM on System Z Solution Test > INTERNET: swgre...@us.ibm.com > > > > > From: Ken Gaillot > To:Ulrich Windl , > users@clusterlabs.org > Date: 01/19/2017 10:26 AM > Subject: Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts for > VirtualDomain resources > > > > On 01/19/2017 01:36 AM, Ulrich Windl wrote: >>>>> Ken Gaillot schrieb am 18.01.2017 um 16:32 in > Nachricht >> <4b02d3fa-4693-473b-8bed-dc98f9e3f...@redhat.com>: >>> On 01/17/2017 04:45 PM, Scott Greenlese wrote: >>>> Ken and Co, >>>> >>>&g
Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts for VirtualDomain resources
Hi guys.. Well, today I confirmed that what Ulrich said is correct. If I update the VirtualDomain resource with the operation name "migrate_to" instead of "migrate-to", it effectively overrides and enforces the 1200ms default value to the new value. I am wondering how I would have known that I was using the wrong operation name, when the initial operation name is already incorrect when the resource is created? This is what the meta data for my resource looked like after making the update: [root@zs95kj VD]# date;pcs resource update zs95kjg110065_res op migrate_to timeout="360s" Thu Jan 26 16:43:11 EST 2017 You have new mail in /var/spool/mail/root [root@zs95kj VD]# date;pcs resource show zs95kjg110065_res Thu Jan 26 16:43:46 EST 2017 Resource: zs95kjg110065_res (class=ocf provider=heartbeat type=VirtualDomain) Attributes: config=/guestxml/nfs1/zs95kjg110065.xml hypervisor=qemu:///system migration_transport=ssh Meta Attrs: allow-migrate=true Operations: start interval=0s timeout=120 (zs95kjg110065_res-start-interval-0s) stop interval=0s timeout=120 (zs95kjg110065_res-stop-interval-0s) monitor interval=30s (zs95kjg110065_res-monitor-interval-30s) migrate-from interval=0s timeout=1200 (zs95kjg110065_res-migrate-from-interval-0s) migrate-to interval=0s timeout=1200 (zs95kjg110065_res-migrate-to-interval-0s) <<< Original op name / value migrate_to interval=0s timeout=360s (zs95kjg110065_res-migrate_to-interval-0s) <<< New op name / value Where does that original op name come from in the VirtualDomain resource definition? How can we get the initial meta value changed and shipped with a valid operation name (i.e. migrate_to), and maybe a more reasonable migrate_to timeout value... something significantly higher than 1200ms , i.e. 1.2 seconds ? Can I report this request as a bugzilla on the RHEL side, or should this go to my internal IBM bugzilla for KVM on System Z development? Anyway, thanks so much for identifying my issue. I can reconfigure my resources to make them tolerate longer migration execution times. Scott Greenlese ... IBM KVM on System Z Solution Test INTERNET: swgre...@us.ibm.com From: Ken Gaillot To: Ulrich Windl , users@clusterlabs.org Date: 01/19/2017 10:26 AM Subject:Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts for VirtualDomain resources On 01/19/2017 01:36 AM, Ulrich Windl wrote: >>>> Ken Gaillot schrieb am 18.01.2017 um 16:32 in Nachricht > <4b02d3fa-4693-473b-8bed-dc98f9e3f...@redhat.com>: >> On 01/17/2017 04:45 PM, Scott Greenlese wrote: >>> Ken and Co, >>> >>> Thanks for the useful information. >>> > > [...] >>> >>> Is this internally coded within the class=ocf provider=heartbeat >>> type=VirtualDomain resource agent? >> >> Aha, I just realized what the issue is: the operation name is >> migrate_to, not migrate-to. >> >> For technical reasons, pacemaker can't validate operation names (at the >> time that the configuration is edited, it does not necessarily have >> access to the agent metadata). > > BUT the set of operations is finite, right? So if those were in some XML schema, the names could be verified at least (not meaning that the operation is actually supported). > BTW: Would a "crm configure verify" detect this kijnd of problem? > > [...] > > Ulrich Yes, it's in the resource agent meta-data. While pacemaker itself uses a small set of well-defined actions, the agent may define any arbitrarily named actions it desires, and the user could configure one of these as a recurring action in pacemaker. Pacemaker itself has to be liberal about where its configuration comes from -- the configuration can be edited on a separate machine, which doesn't have resource agents, and then uploaded to the cluster. So Pacemaker can't do that validation at configuration time. (It could theoretically do some checking after the fact when the configuration is loaded, but this could be a lot of overhead, and there are implementation issues at the moment.) Higher-level tools like crmsh and pcs, on the other hand, can make simplifying assumptions. They can require access to the resource agents so that they can do extra validation. ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources
out, which occurred at 13:55:14 : Jan 17 13:54:54 [27555] zs95kj crmd: ( lrm.c:2005 ) debug: do_lrm_rsc_op:Stopped 0 recurring operations in preparation for zs95kjg110061_res_migrate_to_0 Jan 17 13:54:54 [27555] zs95kj crmd: ( lrm.c:2009 )info: do_lrm_rsc_op:Performing key=337:574:0:43c68253-0e7d-4bb5-a78b-9b3c9af677dd op=zs95kjg110061_res_migrate_to_0 Jan 17 13:54:54 [27555] zs95kj crmd: (lrmd_client.:817 ) trace: lrmd_send_command:sending lrmd_rsc_exec op to lrmd Jan 17 13:54:54 [27555] zs95kj crmd: (lrmd_client.:488 ) trace: lrmd_create_op: Sending call options: 0004, 4 Jan 17 13:54:54 [27552] zs95kj lrmd: ( lrmd.c:1644 ) trace: process_lrmd_message: Processing lrmd_rsc_exec operation from a2a90d95-a766-42ba-84a0-54574c10adee Jan 17 13:54:54 [27552] zs95kj lrmd: ( lrmd.c:347 ) trace: schedule_lrmd_cmd:Scheduling migrate_to on zs95kjg110061_res Jan 17 13:54:54 [27552] zs95kj lrmd: ( lrmd.c:1684 ) debug: process_lrmd_message: Processed lrmd_rsc_exec operation from a2a90d95-a766-42ba-84a0-54574c10adee: rc=941, reply=1, notify=0, exit=-2147471352 Jan 17 13:54:54 [27552] zs95kj lrmd: ( lrmd.c:131 )info: log_execute: executing - rsc:zs95kjg110061_res action:migrate_to call_id:941 Jan 17 13:54:54 [27552] zs95kj lrmd: ( lrmd.c:1184 ) trace: lrmd_rsc_execute_service_lib: Creating action, resource:zs95kjg110061_res action:migrate_to class:ocf provider:heartbeat agent:VirtualDomain Jan 17 13:54:54 [27555] zs95kj crmd: (lrmd_client.:846 ) trace: lrmd_send_command:lrmd_rsc_exec op reply received Jan 17 13:54:54 [27555] zs95kj crmd: (lrmd_client.:852 ) trace: lrmd_send_command:Reply Jan 17 13:54:54 [27555] zs95kj crmd: (lrmd_client.:300 ) trace: lrmd_dispatch_internal: op lrmd_rsc_exec notify event received Jan 17 13:54:54 [27555] zs95kj crmd: ( lrm.c:2385 )info: process_lrm_event:Operation zs95kjg110061_res_monitor_3: Cancelled (node=zs95kjpcs1, call=936, confirmed=true) Jan 17 13:54:54 [27555] zs95kj crmd: ( lrm.c:196 ) debug: update_history_cache: Updating history for 'zs95kjg110061_res' with monitor op VirtualDomain(zs95kjg110061_res)[135045]: 2017/01/17_13:54:54 DEBUG: Virtual domain zs95kjg110061 is currently running. VirtualDomain(zs95kjg110061_res)[135045]: 2017/01/17_13:54:54 INFO: zs95kjg110061: Starting live migration to zs90kppcs1 (using: virsh --connect=qemu:///system --quiet migrate --live zs95kjg110061 qemu +ssh://zs90kppcs1/system ). Jan 17 13:54:54 [27550] zs95kjcib: ( xml.c:1728 )info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm [@id='2']/lrm_resources/lr m_resource[@id='zs95kjg110061_res']/lrm_rsc_op [@id='zs95kjg110061_res_last_0']: @operation_key=zs95kjg110061_res_migrate_to_0, @operation=migrate_to, @crm-debu g-origin=cib_action_update, @transition-key=337:574:0:43c68253-0e7d-4bb5-a78b-9b3c9af677dd, @transition-magic=-1:193;337:574:0:43c68253-0e7d-4bb5-a78b-9b3c9af67 7dd, @call-id=-1, @rc-code=193, @op-s ### 20 seconds after the live migration was initiated... we get the time out: Jan 17 13:55:14 [27552] zs95kj lrmd: ( mainloop.c:949 ) warning: child_timeout_callback: zs95kjg110061_res_migrate_to_0 process (PID 135045) timed out Jan 17 13:55:14 [27552] zs95kj lrmd: (services_lin:314 ) warning: operation_finished: zs95kjg110061_res_migrate_to_0:135045 - timed out after 2ms Jan 17 13:55:14 [27552] zs95kj lrmd: (services_lin:332 ) debug: operation_finished: zs95kjg110061_res_migrate_to_0:135045:stderr [ -- empty -- ] Jan 17 13:55:14 [27552] zs95kj lrmd: (services_lin:336 ) debug: operation_finished: zs95kjg110061_res_migrate_to_0:135045:stdout [ -- empty -- ] Jan 17 13:55:14 [27552] zs95kj lrmd: ( lrmd.c:592 ) trace: cmd_finalize: Resource operation rsc:zs95kjg110061_res action:migrate_to completed (0xa5f1edd0 0xa5f1edd0) Jan 17 13:55:14 [27552] zs95kj lrmd: ( lrmd.c:113 )info: log_finished: finished - rsc:zs95kjg110061_res action:migrate_to call_id:941 pid:135045 exit-code:1 exec-time:20003ms queue-time:0ms Jan 17 13:55:14 [27552] zs95kj lrmd: ( lrmd.c:1292 ) trace: lrmd_rsc_execute: Nothing further to do for zs95kjg110061_res Jan 17 13:55:14 [27555] zs95kj crmd: ( utils.c:1942 ) debug: create_operation_update: do_update_resource: Updating resource zs95kjg110061_res after migrate_to op Timed Out (interval=0) Jan 17 13:55:14 [27555] zs95kj crmd: ( lrm.c:2397 ) error: process_lrm_event:Operation zs95kjg110061_res_migrate_to_0: Timed Out (node=zs95kjpcs1, call=941, timeout=2ms) Jan 17 13:55:14 [27555] zs95kj crmd: ( lrm.c:196 ) debug: update_history_cache: Updating history for 'zs95kjg110061_res&
[ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources
would really prefer to have a good understanding of timeout configuration and recovery behavior first. Thanks! Scott Greenlese ... IBM KVM on System z - Solution Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker quorum behavior
A quick addendum... After sending this post, I decided to stop pacemaker on the single, Online node in the cluster, and this effectively killed the corosync daemon: [root@zs93kl VD]# date;pcs cluster stop Wed Sep 28 16:39:22 EDT 2016 Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... [root@zs93kl VD]# date;ps -ef |grep coro|grep -v grep Wed Sep 28 16:46:19 EDT 2016 [root@zs93kl VD]# Next, I went to a node in "Pending" state, and sure enough... the pcs cluster stop killed the daemon there, too: [root@zs95kj VD]# date;pcs cluster stop Wed Sep 28 16:48:15 EDT 2016 Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... [root@zs95kj VD]# date;ps -ef |grep coro |grep -v grep Wed Sep 28 16:48:38 EDT 2016 [root@zs95kj VD]# So, this answers my own question... cluster stop should kill corosync. So, why isn't the `pcs cluster stop --all` failing to kill corosync? Thanks... Scott Greenlese ... IBM KVM on System Z Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Scott Greenlese/Poughkeepsie/IBM To: kgail...@redhat.com, Cluster Labs - All topics related to open-source clustering welcomed Date: 09/28/2016 04:30 PM Subject:Re: [ClusterLabs] Pacemaker quorum behavior Hi folks.. I have some follow-up questions about corosync daemon status after cluster shutdown. Basically, what should happen to corosync on a cluster node when pacemaker is shutdown on that node? On my 5 node cluster, when I do a global shutdown, the pacemaker processes exit, but corosync processes remain active. Here's an example of where this led me into some trouble... My cluster is still configured to use the "symmetric" resource distribution. I don't have any location constraints in place, so pacemaker tries to evenly distribute resources across all Online nodes. With one cluster node (KVM host) powered off, I did the global cluster stop: [root@zs90KP VD]# date;pcs cluster stop --all Wed Sep 28 15:07:40 EDT 2016 zs93KLpcs1: Unable to connect to zs93KLpcs1 ([Errno 113] No route to host) zs90kppcs1: Stopping Cluster (pacemaker)... zs95KLpcs1: Stopping Cluster (pacemaker)... zs95kjpcs1: Stopping Cluster (pacemaker)... zs93kjpcs1: Stopping Cluster (pacemaker)... Error: unable to stop all nodes zs93KLpcs1: Unable to connect to zs93KLpcs1 ([Errno 113] No route to host) Note: The "No route to host" messages are expected because that node / LPAR is powered down. (I don't show it here, but the corosync daemon is still running on the 4 active nodes. I do show it later). I then powered on the one zs93KLpcs1 LPAR, so in theory I should not have quorum when it comes up and activates pacemaker, which is enabled to autostart at boot time on all 5 cluster nodes. At this point, only 1 out of 5 nodes should be Online to the cluster, and therefore ... no quorum. I login to zs93KLpcs1, and pcs status shows those 4 nodes as 'pending' Online, and "partition with quorum": [root@zs93kl ~]# date;pcs status |less Wed Sep 28 15:25:13 EDT 2016 Cluster name: test_cluster_2 Last updated: Wed Sep 28 15:25:13 2016 Last change: Mon Sep 26 16:15:08 2016 by root via crm_resource on zs95kjpcs1 Stack: corosync Current DC: zs93KLpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition with quorum 106 nodes and 304 resources configured Node zs90kppcs1: pending Node zs93kjpcs1: pending Node zs95KLpcs1: pending Node zs95kjpcs1: pending Online: [ zs93KLpcs1 ] Full list of resources: zs95kjg109062_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg109063_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 . . . Here you can see that corosync is up on all 5 nodes: [root@zs95kj VD]# date;for host in zs90kppcs1 zs95KLpcs1 zs95kjpcs1 zs93kjpcs1 zs93KLpcs1 ; do ssh $host "hostname;ps -ef |grep corosync |grep -v grep"; done Wed Sep 28 15:22:21 EDT 2016 zs90KP root 155374 1 0 Sep26 ?00:10:17 corosync zs95KL root 22933 1 0 11:51 ?00:00:54 corosync zs95kj root 19382 1 0 Sep26 ?00:10:15 corosync zs93kj root 129102 1 0 Sep26 ?00:12:10 corosync zs93kl root 21894 1 0 15:19 ?00:00:00 corosync But, pacemaker is only running on the one, online node: [root@zs95kj VD]# date;for host in zs90kppcs1 zs95KLpcs1 zs95kjpcs1 zs93kjpcs1 zs93KLpcs1 ; do ssh $host "hostname;ps -ef |grep pacemakerd | grep -v grep"; done Wed Sep 28 15:23:29 EDT 2016 zs90KP zs95KL zs95kj zs93kj zs93kl root 23005 1 0 15:19 ?00:00:00 /usr/sbin/pacemakerd -f You have new mail in /var/spool/mail/root [root@zs95kj VD]# This situation wreaks havoc on my VirtualDomain resources, as the majority of them are in FAILED or Stopped state, and to my surprise... many of them show as Started: [root@zs93kl VD]# date;pcs resource show |grep zs93KL Wed Sep 28 15:55:29 EDT 2016 zs95kjg109062_res (ocf::heartbea
Re: [ClusterLabs] Pacemaker quorum behavior
::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110122_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110123_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110124_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110125_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110126_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110128_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110129_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110130_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110131_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110132_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110133_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110134_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110135_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110137_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110138_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110139_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110140_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110141_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110142_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110143_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110144_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110145_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110146_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110148_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110149_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110150_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110152_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110154_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110155_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110156_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110159_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110160_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110161_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110164_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1 zs95kjg110165_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110166_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 Pacemaker is attempting to activate all VirtualDomain resources on the one cluster node. So back to my original question... what should happen when I do a cluster stop? If it should be deactivating, what would prevent this? Also, I have tried simulating a failed cluster node (to trigger a STONITH action) by killing the corosync daemon on one node, but all that does is respawn the daemon ... causing a temporary / transient failure condition, and no fence takes place. Is there a way to kill corosync in such a way that it stays down? Is there a best practice for STONITH testing? As usual, thanks in advance for your advice. Scott Greenlese ... IBM KVM on System Z - Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com From: Ken Gaillot To: users@clusterlabs.org Date: 09/09/2016 06:23 PM Subject:Re: [ClusterLabs] Pacemaker quorum behavior On 09/09/2016 04:27 AM, Klaus Wenninger wrote: > On 09/08/2016 07:31 PM, Scott Greenlese wrote: >> >> Hi Klaus, thanks for your prompt and thoughtful feedback... >> >> Please see my answers nested below (sections entitled, "Scott's >> Reply"). Thanks! >> >> - Scott >> >> >> Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y. >> INTERNET: swgre...@us.ibm.com >> PHONE: 8/293-7301 (845-433-7301) M/S: POK 42HA/P966 >> >> >> Inactive hide details for Klaus Wenninger ---09/08/2016 10:59:27 >> AM---On 09/08/2016 03:55 PM, Scott Greenlese wrote: >Klaus Wenninger >> ---09/08/2016 10:59:27 AM---On 09/08/2016 03:55 PM, Scott Greenlese >> wrote: > >> >> From: Klaus Wenninger >> To: users@clusterlabs.org >> Date: 09/08/2016 10:59 AM >> Subject: Re: [ClusterLabs] Pacemaker quorum behavior >> >> >> >> >> >> On 09/08/2016 03:55 PM, Scott Greenlese wrote: >> > >> > Hi all... >> > >> > I have a few very basic questions for the group. >> > >> > I have a 5 node (Linux on Z LPARs) pacemaker cluster with 100 >> > VirtualDomain pacemaker-remote nodes
Re: [ClusterLabs] "VirtualDomain is active on 2 nodes" due to transient network failure
Hi Ken ,
Below where you commented,
"It's considered good practice to stop
pacemaker+corosync before rebooting a node intentionally (for even more
safety, you can put the node into standby first)."
.. is this something that we document anywhere?
Our 'reboot' action performs a halt (deactivate lpar) and then activate.
Do I run the risk
of guest instances running on multiple hosts in my case? I'm performing
various recovery
scenarios and want to avoid this procedure (reboot without first stopping
cluster), if it's not supported.
By the way, I always put the node in cluster standby before an intentional
reboot.
Thanks!
Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966
From: Ken Gaillot
To: users@clusterlabs.org
Date: 09/02/2016 10:01 AM
Subject:Re: [ClusterLabs] "VirtualDomain is active on 2 nodes" due to
transient network failure
On 09/01/2016 09:39 AM, Scott Greenlese wrote:
> Andreas,
>
> You wrote:
>
> /"Would be good to see your full cluster configuration (corosync.conf
> and cib) - but first guess is: no fencing at all and what is your
> "no-quorum-policy" in Pacemaker?/
>
> /Regards,/
> /Andreas"/
>
> Thanks for your interest. I actually do have a stonith device configured
> which maps all 5 cluster nodes in the cluster:
>
> [root@zs95kj ~]# date;pcs stonith show fence_S90HMC1
> Thu Sep 1 10:11:25 EDT 2016
> Resource: fence_S90HMC1 (class=stonith type=fence_ibmz)
> Attributes: ipaddr=9.12.35.134 login=stonith passwd=lnx4ltic
>
pcmk_host_map=zs95KLpcs1:S95/KVL;zs93KLpcs1:S93/KVL;zs93kjpcs1:S93/KVJ;zs95kjpcs1:S95/KVJ;zs90kppcs1:S90/PACEMAKER
> pcmk_host_list="zs95KLpcs1 zs93KLpcs1 zs93kjpcs1 zs95kjpcs1 zs90kppcs1"
> pcmk_list_timeout=300 pcmk_off_timeout=600 pcmk_reboot_action=off
> pcmk_reboot_timeout=600
> Operations: monitor interval=60s (fence_S90HMC1-monitor-interval-60s)
>
> This fencing device works, too well actually. It seems extremely
> sensitive to node "failures", and I'm not sure how to tune that. Stonith
> reboot actoin is 'off', and the general stonith action (cluster config)
> is also 'off'. In fact, often if I reboot a cluster node (i.e. reboot
> command) that is an active member in the cluster... stonith will power
> off that node while it's on its wait back up. (perhaps requires a
> separate issue thread on this forum?).
That depends on what a reboot does in your OS ... if it shuts down the
cluster services cleanly, you shouldn't get a fence, but if it kills
anything still running, then the cluster will see the node as failed,
and fencing is appropriate. It's considered good practice to stop
pacemaker+corosync before rebooting a node intentionally (for even more
safety, you can put the node into standby first).
>
> My no-quorum-policy is: no-quorum-policy: stop
>
> I don't think I should have lost quorum, only two of the five cluster
> nodes lost their corosync ring connection.
Those two nodes lost quorum, so they should have stopped all their
resources. And the three remaining nodes should have fenced them.
I'd check the logs around the time of the incident. Do the two affected
nodes detect the loss of quorum? Do they attempt to stop their
resources? Do those stops succeed? Do the other three nodes detect the
loss of the two nodes? Does the DC attempt to fence them? Do the fence
attempts succeed?
> Here's the full configuration:
>
>
> [root@zs95kj ~]# cat /etc/corosync/corosync.conf
> totem {
> version: 2
> secauth: off
> cluster_name: test_cluster_2
> transport: udpu
> }
>
> nodelist {
> node {
> ring0_addr: zs93kjpcs1
> nodeid: 1
> }
>
> node {
> ring0_addr: zs95kjpcs1
> nodeid: 2
> }
>
> node {
> ring0_addr: zs95KLpcs1
> nodeid: 3
> }
>
> node {
> ring0_addr: zs90kppcs1
> nodeid: 4
> }
>
> node {
> ring0_addr: zs93KLpcs1
> nodeid: 5
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> logging {
> #Log to a specified file
> to_logfile: yes
> logfile: /var/log/corosync/corosync.log
> #Log timestamp as well
> timestamp: on
>
> #Facility in syslog
> syslog_facility: daemon
>
> logger_subsys {
> #Enable debug for this logger.
>
> debug: off
>
> #This specifies the subsystem identity (name) for which logging is
specified
>
> subsys: QUORUM
>
> }
> #Log to syslog
> to_syslog: yes
>
> #Whether or not turning on the debug information in the log
> debug: on
> }
> [root@zs95kj ~]#
>
>
>
> The full CIB (see attachment)
>
> [root@zs95kj ~]# pcs cluster cib > /tm
Re: [ClusterLabs] Pacemaker quorum behavior
ualDomain): Started zs95kjpcs1
zs95kjg109064_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
zs95kjg109065_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
zs95kjg109066_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
zs95kjg109067_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
zs95kjg109068_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
.,
.
.
PCSD Status:
zs93kjpcs1: Online
zs95kjpcs1: Online
zs95KLpcs1: Online
zs90kppcs1: Offline
zs93KLpcs1: Online
Check resources again:
Wed Sep 7 16:09:52 EDT 2016
### VirtualDomain Resource Statistics: ###
"_res" Virtual Domain resources:
Started on zs95kj: 199
Started on zs93kj: 0
Started on zs95KL: 0
Started on zs93KL: 0
Started on zs90KP: 0
Total Started: 199
Total NOT Started: 1
I have since isolated all the corrupted virtual domain images and disabled
their VirtualDomain resources.
We already rebooted all five cluster nodes, after installing a new KVM
driver on them.
Now, the quorum calculation and behavior seems to be working perfectly as
expected.
I started pacemaker on the nodes, one at a time... and, after 3 of the 5
nodes had pacemaker "Online" ...
resources activated and were evenly distributed across them.
In summary, a lesson learned here is to check status of the pcs process to
be certain pacemaker and corosync
are indeed "offline" and that all threads to that process have terminated.
You had mentioned this command:
pstree -p | grep -A5 $(pidof -x pcs)
I'm not quite sure what the $(pidof -x pcs) represents??
On an "Online" cluster node, I see:
[root@zs93kj ~]# ps -ef |grep pcs |grep -v grep
root 18876 1 0 Sep07 ?
00:00:00 /bin/sh /usr/lib/pcsd/pcsd start
root 18905 18876 0 Sep07 ?00:00:00 /bin/bash -c ulimit -S -c
0 >/dev/null 2>&1 ; /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb
root 18906 18905 0 Sep07 ?00:04:22 /usr/bin/ruby
-I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb
[root@zs93kj ~]#
If I use the 18876 PID on a healthy node, I get..
[root@zs93kj ~]# pstree -p |grep -A5 18876
|-pcsd(18876)---bash(18905)---ruby(18906)-+-{ruby}(19102)
| |-{ruby}(20212)
| `-{ruby}(224258)
|-pkcsslotd(18851)
|-polkitd(19091)-+-{polkitd}(19100)
||-{polkitd}(19101)
Is this what you meant for me to do?If so, I'll be sure to do that next
time I suspect processes are not exiting on cluster kill or stop.
Thanks
Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966
From: Jan Pokorný
To: Cluster Labs - All topics related to open-source clustering
welcomed
Cc: Si Bo Niu , Scott
Loveland/Poughkeepsie/IBM@IBMUS, Michael
Tebolt/Poughkeepsie/IBM@IBMUS
Date: 09/08/2016 02:43 PM
Subject:Re: [ClusterLabs] Pacemaker quorum behavior
On 08/09/16 10:20 -0400, Scott Greenlese wrote:
> Correction...
>
> When I stopped pacemaker/corosync on the four (powered on / active)
> cluster node hosts, I was having an issue with the gentle method of
> stopping the cluster (pcs cluster stop --all),
Can you elaborate on what went wrong with this gentle method, please?
If it seemed to have stuck, you can perhaps run some diagnostics like:
pstree -p | grep -A5 $(pidof -x pcs)
across the nodes to see if what process(es) pcs waits on, next time.
> so I ended up doing individual (pcs cluster kill ) on
> each of the four cluster nodes. I then had to stop the virtual
> domains manually via 'virsh destroy ' on each host.
> Perhaps there was some residual node status affecting my quorum?
Hardly if corosync processes were indeed dead.
--
Jan (Poki)
[attachment "attyopgs.dat" deleted by Scott Greenlese/Poughkeepsie/IBM]
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker quorum behavior
Hi Klaus, thanks for your prompt and thoughtful feedback...
Please see my answers nested below (sections entitled, "Scott's Reply").
Thanks!
- Scott
Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966
From: Klaus Wenninger
To: users@clusterlabs.org
Date: 09/08/2016 10:59 AM
Subject:Re: [ClusterLabs] Pacemaker quorum behavior
On 09/08/2016 03:55 PM, Scott Greenlese wrote:
>
> Hi all...
>
> I have a few very basic questions for the group.
>
> I have a 5 node (Linux on Z LPARs) pacemaker cluster with 100
> VirtualDomain pacemaker-remote nodes
> plus 100 "opaque" VirtualDomain resources. The cluster is configured
> to be 'symmetric' and I have no
> location constraints on the 200 VirtualDomain resources (other than to
> prevent the opaque guests
> from running on the pacemaker remote node resources). My quorum is set
> as:
>
> quorum {
> provider: corosync_votequorum
> }
>
> As an experiment, I powered down one LPAR in the cluster, leaving 4
> powered up with the pcsd service up on the 4 survivors
> but corosync/pacemaker down (pcs cluster stop --all) on the 4
> survivors. I then started pacemaker/corosync on a single cluster
>
"pcs cluster stop" shuts down pacemaker & corosync on my test-cluster but
did you check the status of the individual services?
Scott's reply:
No, I only assumed that pacemaker was down because I got this back on my
pcs status
command from each cluster node:
[root@zs95kj VD]# date;for host in zs93KLpcs1 zs95KLpcs1 zs95kjpcs1
zs93kjpcs1 ; do ssh $host pcs status; done
Wed Sep 7 15:49:27 EDT 2016
Error: cluster is not currently running on this node
Error: cluster is not currently running on this node
Error: cluster is not currently running on this node
Error: cluster is not currently running on this node
What else should I check? The pcsd.service service was still up, since I
didn't not stop that
anywhere. Should I have done, ps -ef |grep -e pacemaker -e corosync to
check the state before
assuming it was really down?
> node (pcs cluster start), and this resulted in the 200 VirtualDomain
> resources activating on the single node.
> This was not what I was expecting. I assumed that no resources would
> activate / start on any cluster nodes
> until 3 out of the 5 total cluster nodes had pacemaker/corosync running.
>
> After starting pacemaker/corosync on the single host (zs95kjpcs1),
> this is what I see :
>
> [root@zs95kj VD]# date;pcs status |less
> Wed Sep 7 15:51:17 EDT 2016
> Cluster name: test_cluster_2
> Last updated: Wed Sep 7 15:51:18 2016 Last change: Wed Sep 7 15:30:12
> 2016 by hacluster via crmd on zs93kjpcs1
> Stack: corosync
> Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) -
> partition with quorum
> 106 nodes and 304 resources configured
>
> Node zs93KLpcs1: pending
> Node zs93kjpcs1: pending
> Node zs95KLpcs1: pending
> Online: [ zs95kjpcs1 ]
> OFFLINE: [ zs90kppcs1 ]
>
> .
> .
> .
> PCSD Status:
> zs93kjpcs1: Online
> zs95kjpcs1: Online
> zs95KLpcs1: Online
> zs90kppcs1: Offline
> zs93KLpcs1: Online
>
> So, what exactly constitutes an "Online" vs. "Offline" cluster node
> w.r.t. quorum calculation? Seems like in my case, it's "pending" on 3
> nodes,
> so where does that fall? Any why "pending"? What does that mean?
>
> Also, what exactly is the cluster's expected reaction to quorum loss?
> Cluster resources will be stopped or something else?
>
Depends on how you configure it using cluster property no-quorum-policy
(default: stop).
Scott's reply:
This is how the policy is configured:
[root@zs95kj VD]# date;pcs config |grep quorum
Thu Sep 8 13:18:33 EDT 2016
no-quorum-policy: stop
What should I expect with the 'stop' setting?
>
>
> Where can I find this documentation?
>
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/
Scott's reply:
OK, I'll keep looking thru this doc, but I don't easily find the
no-quorum-policy explained.
Thanks..
>
>
> Thanks!
>
> Scott Greenlese - IBM Solution Test Team.
>
>
>
> Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com
> PHONE: 8/293-7301 (845-433-7301) M/S: POK 42HA/P966
>
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
___
Re: [ClusterLabs] Pacemaker quorum behavior
Correction...
When I stopped pacemaker/corosync on the four (powered on / active) cluster
node hosts, I was having an issue with
the gentle method of stopping the cluster (pcs cluster stop --all), so I
ended up doing individual (pcs cluster kill )
on each of the four cluster nodes. I then had to stop the virtual domains
manually via 'virsh destroy ' on each host.
Perhaps there was some residual node status affecting my quorum?
Thanks...
Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966
From: Scott Greenlese/Poughkeepsie/IBM@IBMUS
To: users@clusterlabs.org
Cc: Si Bo Niu , Scott
Loveland/Poughkeepsie/IBM@IBMUS, Michael
Tebolt/Poughkeepsie/IBM@IBMUS
Date: 09/08/2016 10:01 AM
Subject:[ClusterLabs] Pacemaker quorum behavior
Hi all...
I have a few very basic questions for the group.
I have a 5 node (Linux on Z LPARs) pacemaker cluster with 100 VirtualDomain
pacemaker-remote nodes
plus 100 "opaque" VirtualDomain resources. The cluster is configured to be
'symmetric' and I have no
location constraints on the 200 VirtualDomain resources (other than to
prevent the opaque guests
from running on the pacemaker remote node resources). My quorum is set as:
quorum {
provider: corosync_votequorum
}
As an experiment, I powered down one LPAR in the cluster, leaving 4 powered
up with the pcsd service up on the 4 survivors
but corosync/pacemaker down (pcs cluster stop --all) on the 4 survivors. I
then started pacemaker/corosync on a single cluster
node (pcs cluster start), and this resulted in the 200 VirtualDomain
resources activating on the single node.
This was not what I was expecting. I assumed that no resources would
activate / start on any cluster nodes
until 3 out of the 5 total cluster nodes had pacemaker/corosync running.
After starting pacemaker/corosync on the single host (zs95kjpcs1), this is
what I see :
[root@zs95kj VD]# date;pcs status |less
Wed Sep 7 15:51:17 EDT 2016
Cluster name: test_cluster_2
Last updated: Wed Sep 7 15:51:18 2016 Last change: Wed Sep 7 15:30:12 2016
by hacluster via crmd on zs93kjpcs1
Stack: corosync
Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition
with quorum
106 nodes and 304 resources configured
Node zs93KLpcs1: pending
Node zs93kjpcs1: pending
Node zs95KLpcs1: pending
Online: [ zs95kjpcs1 ]
OFFLINE: [ zs90kppcs1 ]
.
.
.
PCSD Status:
zs93kjpcs1: Online
zs95kjpcs1: Online
zs95KLpcs1: Online
zs90kppcs1: Offline
zs93KLpcs1: Online
So, what exactly constitutes an "Online" vs. "Offline" cluster node w.r.t.
quorum calculation? Seems like in my case, it's "pending" on 3 nodes,
so where does that fall? Any why "pending"? What does that mean?
Also, what exactly is the cluster's expected reaction to quorum loss?
Cluster resources will be stopped or something else?
Where can I find this documentation?
Thanks!
Scott Greenlese - IBM Solution Test Team.
Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301) M/S: POK 42HA/P966
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Pacemaker quorum behavior
Hi all...
I have a few very basic questions for the group.
I have a 5 node (Linux on Z LPARs) pacemaker cluster with 100 VirtualDomain
pacemaker-remote nodes
plus 100 "opaque" VirtualDomain resources. The cluster is configured to be
'symmetric' and I have no
location constraints on the 200 VirtualDomain resources (other than to
prevent the opaque guests
from running on the pacemaker remote node resources). My quorum is set as:
quorum {
provider: corosync_votequorum
}
As an experiment, I powered down one LPAR in the cluster, leaving 4 powered
up with the pcsd service up on the 4 survivors
but corosync/pacemaker down (pcs cluster stop --all) on the 4 survivors.
I then started pacemaker/corosync on a single cluster
node (pcs cluster start), and this resulted in the 200 VirtualDomain
resources activating on the single node.
This was not what I was expecting. I assumed that no resources would
activate / start on any cluster nodes
until 3 out of the 5 total cluster nodes had pacemaker/corosync running.
After starting pacemaker/corosync on the single host (zs95kjpcs1), this is
what I see :
[root@zs95kj VD]# date;pcs status |less
Wed Sep 7 15:51:17 EDT 2016
Cluster name: test_cluster_2
Last updated: Wed Sep 7 15:51:18 2016 Last change: Wed Sep 7
15:30:12 2016 by hacluster via crmd on zs93kjpcs1
Stack: corosync
Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition
with quorum
106 nodes and 304 resources configured
Node zs93KLpcs1: pending
Node zs93kjpcs1: pending
Node zs95KLpcs1: pending
Online: [ zs95kjpcs1 ]
OFFLINE: [ zs90kppcs1 ]
.
.
.
PCSD Status:
zs93kjpcs1: Online
zs95kjpcs1: Online
zs95KLpcs1: Online
zs90kppcs1: Offline
zs93KLpcs1: Online
So, what exactly constitutes an "Online" vs. "Offline" cluster node w.r.t.
quorum calculation? Seems like in my case, it's "pending" on 3 nodes,
so where does that fall? Any why "pending"? What does that mean?
Also, what exactly is the cluster's expected reaction to quorum loss?
Cluster resources will be stopped or something else?
Where can I find this documentation?
Thanks!
Scott Greenlese - IBM Solution Test Team.
Scott Greenlese ... IBM Solutions Test, Poughkeepsie, N.Y.
INTERNET: swgre...@us.ibm.com
PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] "VirtualDomain is active on 2 nodes" due to transient network failure
Andreas Kurz, I see that you replied to my post, but I can't seem to find any text written from you?? Can you please re-post your reply, or message me offline at swgre...@us.ibm.com? I'm new to the process, so maybe something I did wrong in my post. Thanks for your guidance. Scott G. Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966 From: Andreas Kurz To: Cluster Labs - All topics related to open-source clustering welcomed Date: 08/30/2016 05:06 PM Subject:Re: [ClusterLabs] "VirtualDomain is active on 2 nodes" due to transient network failure Hi, On Tue, Aug 30, 2016 at 10:03 PM, Scott Greenlese wrote: Added an appropriate subject line (was blank). Thanks... Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301) M/S: POK 42HA/P966 - Forwarded by Scott Greenlese/Poughkeepsie/IBM on 08/30/2016 03:59 PM - From: Scott Greenlese/Poughkeepsie/IBM@IBMUS To: Cluster Labs - All topics related to open-source clustering welcomed Date: 08/29/2016 06:36 PM Subject: [ClusterLabs] (no subject) Hi folks, I'm assigned to system test Pacemaker/Corosync on the KVM on System Z platform with pacemaker-1.1.13-10 and corosync-2.3.4-7 . Would be good to see your full cluster configuration (corosync.conf and cib) - but first guess is: no fencing at all and what is your "no-quorum-policy" in Pacemaker? Regards, Andreas I have a cluster with 5 KVM hosts, and a total of 200 ocf:pacemakerVirtualDomain resources defined to run across the 5 cluster nodes (symmertical is true for this cluster). The heartbeat network is communicating over vlan1293, which is hung off a network device, 0230 . In general, pacemaker does a good job of distributing my virtual guest resources evenly across the hypervisors in the cluster. These resource are a mixed bag: - "opaque" and remote "guest nodes" managed by the cluster. - allow-migrate=false and allow-migrate=true - qcow2 (file based) guests and LUN based guests - Sles and Ubuntu OS [root@zs95kj ]# pcs status |less Cluster name: test_cluster_2 Last updated: Mon Aug 29 17:02:08 2016 Last change: Mon Aug 29 16:37:31 2016 by root via crm_resource on zs93kjpcs1 Stack: corosync Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition with quorum 103 nodes and 300 resources configured Node zs90kppcs1: standby Online: [ zs93KLpcs1 zs93kjpcs1 zs95KLpcs1 zs95kjpcs1 ] This morning, our system admin team performed a "non-disruptive" (concurrent) microcode code load on the OSA, which (to our surprise) dropped the network connection for 13 seconds on the S93 CEC, from 11:18:34am to 11:18:47am , to be exact. This temporary outage caused the two cluster nodes on S93 (zs93kjpcs1 and zs93KLpcs1) to drop out of the cluster, as expected. However, pacemaker didn't handle this too well. The end result was numerous VirtualDomain resources in FAILED state: [root@zs95kj log]# date;pcs status |grep VirtualD |grep zs93 |grep FAILED Mon Aug 29 12:33:32 EDT 2016 zs95kjg110104_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110092_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110099_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110102_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110106_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110112_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110115_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110118_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110124_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110127_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110130_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110136_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110139_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110142_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110148_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110152_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110155_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110161_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110164_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110167_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110173_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110176_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110179_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110185_res (ocf
[ClusterLabs] "VirtualDomain is active on 2 nodes" due to transient network failure
Added an appropriate subject line (was blank). Thanks... Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966 - Forwarded by Scott Greenlese/Poughkeepsie/IBM on 08/30/2016 03:59 PM - From: Scott Greenlese/Poughkeepsie/IBM@IBMUS To: Cluster Labs - All topics related to open-source clustering welcomed Date: 08/29/2016 06:36 PM Subject:[ClusterLabs] (no subject) Hi folks, I'm assigned to system test Pacemaker/Corosync on the KVM on System Z platform with pacemaker-1.1.13-10 and corosync-2.3.4-7 . I have a cluster with 5 KVM hosts, and a total of 200 ocf:pacemakerVirtualDomain resources defined to run across the 5 cluster nodes (symmertical is true for this cluster). The heartbeat network is communicating over vlan1293, which is hung off a network device, 0230 . In general, pacemaker does a good job of distributing my virtual guest resources evenly across the hypervisors in the cluster. These resource are a mixed bag: - "opaque" and remote "guest nodes" managed by the cluster. - allow-migrate=false and allow-migrate=true - qcow2 (file based) guests and LUN based guests - Sles and Ubuntu OS [root@zs95kj ]# pcs status |less Cluster name: test_cluster_2 Last updated: Mon Aug 29 17:02:08 2016 Last change: Mon Aug 29 16:37:31 2016 by root via crm_resource on zs93kjpcs1 Stack: corosync Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition with quorum 103 nodes and 300 resources configured Node zs90kppcs1: standby Online: [ zs93KLpcs1 zs93kjpcs1 zs95KLpcs1 zs95kjpcs1 ] This morning, our system admin team performed a "non-disruptive" (concurrent) microcode code load on the OSA, which (to our surprise) dropped the network connection for 13 seconds on the S93 CEC, from 11:18:34am to 11:18:47am , to be exact. This temporary outage caused the two cluster nodes on S93 (zs93kjpcs1 and zs93KLpcs1) to drop out of the cluster, as expected. However, pacemaker didn't handle this too well. The end result was numerous VirtualDomain resources in FAILED state: [root@zs95kj log]# date;pcs status |grep VirtualD |grep zs93 |grep FAILED Mon Aug 29 12:33:32 EDT 2016 zs95kjg110104_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110092_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110099_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110102_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110106_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110112_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110115_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110118_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110124_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110127_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110130_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110136_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110139_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110142_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1 zs95kjg110148_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110152_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110155_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110161_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110164_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110167_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110173_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110176_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110179_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg110185_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 zs95kjg109106_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1 As well as, several VirtualDomain resources showing "Started" on two cluster nodes: zs95kjg110079_res (ocf::heartbeat:VirtualDomain): Started[ zs93kjpcs1 zs93KLpcs1 ] zs95kjg110108_res (ocf::heartbeat:VirtualDomain): Started[ zs93kjpcs1 zs93KLpcs1 ] zs95kjg110186_res (ocf::heartbeat:VirtualDomain): Started[ zs93kjpcs1 zs93KLpcs1 ] zs95kjg110188_res (ocf::heartbeat:VirtualDomain): Started[ zs93kjpcs1 zs93KLpcs1 ] zs95kjg110198_res (ocf::heartbeat:VirtualDomain): Started[ zs93kjpcs1 zs93KLpcs1 ] The virtual machines themselves were in fact, "running" on both hosts. For example: [root@zs93kl ~]# virsh list |grep zs95kjg110079 70 zs95kjg110079 running [root@zs93kj cli]# virsh list |grep zs95kjg110079 18 zs95kjg110079 running On this particular VM, here was file corruption of this file-based qcow2 guest's image, such that you could not ping or ssh, and if you open a virsh console, you get "initramfs" prompt. To recover, we had to mount the volume on a
[ClusterLabs] (no subject)
otice: [Action 429]: Pending pseudo op zs95kjg110079_res_stop_0 on N/A (priority: 0, waiting: 460 491 517 542 567) Aug 29 13:00:04 zs93kl crmd[20001]: notice: [Action 428]: Completed rsc op zs95kjg110079_res_stop_0 on zs93kjpcs1 (priority: 0, waiting: none) System log on zs93kjpcs1: Aug 29 11:20:48 zs93kj crmd[21402]: notice: Recurring action zs95kjg110079_res:817 (zs95kjg110079_res_monitor_3) incomplete at shutdown Aug 29 11:22:07 zs93kj crmd[259639]: notice: Operation zs95kjg110079_res_monitor_0: ok (node=zs93kjpcs1, call=1223, rc=0, cib-update=104, confirmed=true) Aug 29 12:59:42 zs93kj VirtualDomain(zs95kjg110079_res)[9148]: INFO: Issuing graceful shutdown request for domain zs95kjg110079. Finally zs95kjg110079 shuts down on ZS93KJ at 12:59 === Does this "active on two nodes" recovery process look right? What is the recommended procedure to "undo" the resource failures and dual host assignments? It took several hours (short of stopping/starting the entire cluster) to recover them... resource disable, cleanup, enable was the basis ... but it seemed that I would fix one resource and two more would fall out. This seems to be one of the pitfalls of configuring resources in symmetrical mode. I would appreciate any best practice guidelines you have to offer. I saved the system logs on all hosts in case anyone needs more detailed information. I also have pacemaker.log logs. Thanks in advance! Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y. INTERNET: swgre...@us.ibm.com PHONE: 8/293-7301 (845-433-7301)M/S: POK 42HA/P966 ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
