Re: [ClusterLabs] STONITH not communicated back to initiator until token expires
Thanks for your reply Digimer. On Mon, Mar 13, 2017 at 1:35 PM, Digimerwrote: > On 13/03/17 12:07 PM, Chris Walker wrote: > > Hello, > > > > On our two-node EL7 cluster (pacemaker: 1.1.15-11.el7_3.4; corosync: > > 2.4.0-4; libqb: 1.0-1), > > it looks like successful STONITH operations are not communicated from > > stonith-ng back to theinitiator (in this case, crmd) until the STONITHed > > node is removed from the cluster when > > Corosync notices that it's gone (i.e., after the token timeout). > > Others might have more useful info, but my understanding of a lost node > sequence is this; > > 1. Node stops responding, corosync declares it lost after token timeout > 2. Corosync reforms the cluster with remaining node(s), checks if it is > quorate (always true in 2-node) > 3. Corosync informs Pacemaker of the membership change. > 4. Pacemaker invokes stonith, waits for the fence agent to return > "success" (exit code of the agent as per the FenceAgentAPI > [https://docs.pagure.org/ClusterLabs.fence-agents/FenceAgentAPI.md]). If > the method fails, it moves on to the next method. If all methods fail, > it goes back to the first method and tries again, looping indefinitely. > > That's roughly my understanding as well for the case when a node suddenly leaves the cluster (e.g., poweroff), and this case is working as expected for me. I'm seeing delays when a node is marked for STONITH while it's still up (e.g., after a stop operation fails). In this case, what I expect to see is something like: 1. crmd requests that stonith-ng fence the node 2. stonith-ng (might be a different stonith-ng) fences the node and sends a message that it has succeeded 3. stonith-ng (the original from step 1) receives this message and communicates back to crmd that the node has been fenced but what I'm seeing is 1. crmd requests that stonith-ng fence the node 2. stonith-ng fences the node and sends a message saying that it has succeeded 3. nobody hears this message 4. Corosync eventually realizes that the fenced node is no longer part of the config and broadcasts a config change 5. stonith-ng finishes the STONITH operation that was started earlier and communicates back to crmd that the node has been STONITHed I'm less convinced that the sending of the STONITH notify in step 2 is at fault; it also seems possible that a callback is not being run when it should be. The Pacemaker configuration is not important (I've seen this happen on our production clusters and on a small sandbox), but the config is: primitive bug0-stonith stonith:fence_ipmilan \ params pcmk_host_list=bug0 ipaddr=bug0-ipmi action=off login=admin passwd=admin \ meta target-role=Started primitive bug1-stonith stonith:fence_ipmilan \ params pcmk_host_list=bug1 ipaddr=bug1-ipmi action=off login=admin passwd=admin \ meta target-role=Started primitive prm-snmp-heartbeat snmptrap_heartbeat \ params snmphost=bug0 community=public \ op monitor interval=10 timeout=300 \ op start timeout=300 interval=0 \ op stop timeout=300 interval=0 clone cln-snmp-heartbeat prm-snmp-heartbeat \ meta interleave=true globally-unique=false ordered=false notify=false location bug0-stonith-loc bug0-stonith -inf: bug0 location bug1-stonith-loc bug1-stonith -inf: bug1 The corosync config might be more interesting: totem { version: 2 crypto_cipher: none crypto_hash: none secauth: off rrp_mode: passive transport: udpu token: 24 consensus: 1000 interface { ringnumber 0 bindnetaddr: 203.0.113.0 mcastport: 5405 ttl: 1 } } nodelist { node { ring0_addr: 203.0.113.1 nodeid: 1 } node { ring0_addr: 203.0.113.2 nodeid: 2 } } quorum { provider: corosync_votequorum two_node: 1 } > In trace debug logs, I see the STONITH reply sent via the > > cpg_mcast_joined (libqb) function in crm_cs_flush > > (stonith_send_async_reply->send_cluster_text->send_cluster_ > text->send_cpg_iov->crm_cs_flush->cpg_mcast_joined) > > > > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( commands.c:1891 ) trace: > > stonith_send_async_reply:Reply> t="stonith-ng" st_op="st_fence" st_device_id="ustonith:0" > > st_remote_op="39b1f1e0-b76f-4d25-bd15-77b956c914a0" > > st_clientid="823e92da-8470-491a-b662-215526cced22" > > st_clientname="crmd.3973" st_target="bug1" st_device_action="st_fence" > > st_callid="3" st_callopt="0" st_rc="0" st_output="Chassis Power Control: > > Reset\nChassis Power Control: Down/Off\nChassis Power Control: > Down/Off\nC > > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( cpg.c:636 ) trace: > > send_cluster_text: Queueing CPG message 9 to all (1041 bytes, 449 > > bytes payload): > st_op="st_notify" st_device_id="ustonith:0" > > st_remote_op="39b1f1e0-b76f-4d25-bd15-77b956c914a0" > >
Re: [ClusterLabs] Sync Apache config files
You have many options to do that: - lsyncd+csync2/rsync - cron+rsync - glusterfs replicated f's Lsyncd + csync2 is a good option if you need near to real time replication Cron + rsync is a good option for scheduled replication Glusterfs is good for real time replication For 2+ nodes Regards El mar. 13, 2017 3:30 PM, "Dimitri Maziuk"escribió: > On 03/13/2017 12:38 PM, Frank Fiene wrote: > > Hmm, Puppet is also a good idea. Thanks. > > > > I haven’t tried because we have not so many Linux servers. Just a > handful. > > It depends on how much data there is and what the update model is. For > larger datasets zfs incremental snapshots work best IME, although on our > two-node active/passive pairs I haven't had any problems with DRBD > either -- as long as it's not exported via nfs on centos > 7/corosync/pacemaker. > > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > > ___ > Users mailing list: Users@clusterlabs.org > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > > ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Sync Apache config files
On 03/13/2017 12:38 PM, Frank Fiene wrote: > Hmm, Puppet is also a good idea. Thanks. > > I haven’t tried because we have not so many Linux servers. Just a handful. It depends on how much data there is and what the update model is. For larger datasets zfs incremental snapshots work best IME, although on our two-node active/passive pairs I haven't had any problems with DRBD either -- as long as it's not exported via nfs on centos 7/corosync/pacemaker. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Sync Apache config files
Hmm, Puppet is also a good idea. Thanks. I haven’t tried because we have not so many Linux servers. Just a handful. Cheers -- Frank Fiene IT-Security Manager VEKA Group Fon: +49 2526 29-6200 Fax: +49 2526 29-16-6200 mailto: ffi...@veka.com http://www.veka.com PGP-ID: 62112A51 PGP-Fingerprint: 7E12 D61B 40F0 212D 5A55 765D 2A3B B29B 6211 2A51 Threema: VZK5NDWW VEKA AG Dieselstr. 8 48324 Sendenhorst Deutschland/Germany Vorstand/Executive Board: Andreas Hartleif (Vorsitzender/CEO), Dr. Andreas W. Hillebrand, Bonifatius Eichwald, Elke Hartleif, Dr. Werner Schuler, Vorsitzender des Aufsichtsrates/Chairman of Supervisory Board: Ulrich Weimer HRB 8282 AG Münster/District Court of Münster > Am 13.03.2017 um 18:15 schrieb Dimitri Maziuk: > > On 03/13/2017 06:02 AM, Frank Fiene wrote: >> Hi, >> >> one short beginner question: >> >> Is it possible to sync for instance Apache sites-available directory > across all cluster members without DRBD etc? > > Short beginner answer: yes. > >> Can pcsd do this? > > No. > > There's any number of ways to do it but pcs is not one of them. > Automation solutions like chef or puppet can do it (saltstack has an > event-reactor system that can make it transparent, don't know about > others), you could put it on zfs and clone snapshots, syncthing, rsync, > and so on. > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > ___ > Users mailing list: Users@clusterlabs.org > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] STONITH not communicated back to initiator until token expires
On 13/03/17 12:07 PM, Chris Walker wrote: > Hello, > > On our two-node EL7 cluster (pacemaker: 1.1.15-11.el7_3.4; corosync: > 2.4.0-4; libqb: 1.0-1), > it looks like successful STONITH operations are not communicated from > stonith-ng back to theinitiator (in this case, crmd) until the STONITHed > node is removed from the cluster when > Corosync notices that it's gone (i.e., after the token timeout). Others might have more useful info, but my understanding of a lost node sequence is this; 1. Node stops responding, corosync declares it lost after token timeout 2. Corosync reforms the cluster with remaining node(s), checks if it is quorate (always true in 2-node) 3. Corosync informs Pacemaker of the membership change. 4. Pacemaker invokes stonith, waits for the fence agent to return "success" (exit code of the agent as per the FenceAgentAPI [https://docs.pagure.org/ClusterLabs.fence-agents/FenceAgentAPI.md]). If the method fails, it moves on to the next method. If all methods fail, it goes back to the first method and tries again, looping indefinitely. > In trace debug logs, I see the STONITH reply sent via the > cpg_mcast_joined (libqb) function in crm_cs_flush > (stonith_send_async_reply->send_cluster_text->send_cluster_text->send_cpg_iov->crm_cs_flush->cpg_mcast_joined) > > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( commands.c:1891 ) trace: > stonith_send_async_reply:Replyt="stonith-ng" st_op="st_fence" st_device_id="ustonith:0" > st_remote_op="39b1f1e0-b76f-4d25-bd15-77b956c914a0" > st_clientid="823e92da-8470-491a-b662-215526cced22" > st_clientname="crmd.3973" st_target="bug1" st_device_action="st_fence" > st_callid="3" st_callopt="0" st_rc="0" st_output="Chassis Power Control: > Reset\nChassis Power Control: Down/Off\nChassis Power Control: Down/Off\nC > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( cpg.c:636 ) trace: > send_cluster_text: Queueing CPG message 9 to all (1041 bytes, 449 > bytes payload): st_op="st_notify" st_device_id="ustonith:0" > st_remote_op="39b1f1e0-b76f-4d25-bd15-77b956c914a0" > st_clientid="823e92da-8470-491a-b662-215526cced22" st_clientna > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( cpg.c:207 ) trace: > send_cpg_iov:Queueing CPG message 9 (1041 bytes) > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( cpg.c:170 ) trace: > crm_cs_flush:CPG message sent, size=1041 > Mar 13 07:18:22 [6466] bug0 stonith-ng: ( cpg.c:185 ) trace: > crm_cs_flush:Sent 1 CPG messages (0 remaining, last=9): OK (1) > > But I see no further action from stonith-ng until minutes later; > specifically, I don't see remote_op_done run, so the dead node is still > an 'online (unclean)' member of the array and no failover can take place. > > When the token expires (yes, we use a very long token), I see the following: > > Mar 13 07:22:11 [6466] bug0 stonith-ng: (membership.c:1018 ) notice: > crm_update_peer_state_iter: Node bug1 state is now lost | nodeid=2 > previous=member source=crm_update_peer_proc > Mar 13 07:22:11 [6466] bug0 stonith-ng: ( main.c:1275 ) debug: > st_peer_update_callback: Broadcasting our uname because of node 2 > Mar 13 07:22:11 [6466] bug0 stonith-ng: ( cpg.c:636 ) trace: > send_cluster_text: Queueing CPG message 10 to all (666 bytes, 74 > bytes payload): t="stonith-ng" st_op="poke"/> > ... > Mar 13 07:22:11 [6466] bug0 stonith-ng: ( commands.c:2582 ) debug: > stonith_command: Processing st_notify reply 0 from bug0 ( 0) > Mar 13 07:22:11 [6466] bug0 stonith-ng: (remote.c:1945 ) debug: > process_remote_stonith_exec: Marking call to poweroff for bug1 on > behalf of crmd.3973@39b1f1e0-b76f-4d25-bd15-77b956c914a0.bug1: OK (0) > > and the STONITH command is finally communicated back to crmd as complete > and failover commences. > > Is this delay a feature of the cpg_mcast_joined function? If I > understand correctly (unlikely), it looks like cpg_mcast_joined is not > completing because one of the nodes in the group is missing, but I > haven't looked at that code closely yet. Is it advisable to have > stonith-ng broadcast a membership change when it successfully fences a node? > > Attaching logs with PCMK_debug=stonith-ng > and > PCMK_trace_functions=stonith_send_async_reply,send_cluster_text,send_cpg_iov,crm_cs_flush > > Thanks in advance, > Chris Can you share your full pacemaker config (please obfuscate passwords). -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs:
Re: [ClusterLabs] Sync Apache config files
On 03/13/2017 06:02 AM, Frank Fiene wrote: > Hi, > > one short beginner question: > > Is it possible to sync for instance Apache sites-available directory across all cluster members without DRBD etc? Short beginner answer: yes. > Can pcsd do this? No. There's any number of ways to do it but pcs is not one of them. Automation solutions like chef or puppet can do it (saltstack has an event-reactor system that can make it transparent, don't know about others), you could put it on zfs and clone snapshots, syncthing, rsync, and so on. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] STONITH not communicated back to initiator until token expires
Hello, On our two-node EL7 cluster (pacemaker: 1.1.15-11.el7_3.4; corosync: 2.4.0-4; libqb: 1.0-1), it looks like successful STONITH operations are not communicated from stonith-ng back to theinitiator (in this case, crmd) until the STONITHed node is removed from the cluster when Corosync notices that it's gone (i.e., after the token timeout). In trace debug logs, I see the STONITH reply sent via the cpg_mcast_joined (libqb) function in crm_cs_flush (stonith_send_async_reply->send_cluster_text->send_cluster_text->send_cpg_iov->crm_cs_flush->cpg_mcast_joined) Mar 13 07:18:22 [6466] bug0 stonith-ng: ( commands.c:1891 ) trace: stonith_send_async_reply:Reply ... Mar 13 07:22:11 [6466] bug0 stonith-ng: ( commands.c:2582 ) debug: stonith_command: Processing st_notify reply 0 from bug0 ( 0) Mar 13 07:22:11 [6466] bug0 stonith-ng: (remote.c:1945 ) debug: process_remote_stonith_exec: Marking call to poweroff for bug1 on behalf of crmd.3973@39b1f1e0-b76f-4d25-bd15-77b956c914a0.bug1: OK (0) and the STONITH command is finally communicated back to crmd as complete and failover commences. Is this delay a feature of the cpg_mcast_joined function? If I understand correctly (unlikely), it looks like cpg_mcast_joined is not completing because one of the nodes in the group is missing, but I haven't looked at that code closely yet. Is it advisable to have stonith-ng broadcast a membership change when it successfully fences a node? Attaching logs with PCMK_debug=stonith-ng and PCMK_trace_functions=stonith_send_async_reply,send_cluster_text,send_cpg_iov,crm_cs_flush Thanks in advance, Chris pacemaker.log.bz2 Description: BZip2 compressed data ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Sync Apache config files
Hi, As far as I know pcsd has no such functionality. This is possible using Cluster Sync tool csync2 http://oss.linbit.com/csync2/. On 13/03/17 13:02, Frank Fiene wrote: Hi, one short beginner question: Is it possible to sync for instance Apache sites-available directory across all cluster members without DRBD etc? Can pcsd do this? Kind Regards! Frank — Frank Fiene IT-Security Manager VEKA Group Fon: +49 2526 29-6200 Fax: +49 2526 29-16-6200 mailto: ffi...@veka.com http://www.veka.com PGP-ID: 62112A51 PGP-Fingerprint: 7E12 D61B 40F0 212D 5A55 765D 2A3B B29B 6211 2A51 Threema: VZK5NDWW VEKA AG Dieselstr. 8 48324 Sendenhorst Deutschland/Germany Vorstand/Executive Board: Andreas Hartleif (Vorsitzender/CEO), Dr. Andreas W. Hillebrand, Bonifatius Eichwald, Elke Hartleif, Dr. Werner Schuler, Vorsitzender des Aufsichtsrates/Chairman of Supervisory Board: Ulrich Weimer HRB 8282 AG Münster/District Court of Münster ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Regards Denis Gribkov smime.p7s Description: S/MIME Cryptographic Signature ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Sync Apache config files
Hi, one short beginner question: Is it possible to sync for instance Apache sites-available directory across all cluster members without DRBD etc? Can pcsd do this? Kind Regards! Frank — Frank Fiene IT-Security Manager VEKA Group Fon: +49 2526 29-6200 Fax: +49 2526 29-16-6200 mailto: ffi...@veka.com http://www.veka.com PGP-ID: 62112A51 PGP-Fingerprint: 7E12 D61B 40F0 212D 5A55 765D 2A3B B29B 6211 2A51 Threema: VZK5NDWW VEKA AG Dieselstr. 8 48324 Sendenhorst Deutschland/Germany Vorstand/Executive Board: Andreas Hartleif (Vorsitzender/CEO), Dr. Andreas W. Hillebrand, Bonifatius Eichwald, Elke Hartleif, Dr. Werner Schuler, Vorsitzender des Aufsichtsrates/Chairman of Supervisory Board: Ulrich Weimer HRB 8282 AG Münster/District Court of Münster ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] corosync cannot acquire quorum
On 11/03/17 02:50, cys wrote: > We have a cluster containing 3 nodes(nodeA, nodeB, nodeC). > After nodeA is taken offline(by ifdown, this may be not right?), ifdown isn't right, no. you need to do a physical cable pull or use iptables to simulate loss of traffic, ifdown does odd things to corosync! Chrissie nodeC > cannot acquire quorum while nodeB can. > NodeC: corosync-quorumtool -s > Quorum information > -- > Date: Sat Mar 11 10:42:22 2017 > Quorum provider: corosync_votequorum > Nodes2 > Node ID: 2 > Ring ID: 2/24 > Quorate: No > > Votequorum information > -- > Expected votes: 3 > Highest expected: 3 > Total votes: 2 > Quorum: 2 Activity blocked > Flags:WaitForAll > > Membership information > -- > Nodeid Votes Name > > 2 1 68.68.68.15 (local) > 3 1 68.68.68.16 > > NodeB: corosync-quorumtools -s > Quorum information > -- > Date: Sat Mar 11 10:45:44 2017 > Quorum provider: corosync_votequorum > Nodes:2 > Node ID: 3 > Ring ID: 2/24 > Quorate: Yes > > Votequorum information > -- > Expected votes: 3 > Highest expected: 3 > Total votes: 2 > Quorum: 2 > Flags:Quorate > > Membership information > -- > Nodeid Votes Name > 2 1 68.68.68.15 > 3 1 68.68.68.16 (local) > > So what's the problem? > Thanks. > > > ___ > Users mailing list: Users@clusterlabs.org > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org