[Pacemaker] reorg of network:ha-clustering repo on build.opensuse.org
Hi All, This is just a quick heads-up. We're in the process of reorganising the network:ha-clustering repository on build.opensuse.org. If you don't use any of the software from this repo feel free to stop reading now :) Currently we have: - network:ha-clustering (stable builds for various distros) - network:ha-clustering:Factory (devel project for openSUSE:Factory) This is going to change to: - network:ha-clustering:Stable (stable builds for various distros) - network:ha-clustering:Unstable (unstable/dev, various distros) - network:ha-clustering:Factory (devel project for openSUSE:Factory) This means that if you're currently using packages from network:ha-clustering, you'll need to point to network:ha-clustering:Stable instead (once we've finished shuffling everything around). I'll send another email out when this is done. Regards, Tim -- Tim Serong Senior Clustering Engineer SUSE tser...@suse.com ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] heartbeat 3.0 start error
On 25/07/2013, at 2:36 PM, claire huang claire.hu...@utstarcom.cn wrote: Andrew, Hi！there is one question to ask for your help. 1、My os is- [root@2U_222 cluster]# lsb_release -a LSB Version: :core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.2 (Final) Release: 6.2 Codename: Final 2、I install heartbeat 3.0 on my server, Ok, but you're not using it. You're using corosync instead. which is corosync+pacemaker+heartbeat。But，when I start the command “corosync”， I hope the correct situation is： ps aufx root 5001 0.3 0.0 553648 6764 ?Ssl 03:13 0:00 corosync root 5006 0.0 0.0 86032 2308 ?S03:13 0:00 \_ /usr/lib64/heartbeat/stonithd 499 5007 0.7 0.0 92444 5228 ?S03:13 0:00 \_ /usr/lib64/heartbeat/cib root 5008 0.0 0.0 74004 2316 ?S03:13 0:00 \_ /usr/lib64/heartbeat/lrmd 499 5009 0.0 0.0 94656 2592 ?S03:13 0:00 \_ /usr/lib64/heartbeat/attrd 499 5010 0.0 0.0 86712 1956 ?S03:13 0:00 \_ /usr/lib64/heartbeat/pengine 499 5011 0.2 0.0 94868 3232 ?S03:13 0:00 \_ /usr/lib64/heartbeat/crmd But, in fact , it sometimes is Ps aufx root 6542 0.0 0.0 483772 4504 ?Ssl 19:06 0:00 corosync root 6548 0.0 0.0 194256 2236 ?S19:06 0:00 \_ corosync This process would have become /usr/lib64/heartbeat/stonithd, but fork() does not play well with threads. So if you must continue using the plugin, at least use ver: 1 in combination with the mcp: http://blog.clusterlabs.org/blog/2010/introducing-the-pacemaker-master-control-process-for-corosync-based-clusters/ But on RHEL6/CentOS6 you are far better of NOT using the plugin: http://blog.clusterlabs.org/blog/2013/pacemaker-on-rhel6-dot-4/ 499 6549 0.0 0.0 92300 5032 ?S19:06 0:00 \_ /usr/lib64/heartbeat/cib root 6550 0.0 0.0 74008 2384 ?S19:06 0:00 \_ /usr/lib64/heartbeat/lrmd 499 6551 0.0 0.0 94656 2600 ?S19:06 0:00 \_ /usr/lib64/heartbeat/attrd 499 6552 0.0 0.0 87560 3828 ?S19:06 0:00 \_ /usr/lib64/heartbeat/pengine 499 6553 0.0 0.0 95352 3816 ?S19:06 0:00 \_ /usr/lib64/heartbeat/crmd Then,I open the debug,and find some information like that: Vi /var/log/cluster/corosync.log: Jul 25 03:17:01 2U_222 crmd: [16867]: debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/st_command Jul 25 03:17:01 2U_222 crmd: [16867]: debug: init_client_ipc_comms_nodispatch: Could not init comms on: /var/run/crm/st_command Jul 25 03:17:01 2U_222 crmd: [16867]: debug: stonith_api_signon: Connection to command channel failed Jul 25 03:17:01 2U_222 crmd: [16867]: debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/st_callback Jul 25 03:17:01 2U_222 crmd: [16867]: debug: init_client_ipc_comms_nodispatch: Could not init comms on: /var/run/crm/st_callback Jul 25 03:17:01 2U_222 crmd: [16867]: debug: stonith_api_signon: Connection to callback channel failed Jul 25 03:17:01 2U_222 crmd: [16867]: debug: stonith_api_signon: Connection to STONITH failed: Not connected Jul 25 03:17:01 2U_222 crmd: [16867]: debug: stonith_api_signoff: Signing out of the STONITH Service Jul 25 03:17:01 2U_222 crmd: [16867]: ERROR: te_connect_stonith: Sign-in failed: triggered a retry Jul 25 03:17:01 2U_222 crmd: [16867]: info: te_connect_stonith: Attempting connection to fencing daemon... And then,I find the socket st_command can not init correct. but, there is no log about socket init failed error. Could you help me about this? Thank you very much. best wishes, claire huang Email: claire.hu...@utstarcom.cn ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] Speeding up failover
We have a master-slave setup for Redis, running 6 instances of Redis on each physical host, and one floating IP between them. Each redis instance is part of a single group. When we fail over the IP in production, I'm observing this sequence of events: Pacemaker brings down the floating IP Pacemaker demotes the master redis instance Pacemaker stops each running redis process in sequence (essentially stopping the group) Pacemaker promotes the slave Pacemaker brings up the floating IP on the former slave (This follows documented behaviour as I understand it, see http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg05344.html for someone else with a similar problem). Under production traffic load, each redis process takes about 4 to 5 seconds to sync to disk and cleanup. This means that a simple failover takes between 24 and 30 seconds, which is a bit too long for us. Acceptable failover times would be less than 5 seconds (the lower the better). Is there a configuration option to change the failover process to *not* stop the group before promoting the secondary? Alternatively, suggestions on how to get pacemaker to manage only the state of the redis process but not the process itself are welcome (A process failure can be diagnosed by monitoring the response or lack thereof from redis itself, so a dead or non responding process can be treated alike as far as monitoring it goes). Devdas Bhagat ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] Two-Nodes Cluster fencing : Best Practices
Hi, I've just made a two-nodes Active/Passive cluster to have an iSCSI Failover SAN. Some details about my configuration : - I've two nodes with 2 bonds : 1 for DRBD replication and 1 for communication - iSCSI Target, iSCSI Lun and VirtualIP are constraints together to start on Master DRBD node All work fine, but now, I need to configure fencing. I've 2 DELL PowerEdge servers with iDRAC6. First question, is 'external/drac5' compatible with iDrac6 (I've read all and nothing about this...) ? Second question, is that configuration sufficient (with ipmi) ? configure primitive pStN1 stonith:external/ipmi hostname=node1 ipaddr=192.168.100.1 userid=ipmiuser passwd=ipmipwd interface=lan configure primitive pStN2 stonith:external/ipmi hostname=node2 ipaddr=192.168.100.2 userid=ipmiuser passwd=ipmipwd interface=lan location lStN1 pStN1 inf: node1 location lStN2 pStN2 inf: node2 And after all : configure property stonith-enabled=true configure property stonith-action=poweroff Third (and last) question, what about quorum ? At the moment I've 'no-quorum-policy=ignore' but it's a risk isn't it ? Don't hesitate to request me for more information if needed, Regards, Bruno. -- Bruno MACADRE --- Ingénieur Systèmes et Réseau | Systems and Network Engineer Département Informatique | Department of computer science Responsable Info SER | SER IT Manager Université de Rouen | University of Rouen --- Coordonnées / Contact : Université de Rouen Faculté des Sciences et Techniques - Madrillet Avenue de l'Université - BP12 76801 St Etienne du Rouvray CEDEX FRANCE Tél : +33 (0)2-32-95-51-86 --- ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Question about the behavior when a pacemaker's process crashed
(13.07.25 11:00), Andrew Beekhof wrote: On 24/07/2013, at 7:40 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.18 19:23), Andrew Beekhof wrote: On 17/07/2013, at 6:53 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.16 21:18), Andrew Beekhof wrote: On 16/07/2013, at 7:04 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.15 11:00), Andrew Beekhof wrote: On 12/07/2013, at 6:28 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: Hi, I'm using pacemaker-1.1.10. When a pacemaker's process crashed, the node is sometimes fenced or is not sometimes fenced. Is this the assumed behavior? Yes. Sometimes the dev1 respawns the processes fast enough that dev2 gets the hey, i'm back notification before the PE gets run and fencing can be initiated. In such cases, there is nothing to be gained from fencing - dev1 is reachable and responding. OK... but I want pacemaker to certainly perform either behavior (fence is performed or fence is not performed), since operation is troublesome. I think that it is better if user can specify behavior as an option. This makes no sense. Sorry. It is wrong to induce more downtime than absolutely necessary just to make a test pass. If careful of the increase in downtime, isn't it better to prevent fencing, in this case? With hindsight, yes. But we have no way of knowing at the time. If you want pacemaker to wait some time for it to come back, you can set crmd-transition-delay which will achieve the same thing it does for attrd. I think that only a little is suitable for my demand because crmd-transition-delay is delay. The only alternative to a delay, either by crmd-transition-delay or some other means, is that the crmd predicts the future. Because pacemakerd respawns a broken child process, so the cluster will return to a online state. If so, does subsequent fencing not increase a downtime? Yes, but only we know that because we have more knowledge than the cluster. Is it because stack is corosync? No. In pacemaker-1.0 with heartbeat, behavior when a child process crashed can be specified by ha.cf. - when specified 'pacemaker respawn', the cluster will recover to online. The node may still end up being fenced even with pacemaker respawn. If the node does not recover fast enough, relative to the some process died notification, then the node will get fenced. If the hey the process is back again notification gets held up due to network congestion, then the node will get fenced. Like most things in clustering, timing is hugely significant - consider a resource that fails just before vs. just after a monitor action is run Now it could be that heartbeat is consistently slow sending out the some process died notification (I recall it does not send them at all sometimes), but that would be a bug not a feature. Sorry, I mistook it. You're right. - when specified 'pacemaker on', the node will reboot by oneself. by oneself? Not because the other side fences it? Yes, by oneself. [14:34:25 root@vm3 ~]$ gdb /usr/lib64/heartbeat/heartbeat 9876 : [14:35:33 root@vm3 ~]$ pkill -9 crmd : (gdb) b cl_reboot Breakpoint 2 at 0x7f0e433bdcf8 (gdb) c Continuing. Breakpoint 2, 0x7f0e433bdcf8 in cl_reboot () from /usr/lib64/libplumb.so.2 (gdb) bt #0 0x7f0e433bdcf8 in cl_reboot () from /usr/lib64/libplumb.so.2 #1 0x0040d8e4 in ManagedChildDied (p=0x117f6e0, status=value optimized out, signo=9, exitcode=0, waslogged=1) at heartbeat.c:3906 #2 0x7f0e433c8fcf in ReportProcHasDied () from /usr/lib64/libplumb.so.2 #3 0x7f0e433c140c in ?? () from /usr/lib64/libplumb.so.2 #4 0x7f0e433c0fe0 in ?? () from /usr/lib64/libplumb.so.2 #5 0x003240c38f0e in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #6 0x003240c3c938 in ?? () from /lib64/libglib-2.0.so.0 #7 0x003240c3cd55 in g_main_loop_run () from /lib64/libglib-2.0.so.0 #8 0x0040e8b8 in master_control_process () at heartbeat.c:1650 #9 initialize_heartbeat () at heartbeat.c:1041 #10 0x0040f38d in main (argc=value optimized out, argv=value optimized out, envp= 0x7fffe0ba9bd8) at heartbeat.c:5133 (gdb) n Message from syslogd@vm3 at Jul 25 14:36:57 ... heartbeat: [9876]: EMERG: Rebooting system. Reason: /usr/lib64/heartbeat/crmd I want to perform a setup and operation (established practice) equivalent to it. This is a patch to add the option which can choose to reboot a machine at the time of child process failure. https://github.com/inouekazu/pacemaker/commit/c1ac1048d8 What do you think? Best regards. It makes writing CTS tests hard, but it is not incorrect. procedure: $ systemctl start pacemaker $ crm configure load update test.cli $ pkill -9 lrmd attachment: STONITH.tar.bz2 : it's crm_report when fenced notSTONITH.tar.bz2 : it's crm_report when not fenced Best regards. notSTONITH.tar.bz2STONITH.tar.bz2___
Re: [Pacemaker] Two-Nodes Cluster fencing : Best Practices
Some modifications about my first mail : After some researches I found that external/ipmi isn't available on my system, so I must use fence-agents. My second question must be modified to relfect this changes like this : configure primitive pStN1 stonith:fence_ipmilan params ipaddr=192.168.100.1 login=ipmiuser passwd=ipmipwd configure primitive pStN2 stonith:fence_ipmilan params ipaddr=192.168.100.2 login=ipmiuser passwd=ipmipwd Regards, Bruno Le 25/07/2013 10:39, Bruno MACADRÉ a écrit : Hi, I've just made a two-nodes Active/Passive cluster to have an iSCSI Failover SAN. Some details about my configuration : - I've two nodes with 2 bonds : 1 for DRBD replication and 1 for communication - iSCSI Target, iSCSI Lun and VirtualIP are constraints together to start on Master DRBD node All work fine, but now, I need to configure fencing. I've 2 DELL PowerEdge servers with iDRAC6. First question, is 'external/drac5' compatible with iDrac6 (I've read all and nothing about this...) ? Second question, is that configuration sufficient (with ipmi) ? configure primitive pStN1 stonith:external/ipmi hostname=node1 ipaddr=192.168.100.1 userid=ipmiuser passwd=ipmipwd interface=lan configure primitive pStN2 stonith:external/ipmi hostname=node2 ipaddr=192.168.100.2 userid=ipmiuser passwd=ipmipwd interface=lan location lStN1 pStN1 inf: node1 location lStN2 pStN2 inf: node2 And after all : configure property stonith-enabled=true configure property stonith-action=poweroff Third (and last) question, what about quorum ? At the moment I've 'no-quorum-policy=ignore' but it's a risk isn't it ? Don't hesitate to request me for more information if needed, Regards, Bruno. -- Bruno MACADRE --- Ingénieur Systèmes et Réseau | Systems and Network Engineer Département Informatique | Department of computer science Responsable Info SER | SER IT Manager Université de Rouen | University of Rouen --- Coordonnées / Contact : Université de Rouen Faculté des Sciences et Techniques - Madrillet Avenue de l'Université - BP12 76801 St Etienne du Rouvray CEDEX FRANCE Tél : +33 (0)2-32-95-51-86 --- ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] ifdown ethX + corosync + DRBD = split-brain?
19.07.2013 14:38, Howley, Tom wrote: Hi, I have been doing some testing of a fairly standard pacemaker/corosync setup with DRBD (with resource-level fencing) and have noticed the following in relation to testing network failures: - Handling of all ports being blocked is OK, based on hundreds of tests. - Handling of cable-pulls seems OK, based on only 10 tests. - ifdown ethX leads to split-brain roughly 50% of the time due to two underlying issues: 1. corosync (possibly by design) handles loss of network interface differently to other network failures. I can only see this from the point of view of the logs: [TOTEM ] The network interface is down., which is different from cable-pull log, where I don't see that message. I'm guessing this as I don't know the code. 2. corosync allows a non-quorate partition, in my case a single node, to update the CIB. This behaviour has been previously confirmed in reply to previous mails on this list and it has been mentioned that there may be improvements in this area in the future. This on its own seems like a bug to me. My question is: is it possible for me to configure corosync/drbd to handle the ifdown scenario or do I simply have to tell people do not test with ifdown, as I have seen mentioned in a few places on the web? If I do have to leave out ifdown testing, how can I be sure that I haven't missed out testing some real network failure scenario. When you shut down an interface, IP is removed. As a result, DRBD can not bind to IP. In real life, it's not going to happen. So just tell people do not test with ifdown. -- WBR, Viacheslav Dubrovskyi ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] order required if group is present?
Hi List, i have 5 resources configured (p_bond1, p_conntrackd, p_vlan118,p_vlan119, p_openvpn) additionally I have put all of them in a group with: group cluster1 p_bond1,p_vlan118,p_vlan119,p_openvpn,p_conntrackd By this, crm is starting the resources in the order, the group is defined (p_bond1,p_vlan118 and so on...) Is this an expected behavior? If so, it's providing the function `order` was made for? Thanks in advance Stefan ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Node recover causes resource to migrate
Hello, As suggested I'm trying to add the nodes via corosync-objctl. My current config file is this one: https://gist.github.com/therobot/4327cd0a2598d1d6bb93 using 5001 as the nodeid on the second node. Then I try to add the nodes with the following commands: corosync-objctl -n totem.interface.member.memberaddr=10.34.191.212 corosync-objctl -n runtime.totem.pg.mrp.srp.members.5001.join_count=1 corosync-objctl -n runtime.totem.pg.mrp.srp.members.5000.status=joined corosync-objctl -n untime.totem.pg.mrp.srp.members.5001.ip=r(0) ip(10.34.191.212) (I have a problem because the space between r(0) and ip is not respected by corosync-objctl) Am I on the right track? Should I abandon my quest to have an HA solution on EC2 public network? Thanks, Jacobo García López de Araujo http://thebourbaki.com | http://twitter.com/clapkent On Wed, Jul 24, 2013 at 2:43 PM, Andrew Beekhof and...@beekhof.net wrote: On 24/07/2013, at 10:09 PM, Lars Marowsky-Bree l...@suse.com wrote: On 2013-07-24T21:40:40, Andrew Beekhof and...@beekhof.net wrote: Statically assigned nodeids? Wouldn't hurt, but you still need to bring down the still-active node to get it to talk to the new node. Which sucks Hm. But ... corosync/pacemaker ought to identify the node via the nodeid. If it comes back with a different IP address, that shouldn't be a problem. Oh. *thud* Just realized that it's bound to be one for unicast communications, not so much mcast. Exactly. Seems we may need some corosync magic commands to edit the nodelist at runtime. (Or is that already possible and I just don't know how? ;-) I believe it might be possible - I just don't know it. Might even be better to have it happen automagically - after-all the new node knows the existing node's address. But good luck getting that one through. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Two-Nodes Cluster fencing : Best Practices
With two-node clusters, quorum can't be used. This is fine *if* you have good fencing. If the nodes partition (ie: network failure), both will try to fence the other. In theory, the faster node will power off the other node before the slower node can kill the faster node. In practice, this isn't always the case. IPMI (and iDRAC, etc) are independent devices. So it is possible for both nodes to initiate a power-down on the other before either dies. To avoid this, you will want to set a delay for the primary/active node's fence primitive. Say node1 is your active node and node2 is your backup. You would set a delay of, say, 15 seconds against node1. Now if there is a partition, node1 would look up how to fence node2 and immediately initiate power off. Node 2, however, would look up how to fence node1, see a 15 second delay, and start a timer before calling the power-off. Of course, node2 will die before the timer expires. You can also disabled acpid on the nodes, too. With that disabled, pressing the power button will result in a near-instant off. If you do this, reducing your delay to 5 seconds would probably be plenty. There is another issue to be aware of; Fence loops. The problem with two node clusters and not using quorum is that a single node can fence the other. So lets continue our example above... Node 2 will eventually reboot. If you have pacemaker set to start on boot, it will start, wait to connect to node1 (which it can't because the network failure remains), call a fence to put node1 into a known state, pause for 15 seconds and then initiate a power off. Node 1 dies and the services recover on Node 2. Now, node1 boots back up, starts it's pacemaker Endless loop of fence - recover until the network is fixed. To avoid this, simple do not start pacemaker on boot. As to the specifics, you can test fencing configurations easily by directly calling the fence agent at the command line. I do not use DRAC, so I can't speak to specifics. I think you need to set lanplus and possibly define the console prompt to expect. Using a generic IPMI as an example; fence_ipmilan -a 192.168.100.1 -l ipmiuser -p ipmipwd -o status fence_ipmilan -a 192.168.100.2 -l ipmiuser -p ipmipwd -o status If this returns the power state, then it is simple to convert to a pacemaker config. configure primitive pStN1 stonith:fence_ipmilan params \ ipaddr=192.168.100.1 login=ipmiuser passwd=ipmipwd delay=15 \ op monitor interval=60s configure primitive pStN2 stonith:fence_ipmilan params \ ipaddr=192.168.100.2 login=ipmiuser passwd=ipmipwd \ op monitor interval=60s Again, I *think* you need to set a couple extra options for DRAC. Experiment at the command line before moving to the pacemaker config. Once you have the command line version working, you should be able to set it up in pacemaker. If you have trouble though, share the CLI call and we can help with the pacemaker config. On 25/07/13 05:39, Bruno MACADRÉ wrote: Some modifications about my first mail : After some researches I found that external/ipmi isn't available on my system, so I must use fence-agents. My second question must be modified to relfect this changes like this : configure primitive pStN1 stonith:fence_ipmilan params ipaddr=192.168.100.1 login=ipmiuser passwd=ipmipwd configure primitive pStN2 stonith:fence_ipmilan params ipaddr=192.168.100.2 login=ipmiuser passwd=ipmipwd Regards, Bruno Le 25/07/2013 10:39, Bruno MACADRÉ a écrit : Hi, I've just made a two-nodes Active/Passive cluster to have an iSCSI Failover SAN. Some details about my configuration : - I've two nodes with 2 bonds : 1 for DRBD replication and 1 for communication - iSCSI Target, iSCSI Lun and VirtualIP are constraints together to start on Master DRBD node All work fine, but now, I need to configure fencing. I've 2 DELL PowerEdge servers with iDRAC6. First question, is 'external/drac5' compatible with iDrac6 (I've read all and nothing about this...) ? Second question, is that configuration sufficient (with ipmi) ? configure primitive pStN1 stonith:external/ipmi hostname=node1 ipaddr=192.168.100.1 userid=ipmiuser passwd=ipmipwd interface=lan configure primitive pStN2 stonith:external/ipmi hostname=node2 ipaddr=192.168.100.2 userid=ipmiuser passwd=ipmipwd interface=lan location lStN1 pStN1 inf: node1 location lStN2 pStN2 inf: node2 And after all : configure property stonith-enabled=true configure property stonith-action=poweroff Third (and last) question, what about quorum ? At the moment I've 'no-quorum-policy=ignore' but it's a risk isn't it ? Don't hesitate to request me for more information if needed, Regards, Bruno. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education?
Re: [Pacemaker] order required if group is present?
Hi Stefan, a) yes, the ordered behaviour is intentional. b) In former version you could change this behaviour with an attribute. But this attribute is depreciated in newer versions of pacemaker. c) The solution for parallel starting resources are resource sets. Best regards Andreas Mock P.S.: Always give information about used versions of elements of the cluster stack. Behaviour changed over time. Von: Bauer, Stefan (IZLBW Extern) [mailto:stefan.ba...@iz.bwl.de] Gesendet: Donnerstag, 25. Juli 2013 12:53 An: Pacemaker@oss.clusterlabs.org Betreff: [Pacemaker] order required if group is present? Hi List, i have 5 resources configured (p_bond1, p_conntrackd, p_vlan118,p_vlan119, p_openvpn) additionally I have put all of them in a group with: group cluster1 p_bond1,p_vlan118,p_vlan119,p_openvpn,p_conntrackd By this, crm is starting the resources in the order, the group is defined (p_bond1,p_vlan118 and so on.) Is this an expected behavior? If so, it's providing the function `order` was made for? Thanks in advance Stefan ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Two-Nodes Cluster fencing : Best Practices
- Original Message - From: Digimer li...@alteeve.ca To: The Pacemaker cluster resource manager pacemaker@oss.clusterlabs.org Sent: Thursday, July 25, 2013 10:53:27 AM Subject: Re: [Pacemaker] Two-Nodes Cluster fencing : Best Practices With two-node clusters, quorum can't be used. This is fine *if* you have good fencing. If the nodes partition (ie: network failure), both will try to fence the other. In theory, the faster node will power off the other node before the slower node can kill the faster node. In practice, this isn't always the case. IPMI (and iDRAC, etc) are independent devices. So it is possible for both nodes to initiate a power-down on the other before either dies. To avoid this, you will want to set a delay for the primary/active node's fence primitive. Say node1 is your active node and node2 is your backup. You would set a delay of, say, 15 seconds against node1. Now if there is a partition, node1 would look up how to fence node2 and immediately initiate power off. Node 2, however, would look up how to fence node1, see a 15 second delay, and start a timer before calling the power-off. Of course, node2 will die before the timer expires. You can also disabled acpid on the nodes, too. With that disabled, pressing the power button will result in a near-instant off. If you do this, reducing your delay to 5 seconds would probably be plenty. There is another issue to be aware of; Fence loops. The problem with two node clusters and not using quorum is that a single node can fence the other. So lets continue our example above... Node 2 will eventually reboot. If you have pacemaker set to start on boot, it will start, wait to connect to node1 (which it can't because the network failure remains), call a fence to put node1 into a known state, pause for 15 seconds and then initiate a power off. Node 1 dies and the services recover on Node 2. Now, node1 boots back up, starts it's pacemaker Endless loop of fence - recover until the network is fixed. To avoid this, simple do not start pacemaker on boot. As to the specifics, you can test fencing configurations easily by directly calling the fence agent at the command line. I do not use DRAC, so I can't speak to specifics. I think you need to set lanplus and possibly define the console prompt to expect. Using a generic IPMI as an example; fence_ipmilan -a 192.168.100.1 -l ipmiuser -p ipmipwd -o status fence_ipmilan -a 192.168.100.2 -l ipmiuser -p ipmipwd -o status If this returns the power state, then it is simple to convert to a pacemaker config. configure primitive pStN1 stonith:fence_ipmilan params \ ipaddr=192.168.100.1 login=ipmiuser passwd=ipmipwd delay=15 \ op monitor interval=60s configure primitive pStN2 stonith:fence_ipmilan params \ ipaddr=192.168.100.2 login=ipmiuser passwd=ipmipwd \ op monitor interval=60s Again, I *think* you need to set a couple extra options for DRAC. Experiment at the command line before moving to the pacemaker config. Once you have the command line version working, you should be able to set it up in pacemaker. If you have trouble though, share the CLI call and we can help with the pacemaker config. I use external/ipmi with my iDRACs (5's and 6's) with the following pacemaker config: primitive p_ipmilan_condor stonith:external/ipmi \ params hostname=Condor ipaddr=192.168.x.x userid=root passwd=XX \ The iDRAC needs the following settings for this to work: IPMI over LAN – ON Security setup – root as the user, set the BMC/iDRAC password Sounds like you will need to convert to a provided fence agent but hopefully this helps some. HTH Jake On 25/07/13 05:39, Bruno MACADRÉ wrote: Some modifications about my first mail : After some researches I found that external/ipmi isn't available on my system, so I must use fence-agents. My second question must be modified to relfect this changes like this : configure primitive pStN1 stonith:fence_ipmilan params ipaddr=192.168.100.1 login=ipmiuser passwd=ipmipwd configure primitive pStN2 stonith:fence_ipmilan params ipaddr=192.168.100.2 login=ipmiuser passwd=ipmipwd Regards, Bruno Le 25/07/2013 10:39, Bruno MACADRÉ a écrit : Hi, I've just made a two-nodes Active/Passive cluster to have an iSCSI Failover SAN. Some details about my configuration : - I've two nodes with 2 bonds : 1 for DRBD replication and 1 for communication - iSCSI Target, iSCSI Lun and VirtualIP are constraints together to start on Master DRBD node All work fine, but now, I need to configure fencing. I've 2 DELL PowerEdge servers with iDRAC6. First question, is 'external/drac5' compatible with iDrac6 (I've read all and nothing about this...) ? Second question, is that configuration sufficient (with ipmi) ?
Re: [Pacemaker] order required if group is present?
On 26/07/2013, at 12:59 AM, Andreas Mock andreas.m...@web.de wrote: Hi Stefan, a) yes, the ordered behaviour is intentional. b) In former version you could change this behaviour with an attribute. But this attribute is depreciated in newer versions of pacemaker. c) The solution for parallel starting resources are resource sets. d) groups are essentially a shortcut for a colocation and ordering constraints. Best regards Andreas Mock P.S.: Always give information about used versions of elements of the cluster stack. Behaviour changed over time. Von: Bauer, Stefan (IZLBW Extern) [mailto:stefan.ba...@iz.bwl.de] Gesendet: Donnerstag, 25. Juli 2013 12:53 An: Pacemaker@oss.clusterlabs.org Betreff: [Pacemaker] order required if group is present? Hi List, i have 5 resources configured (p_bond1, p_conntrackd, p_vlan118,p_vlan119, p_openvpn) additionally I have put all of them in a group with: group cluster1 p_bond1,p_vlan118,p_vlan119,p_openvpn,p_conntrackd By this, crm is starting the resources in the order, the group is defined (p_bond1,p_vlan118 and so on…) Is this an expected behavior? If so, it’s providing the function `order` was made for? Thanks in advance Stefan ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] Announce: Pacemaker 1.1.10 now available
Announcing the release of Pacemaker 1.1.10 https://github.com/ClusterLabs/pacemaker/releases/Pacemaker-1.1.10 There were three changes of note since rc7: + Bug cl#5161 - crmd: Prevent memory leak in operation cache + cib: Correctly read back archived configurations if the primary is corrupted + cman: Do not pretend we know the state of nodes we've never seen Along with assorted bug fixes, the major topics for this release were: - stonithd fixes - fixing memory leaks, often caused by incorrect use of glib reference counting - supportability improvements (code cleanup and deduplication, standardized error codes) Release candidates for the next Pacemaker release (1.1.11) can be expected some time around Novemeber. A big thankyou to everyone that spent time testing the release candidates and/or contributed patches. However now that Pacemaker is perfect, anyone reporting bugs will be shot :-) To build `rpm` packages: 1. Clone the current sources: # git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git # cd pacemaker 1. Install dependancies (if you haven't already) [Fedora] # sudo yum install -y yum-utils [ALL]# make rpm-dep 1. Build Pacemaker # make release 1. Copy and deploy as needed ## Details - 1.1.10 - final Changesets: 602 Diff: 143 files changed, 8162 insertions(+), 5159 deletions(-) ## Highlights ### Features added since Pacemaker-1.1.9 + Core: Convert all exit codes to positive errno values + crm_error: Add the ability to list and print error symbols + crm_resource: Allow individual resources to be reprobed + crm_resource: Allow options to be set recursively + crm_resource: Implement --ban for moving resources away from nodes and --clear (replaces --unmove) + crm_resource: Support OCF tracing when using --force-(check|start|stop) + PE: Allow active nodes in our current membership to be fenced without quorum + PE: Suppress meaningless IDs when displaying anonymous clone status + Turn off auto-respawning of systemd services when the cluster starts them + Bug cl#5128 - pengine: Support maintenance mode for a single node ### Changes since Pacemaker-1.1.9 + crmd: cib: stonithd: Memory leaks resolved and improved use of glib reference counting + attrd: Fixes deleted attributes during dc election + Bug cf#5153 - Correctly display clone failcounts in crm_mon + Bug cl#5133 - pengine: Correctly observe on-fail=block for failed demote operation + Bug cl#5148 - legacy: Correctly remove a node that used to have a different nodeid + Bug cl#5151 - Ensure node names are consistently compared without case + Bug cl#5152 - crmd: Correctly clean up fenced nodes during membership changes + Bug cl#5154 - Do not expire failures when on-fail=block is present + Bug cl#5155 - pengine: Block the stop of resources if any depending resource is unmanaged + Bug cl#5157 - Allow migration in the absence of some colocation constraints + Bug cl#5161 - crmd: Prevent memory leak in operation cache + Bug cl#5164 - crmd: Fixes crash when using pacemaker-remote + Bug cl#5164 - pengine: Fixes segfault when calculating transition with remote-nodes. + Bug cl#5167 - crm_mon: Only print stopped node list for incomplete clone sets + Bug cl#5168 - Prevent clones from being bounced around the cluster due to location constraints + Bug cl#5170 - Correctly support on-fail=block for clones + cib: Correctly read back archived configurations if the primary is corrupted + cib: The result is not valid when diffs fail to apply cleanly for CLI tools + cib: Restore the ability to embed comments in the configuration + cluster: Detect and warn about node names with capitals + cman: Do not pretend we know the state of nodes we've never seen + cman: Do not unconditionally start cman if it is already running + cman: Support non-blocking CPG calls + Core: Ensure the blackbox is saved on abnormal program termination + corosync: Detect the loss of members for which we only know the nodeid + corosync: Do not pretend we know the state of nodes we've never seen + corosync: Ensure removed peers are erased from all caches + corosync: Nodes that can persist in sending CPG messages must be alive afterall + crmd: Do not get stuck in S_POLICY_ENGINE if a node we couldn't fence returns + crmd: Do not update fail-count and last-failure for old failures + crmd: Ensure all membership operations can complete while trying to cancel a transition + crmd: Ensure operations for cleaned up resources don't block recovery + crmd: Ensure we return to a stable state if there have been too many fencing failures + crmd: Initiate node shutdown if another node claims to have successfully fenced us + crmd: Prevent messages for remote crmd clients from being relayed to wrong daemons + crmd: Properly handle recurring monitor operations for remote-node agent + crmd: Store last-run and last-rc-change for
Re: [Pacemaker] How to perform a clean shutdown of Pacemaker in the event of network connection loss
Ops sorry for providing the wrong version number. I am using Corosync 2.3.0-1 and Pacemaker 1.1.9-8. What should be done to clear the resources properly that are still running in the event that pacemaker isn't running? On Thu, Jul 25, 2013 at 10:39 AM, Andrew Beekhof and...@beekhof.net wrote: On 24/07/2013, at 6:40 PM, Tan Tai hock taih...@gmail.com wrote: I did not enable fencing. I observe the process running and see that when the node is up, I will see the following processes: Corosync -- /usr/sbin/corosync Pacemaker - /usr/libexec/pacemaker/lrmd /usr/libexec/pacemaker/pengine pacemakerd /usr/libexec/pacemaker/stonith /usr/libexec/pacemaker/cib /usr/libexec/pacemaker/crmd If I were to shutdown the network connection of any node and then list out the processes, I will see that /usr/sbin/corosync is no longer running and for Pacemaker, the following processes are left: Pacemaker - /usr/libexec/pacemaker/lrmd /usr/libexec/pacemaker/pengine corosync has probably crashed and taken most of pacemaker with it (all this bits that connect to corosync). what versions are you running? because neither Pacemaker 2.3 nor Corosync 1.19 exist. In any case, the resources are still running there - even if pacemaker isn't. If you want to simulate an outage, use firewall rules to block traffic to/from the node. If you want to simulate someone opening the box and pulling out it's NIC - keep using ifconfig down. If there is no network connectivity loss and I perform a clean shutdown, I do not see any of the processes listed for Corosync and Pacemaker. I tried to kill the remaining process after network connection is lost but that does not prevent the fallen node from getting back the resource if it used to be holding it before going down. Is there a way to perform a clean shutdown if pacemaker was shutdown improperly? On Wed, Jul 24, 2013 at 8:21 AM, Andrew Beekhof and...@beekhof.net wrote: On 24/07/2013, at 9:54 AM, Tan Tai hock taih...@gmail.com wrote: No I did not. It seems like corosync and pacemaker stop running when the network connection is lost. Do you have fencing enabled? If not, I'd be surprised if corosync or pacemaker stopped running. I am trying to simulate a scenario whereby a node which started the resource loses network connection and observe how it reacts upon joining back the cluster. Is there any proper way to shutdown both corosync and pacemaker in such scenario? They are not supposed to stop running just because connectivity was lost. On Jul 24, 2013 6:55 AM, Andrew Beekhof and...@beekhof.net wrote: On 23/07/2013, at 11:28 AM, Tan Tai hock taih...@gmail.com wrote: Hi, I have currently set up 3 machines with Pacemaker 2.3 with Corosync 1.19. I have tested some scenarios and have encountered some problem which I hope to get some advice on. My scenario is as follows: The 3 machines, name A,B,C are all running with A being the node which started the resource as seen in cm_mon. If I were to cut off the network connection for A, B will take over as the node which started the resource. I then resume the network connection and start both corosync and pacemaker on A again Did you stop it there first? and the node which started the resource now returns to node A. I have set stickness and perform an identical test but with proper shutdown of pacemaker and corosync and it is working fine. Is there anyway to perform a clean shutdown in the event that a node loses network connection so that it will not attempt to take back the resource it used to be holding before it was uncleanly shutdown? Thanks ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
Congrats!! I know this was a long time in the making. digimer On 25/07/13 20:43, Andrew Beekhof wrote: Announcing the release of Pacemaker 1.1.10 https://github.com/ClusterLabs/pacemaker/releases/Pacemaker-1.1.10 There were three changes of note since rc7: + Bug cl#5161 - crmd: Prevent memory leak in operation cache + cib: Correctly read back archived configurations if the primary is corrupted + cman: Do not pretend we know the state of nodes we've never seen Along with assorted bug fixes, the major topics for this release were: - stonithd fixes - fixing memory leaks, often caused by incorrect use of glib reference counting - supportability improvements (code cleanup and deduplication, standardized error codes) Release candidates for the next Pacemaker release (1.1.11) can be expected some time around Novemeber. A big thankyou to everyone that spent time testing the release candidates and/or contributed patches. However now that Pacemaker is perfect, anyone reporting bugs will be shot :-) To build `rpm` packages: 1. Clone the current sources: # git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git # cd pacemaker 1. Install dependancies (if you haven't already) [Fedora] # sudo yum install -y yum-utils [ALL] # make rpm-dep 1. Build Pacemaker # make release 1. Copy and deploy as needed ## Details - 1.1.10 - final Changesets: 602 Diff: 143 files changed, 8162 insertions(+), 5159 deletions(-) ## Highlights ### Features added since Pacemaker-1.1.9 + Core: Convert all exit codes to positive errno values + crm_error: Add the ability to list and print error symbols + crm_resource: Allow individual resources to be reprobed + crm_resource: Allow options to be set recursively + crm_resource: Implement --ban for moving resources away from nodes and --clear (replaces --unmove) + crm_resource: Support OCF tracing when using --force-(check|start|stop) + PE: Allow active nodes in our current membership to be fenced without quorum + PE: Suppress meaningless IDs when displaying anonymous clone status + Turn off auto-respawning of systemd services when the cluster starts them + Bug cl#5128 - pengine: Support maintenance mode for a single node ### Changes since Pacemaker-1.1.9 + crmd: cib: stonithd: Memory leaks resolved and improved use of glib reference counting + attrd: Fixes deleted attributes during dc election + Bug cf#5153 - Correctly display clone failcounts in crm_mon + Bug cl#5133 - pengine: Correctly observe on-fail=block for failed demote operation + Bug cl#5148 - legacy: Correctly remove a node that used to have a different nodeid + Bug cl#5151 - Ensure node names are consistently compared without case + Bug cl#5152 - crmd: Correctly clean up fenced nodes during membership changes + Bug cl#5154 - Do not expire failures when on-fail=block is present + Bug cl#5155 - pengine: Block the stop of resources if any depending resource is unmanaged + Bug cl#5157 - Allow migration in the absence of some colocation constraints + Bug cl#5161 - crmd: Prevent memory leak in operation cache + Bug cl#5164 - crmd: Fixes crash when using pacemaker-remote + Bug cl#5164 - pengine: Fixes segfault when calculating transition with remote-nodes. + Bug cl#5167 - crm_mon: Only print stopped node list for incomplete clone sets + Bug cl#5168 - Prevent clones from being bounced around the cluster due to location constraints + Bug cl#5170 - Correctly support on-fail=block for clones + cib: Correctly read back archived configurations if the primary is corrupted + cib: The result is not valid when diffs fail to apply cleanly for CLI tools + cib: Restore the ability to embed comments in the configuration + cluster: Detect and warn about node names with capitals + cman: Do not pretend we know the state of nodes we've never seen + cman: Do not unconditionally start cman if it is already running + cman: Support non-blocking CPG calls + Core: Ensure the blackbox is saved on abnormal program termination + corosync: Detect the loss of members for which we only know the nodeid + corosync: Do not pretend we know the state of nodes we've never seen + corosync: Ensure removed peers are erased from all caches + corosync: Nodes that can persist in sending CPG messages must be alive afterall + crmd: Do not get stuck in S_POLICY_ENGINE if a node we couldn't fence returns + crmd: Do not update fail-count and last-failure for old failures + crmd: Ensure all membership operations can complete while trying to cancel a transition + crmd: Ensure operations for cleaned up resources don't block recovery + crmd: Ensure we return to a stable state if there have been too many fencing failures + crmd: Initiate node shutdown if another node claims to have successfully fenced us + crmd: Prevent messages for remote crmd clients
Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
Hi My report is late for 1.1.10 :( I am using pacemaker 1.1.10-0.1.ab2e209.git. It seems that master's monitor is stopped when slave is started. Does someone encounter same problem ? I attach a log and settings. Thanks, Takatoshi MATSUO 2013/7/26 Digimer li...@alteeve.ca: Congrats!! I know this was a long time in the making. digimer On 25/07/13 20:43, Andrew Beekhof wrote: Announcing the release of Pacemaker 1.1.10 https://github.com/ClusterLabs/pacemaker/releases/Pacemaker-1.1.10 There were three changes of note since rc7: + Bug cl#5161 - crmd: Prevent memory leak in operation cache + cib: Correctly read back archived configurations if the primary is corrupted + cman: Do not pretend we know the state of nodes we've never seen Along with assorted bug fixes, the major topics for this release were: - stonithd fixes - fixing memory leaks, often caused by incorrect use of glib reference counting - supportability improvements (code cleanup and deduplication, standardized error codes) Release candidates for the next Pacemaker release (1.1.11) can be expected some time around Novemeber. A big thankyou to everyone that spent time testing the release candidates and/or contributed patches. However now that Pacemaker is perfect, anyone reporting bugs will be shot :-) To build `rpm` packages: 1. Clone the current sources: # git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git # cd pacemaker 1. Install dependancies (if you haven't already) [Fedora] # sudo yum install -y yum-utils [ALL] # make rpm-dep 1. Build Pacemaker # make release 1. Copy and deploy as needed ## Details - 1.1.10 - final Changesets: 602 Diff: 143 files changed, 8162 insertions(+), 5159 deletions(-) ## Highlights ### Features added since Pacemaker-1.1.9 + Core: Convert all exit codes to positive errno values + crm_error: Add the ability to list and print error symbols + crm_resource: Allow individual resources to be reprobed + crm_resource: Allow options to be set recursively + crm_resource: Implement --ban for moving resources away from nodes and --clear (replaces --unmove) + crm_resource: Support OCF tracing when using --force-(check|start|stop) + PE: Allow active nodes in our current membership to be fenced without quorum + PE: Suppress meaningless IDs when displaying anonymous clone status + Turn off auto-respawning of systemd services when the cluster starts them + Bug cl#5128 - pengine: Support maintenance mode for a single node ### Changes since Pacemaker-1.1.9 + crmd: cib: stonithd: Memory leaks resolved and improved use of glib reference counting + attrd: Fixes deleted attributes during dc election + Bug cf#5153 - Correctly display clone failcounts in crm_mon + Bug cl#5133 - pengine: Correctly observe on-fail=block for failed demote operation + Bug cl#5148 - legacy: Correctly remove a node that used to have a different nodeid + Bug cl#5151 - Ensure node names are consistently compared without case + Bug cl#5152 - crmd: Correctly clean up fenced nodes during membership changes + Bug cl#5154 - Do not expire failures when on-fail=block is present + Bug cl#5155 - pengine: Block the stop of resources if any depending resource is unmanaged + Bug cl#5157 - Allow migration in the absence of some colocation constraints + Bug cl#5161 - crmd: Prevent memory leak in operation cache + Bug cl#5164 - crmd: Fixes crash when using pacemaker-remote + Bug cl#5164 - pengine: Fixes segfault when calculating transition with remote-nodes. + Bug cl#5167 - crm_mon: Only print stopped node list for incomplete clone sets + Bug cl#5168 - Prevent clones from being bounced around the cluster due to location constraints + Bug cl#5170 - Correctly support on-fail=block for clones + cib: Correctly read back archived configurations if the primary is corrupted + cib: The result is not valid when diffs fail to apply cleanly for CLI tools + cib: Restore the ability to embed comments in the configuration + cluster: Detect and warn about node names with capitals + cman: Do not pretend we know the state of nodes we've never seen + cman: Do not unconditionally start cman if it is already running + cman: Support non-blocking CPG calls + Core: Ensure the blackbox is saved on abnormal program termination + corosync: Detect the loss of members for which we only know the nodeid + corosync: Do not pretend we know the state of nodes we've never seen + corosync: Ensure removed peers are erased from all caches + corosync: Nodes that can persist in sending CPG messages must be alive afterall + crmd: Do not get stuck in S_POLICY_ENGINE if a node we couldn't fence returns + crmd: Do not update fail-count and last-failure for old failures + crmd: Ensure all membership
Re: [Pacemaker] Question about the behavior when a pacemaker's process crashed
(13.07.25 18:03), Kazunori INOUE wrote: (13.07.25 11:00), Andrew Beekhof wrote: On 24/07/2013, at 7:40 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.18 19:23), Andrew Beekhof wrote: On 17/07/2013, at 6:53 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.16 21:18), Andrew Beekhof wrote: On 16/07/2013, at 7:04 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: (13.07.15 11:00), Andrew Beekhof wrote: On 12/07/2013, at 6:28 PM, Kazunori INOUE inouek...@intellilink.co.jp wrote: Hi, I'm using pacemaker-1.1.10. When a pacemaker's process crashed, the node is sometimes fenced or is not sometimes fenced. Is this the assumed behavior? Yes. Sometimes the dev1 respawns the processes fast enough that dev2 gets the hey, i'm back notification before the PE gets run and fencing can be initiated. In such cases, there is nothing to be gained from fencing - dev1 is reachable and responding. OK... but I want pacemaker to certainly perform either behavior (fence is performed or fence is not performed), since operation is troublesome. I think that it is better if user can specify behavior as an option. This makes no sense. Sorry. It is wrong to induce more downtime than absolutely necessary just to make a test pass. If careful of the increase in downtime, isn't it better to prevent fencing, in this case? With hindsight, yes. But we have no way of knowing at the time. If you want pacemaker to wait some time for it to come back, you can set crmd-transition-delay which will achieve the same thing it does for attrd. I think that only a little is suitable for my demand because crmd-transition-delay is delay. The only alternative to a delay, either by crmd-transition-delay or some other means, is that the crmd predicts the future. Because pacemakerd respawns a broken child process, so the cluster will return to a online state. If so, does subsequent fencing not increase a downtime? Yes, but only we know that because we have more knowledge than the cluster. Is it because stack is corosync? No. In pacemaker-1.0 with heartbeat, behavior when a child process crashed can be specified by ha.cf. - when specified 'pacemaker respawn', the cluster will recover to online. The node may still end up being fenced even with pacemaker respawn. If the node does not recover fast enough, relative to the some process died notification, then the node will get fenced. If the hey the process is back again notification gets held up due to network congestion, then the node will get fenced. Like most things in clustering, timing is hugely significant - consider a resource that fails just before vs. just after a monitor action is run Now it could be that heartbeat is consistently slow sending out the some process died notification (I recall it does not send them at all sometimes), but that would be a bug not a feature. Sorry, I mistook it. You're right. - when specified 'pacemaker on', the node will reboot by oneself. by oneself? Not because the other side fences it? Yes, by oneself. [14:34:25 root@vm3 ~]$ gdb /usr/lib64/heartbeat/heartbeat 9876 : [14:35:33 root@vm3 ~]$ pkill -9 crmd : (gdb) b cl_reboot Breakpoint 2 at 0x7f0e433bdcf8 (gdb) c Continuing. Breakpoint 2, 0x7f0e433bdcf8 in cl_reboot () from /usr/lib64/libplumb.so.2 (gdb) bt #0 0x7f0e433bdcf8 in cl_reboot () from /usr/lib64/libplumb.so.2 #1 0x0040d8e4 in ManagedChildDied (p=0x117f6e0, status=value optimized out, signo=9, exitcode=0, waslogged=1) at heartbeat.c:3906 #2 0x7f0e433c8fcf in ReportProcHasDied () from /usr/lib64/libplumb.so.2 #3 0x7f0e433c140c in ?? () from /usr/lib64/libplumb.so.2 #4 0x7f0e433c0fe0 in ?? () from /usr/lib64/libplumb.so.2 #5 0x003240c38f0e in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #6 0x003240c3c938 in ?? () from /lib64/libglib-2.0.so.0 #7 0x003240c3cd55 in g_main_loop_run () from /lib64/libglib-2.0.so.0 #8 0x0040e8b8 in master_control_process () at heartbeat.c:1650 #9 initialize_heartbeat () at heartbeat.c:1041 #10 0x0040f38d in main (argc=value optimized out, argv=value optimized out, envp= 0x7fffe0ba9bd8) at heartbeat.c:5133 (gdb) n Message from syslogd@vm3 at Jul 25 14:36:57 ... heartbeat: [9876]: EMERG: Rebooting system. Reason: /usr/lib64/heartbeat/crmd I want to perform a setup and operation (established practice) equivalent to it. This is a patch to add the option which can choose to reboot a machine at the time of child process failure. https://github.com/inouekazu/pacemaker/commit/c1ac1048d8 oops! please refer to the commit here. https://github.com/inouekazu/pacemaker/commit/cbaf0e38ac What do you think? Best regards. It makes writing CTS tests hard, but it is not incorrect. procedure: $ systemctl start pacemaker $ crm configure load update test.cli $ pkill -9 lrmd attachment: STONITH.tar.bz2 : it's crm_report when fenced
Re: [Pacemaker] order required if group is present?
Hi together, thank you very much for pointing that out. stefan -Ursprüngliche Nachricht- Von: Andrew Beekhof [mailto:and...@beekhof.net] Gesendet: Freitag, 26. Juli 2013 01:24 An: The Pacemaker cluster resource manager Betreff: Re: [Pacemaker] order required if group is present? On 26/07/2013, at 12:59 AM, Andreas Mock andreas.m...@web.de wrote: Hi Stefan, a) yes, the ordered behaviour is intentional. b) In former version you could change this behaviour with an attribute. But this attribute is depreciated in newer versions of pacemaker. c) The solution for parallel starting resources are resource sets. d) groups are essentially a shortcut for a colocation and ordering constraints. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
