Re: [Pacemaker] Initial quorum
Devin Reade writes: > > --On Wednesday, July 20, 2011 09:19:33 AM + pskrap > wrote: > > > I have a cluster where some of the resources cannot run on the same node. > > All resources must be running to provide a functioning service. This > > means that a certain amount of nodes needs to be up before it makes > > sense for the cluster to start any resources. > > Without knowing anything about your application, I would tend to question > this statement. Is it true that you must not start *any* resources before > you have enough nodes, or is sufficient to say that the application > is not considered up until all resources are started? It may not > make sense to run any, but does it do any harm? > > If you *can* start at least some resources before all nodes are available, > then I would expect that you could get by with defining colocation > constraints to ensure that some resources don't run on the same nodes, > perhaps augmenting things with some order constraints if necessary. > > If your applications die or do other horrible stuff when only some subset > are running then I'd have a talk with your application developers > as it sounds like a larger robustness problem. > > Devin > No, there are no crash issues etc when all resources are not running. The application is just not usable until all resources are started. As for the harm, the resources which have constraints preventing them from running will fail, but I guess they will recover as more nodes are added. The harm is mostly in the fact that starting nodes one by one will cause the resources to be unevenly distributed over the nodes since everything will start on the nodes in the order they are installed. I know I can give a preferred node to a resource and allow it to relocate when it becomes available. However, this application provides a real-time service so I only want resources to relocate when it is absolutely necessary. Therefore I have given the resources a preferred node, but do not allow them to relocate when it becomes available. So I guess the overall harm is limited even though it exists. I was just looking for a cleaner startup for the system. Since you did not mention any way to do what my question was about I assume it is currently not possible to do what I asked for. I do think such an option would be useful though. Logically it does not make sense for the cluster to be starting resources for an application before the cluster have enough nodes for the application to be able to run. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
[Pacemaker] Fw: Fw: Configuration for FS over DRBD over LVM
Okay, this configuration works on one node (I am waiting for a hardware problem to be fixed before testing with second node): node cnode-1-3-5 node cnode-1-3-6 primitive glance-drbd ocf:linbit:drbd \ params drbd_resource="glance-repos-drbd" \ op start interval="0" timeout="240" \ op stop interval="0" timeout="100" \ op monitor interval="59s" role="Master" timeout="30s" \ op monitor interval="61s" role="Slave" timeout="30s" primitive glance-fs ocf:heartbeat:Filesystem \ params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \ op start interval="0" timeout="60" \ op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \ op stop interval="0" timeout="120" primitive glance-ip ocf:heartbeat:IPaddr2 \ params ip="10.4.0.25" nic="br100:1" \ op monitor interval="5s" primitive glance-repos ocf:heartbeat:LVM \ params volgrpname="glance-repos" exclusive="true" \ op start interval="0" timeout="30" \ op stop interval="0" timeout="30" group glance-repos-fs-group glance-fs glance-ip \ meta target-role="Started" ms ms_drbd glance-drbd \ meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master" colocation coloc-rule-w-master inf: ms_drbd:Master glance-repos-fs-group colocation coloc-rule-w-master2 inf: glance-repos ms_drbd:Master order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:start order glance-order-fs-after-drbd-stop inf: glance-repos-fs-group:stop ms_drbd:demote order glance-order-fs-after-drbd-stop2 inf: ms_drbd:demote ms_drbd:stop order glance-order-fs-after-drbd-stop3 inf: ms_drbd:stop glance-repos:stop order glance-order-fs-after-drbd2 inf: ms_drbd:start ms_drbd:promote order glance-order-fs-after-drbd3 inf: ms_drbd:promote glance-repos-fs-group:start property $id="cib-bootstrap-options" \ dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \ cluster-infrastructure="openais" \ expected-quorum-votes="1" \ stonith-enabled="false" \ no-quorum-policy="ignore" \ last-lrm-refresh="1310768814" I will let everyone know how testing goes. Thanks, Bob - Forwarded Message - From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Wednesday, July 20, 2011 1:38 PM Subject: [Pacemaker] Fw: Configuration for FS over DRBD over LVM One correction: I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote ms_drbd:stop glance-repos:stop I called out the stop of DRBD before the stop of LVM. The syslog attached previously is for this configuration. Thanks, Bob From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Wednesday, July 20, 2011 11:32 AM Subject: [Pacemaker] Fw: Configuration for FS over DRBD over LVM I tried another test based on this thread: http://www.gossamer-threads.com/lists/linuxha/pacemaker/65928?search_string=lvm%20drbd;#65928 I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote glance-repos:stop The stop actions were called in this order: stop file system demote DRBD stop LVM * stop DRBD * instead of: stop file system demote DRBD stop DRBD ** stop LVM ** I see these messages in the log which I believe are debug messages based on reading other threads: pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-begin pengine: [21021]: debug: text2task: Unsupported action: glan
Re: [Pacemaker] Initial quorum
--On Wednesday, July 20, 2011 09:19:33 AM + pskrap wrote: > I have a cluster where some of the resources cannot run on the same node. > All resources must be running to provide a functioning service. This > means that a certain amount of nodes needs to be up before it makes > sense for the cluster to start any resources. Without knowing anything about your application, I would tend to question this statement. Is it true that you must not start *any* resources before you have enough nodes, or is sufficient to say that the application is not considered up until all resources are started? It may not make sense to run any, but does it do any harm? If you *can* start at least some resources before all nodes are available, then I would expect that you could get by with defining colocation constraints to ensure that some resources don't run on the same nodes, perhaps augmenting things with some order constraints if necessary. If your applications die or do other horrible stuff when only some subset are running then I'd have a talk with your application developers as it sounds like a larger robustness problem. Devin ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
[Pacemaker] Fw: Configuration for FS over DRBD over LVM
One correction: I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote ms_drbd:stop glance-repos:stop I called out the stop of DRBD before the stop of LVM. The syslog attached previously is for this configuration. Thanks, Bob From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Wednesday, July 20, 2011 11:32 AM Subject: [Pacemaker] Fw: Configuration for FS over DRBD over LVM I tried another test based on this thread: http://www.gossamer-threads.com/lists/linuxha/pacemaker/65928?search_string=lvm%20drbd;#65928 I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote glance-repos:stop The stop actions were called in this order: stop file system demote DRBD stop LVM * stop DRBD * instead of: stop file system demote DRBD stop DRBD ** stop LVM ** I see these messages in the log which I believe are debug messages based on reading other threads: pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-end I have attached a syslog-pacemaker log of the "/etc/init.d/corosync start" through "/etc/init.d/corosync stop" sequence. Thanks, Bob - Forwarded Message - From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Tuesday, July 19, 2011 4:38 PM Subject: [Pacemaker] Configuration for FS over DRBD over LVM Hi, I am trying to configure an FS running on top of DRBD on top of LVM or: FS | DRBD | LVM I am using Pacemaker 1.0.8, Ubuntu 10.04 and DRBD 8.3.7. Reviewing all the manuals (Pacemaker Explained 1.0, DRBD 8.4 User Guide, etc) I came up with this Pacemaker configuration: node cnode-1-3-5 node cnode-1-3-6 primitive glance-drbd ocf:linbit:drbd \ params drbd_resource="glance-repos-drbd" \ op start interval="0" timeout="240" \ op stop interval="0" timeout="100" \ op monitor interval="59s" role="Master" timeout="30s" \ op monitor inte
Re: [Pacemaker] The placement strategy of the group resource does not work well
Hi, Andrew I confirmed that a problem was revised. Many thanks!! Yuusuke (2011/07/19 10:42), Andrew Beekhof wrote: This should now be fixed in: http://hg.clusterlabs.org/pacemaker/devel/rev/960a7e3da680 Its based on your patches but is a little more generic. On Mon, Jul 11, 2011 at 10:22 PM, Yuusuke IIDA wrote: Hi, Yan After trying your correction , I had the following problems. * When trouble happened in a resource below the top of group , it did not come in fail over in the node that intended. I attach crm_report when a problem occurred. I tried a correction again. The patch which I attached traces it to a lower resource of group and gets score. Is there any problem with this correction? Regards, Yuusuke (2011/07/07 21:45), Gao,Yan wrote: Hi Yuusuke, Thanks a lot for the report and the patch! I somewhat prefer simplifying the policy, like the attached patch. What do you think about it? Could you please give it a try? Regards, Yan On 06/30/11 16:36, Yuusuke IIDA wrote: Hi, Yan I wrote a patch for Pacemaker-dev to solve this problem. I want you to confirm the patch. I want a repository to merge it if there is not a problem. Best Regards, Yuusuke (2011/06/20 16:10), Yuusuke IIDA wrote: Hi, Yan I wanted to scatter transition of the group resource, however failed. * Initial state Online: [act1 sby1 sby2] Full list of resources: Resource Group: grp1 rsc1 (ocf::pacemaker:Dummy): Started act1 Resource Group: grp2 rsc2 (ocf::pacemaker:Dummy): Started act1 Resource Group: grp3 rsc3 (ocf::pacemaker:Dummy): Started act1 * When I let you break down in order of grp1 -> grp2 -> grp3. Online: [act1 sby1 sby2] Full list of resources: Resource Group: grp1 rsc1 (ocf::pacemaker:Dummy): Started sby1 Resource Group: grp2 rsc2 (ocf::pacemaker:Dummy): Started sby2 Resource Group: grp3 rsc3 (ocf::pacemaker:Dummy): Started sby1 * When I let you break down in order of grp3 -> grp2 -> grp1. (there is a problem) Online: [act1 sby1 sby2] Full list of resources: Resource Group: grp1 rsc1 (ocf::pacemaker:Dummy): Started sby1 Resource Group: grp2 rsc2 (ocf::pacemaker:Dummy): Started sby1 Resource Group: grp3 rsc3 (ocf::pacemaker:Dummy): Started sby1 As for this, will not it be right like primitive resource to be dispersed transition? I attach hb_report when a problem happened. Best Regards, Yuusuke ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker -- METRO SYSTEMS CO., LTD Yuusuke Iida Mail: iiday...@intellilink.co.jp ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker -- METRO SYSTEMS CO., LTD Yuusuke Iida Mail: iiday...@intellilink.co.jp ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] A question and demand to a resource placement strategy function
Hi, Andrew
I confirmed that a problem was revised.
Many thanks!!
Yuusuke
(2011/07/19 10:42), Andrew Beekhof wrote:
This should also now be fixed in:
http://hg.clusterlabs.org/pacemaker/devel/rev/960a7e3da680
On Tue, Jul 5, 2011 at 9:43 PM, Yuusuke IIDA wrote:
Hi, Andrew
I know that there is the next processing in "pengine".
# cat -n pengine/utils.c
[snip]
322 /* now try to balance resources across the cluster */
323 if(node1->details->num_resources
324< node2->details->num_resources) {
325 do_crm_log_unlikely(level, "%s (%d)< %s (%d) : resources",
326 node1->details->uname,
node1->details->num_resources,
327 node2->details->uname,
node2->details->num_resources);
328 return -1;
329
330 } else if(node1->details->num_resources
331> node2->details->num_resources) {
332 do_crm_log_unlikely(level, "%s (%d)> %s (%d) : resources",
333 node1->details->uname,
node1->details->num_resources,
334 node2->details->uname,
node2->details->num_resources);
335 return 1;
336 }
This processing is a thing to give priority to a node with a little number
of the resources.
And this processing acts regardless of setting of "placement-strategy".
I understand so it.
This processing works expected at the time of next.
A turn of the trouble of resources : rsc1 -> rsc2 -> rsc3
Online: [ act1 act2 act3 sby2 sby1 ]
Full list of resources:
rsc1(ocf::pacemaker:Dummy): Started sby1
rsc2(ocf::pacemaker:Dummy): Started sby2
rsc3(ocf::pacemaker:Dummy): Started sby1
Failed actions:
rsc1_monitor_5000 (node=act1, call=6, rc=7, status=complete): not running
rsc2_monitor_5000 (node=act2, call=6, rc=7, status=complete): not running
rsc3_monitor_5000 (node=act3, call=6, rc=7, status=complete): not running
However , at the time of next , I do not work as expected.(this is a
problem)
A turn of the trouble of resources : rsc3 -> rsc2 -> rsc1
Online: [ act1 act2 act3 sby2 sby1 ]
Full list of resources:
rsc1(ocf::pacemaker:Dummy): Started sby1
rsc2(ocf::pacemaker:Dummy): Started sby1
rsc3(ocf::pacemaker:Dummy): Started sby1
Failed actions:
rsc1_monitor_5000 (node=act1, call=6, rc=7, status=complete): not running
rsc2_monitor_5000 (node=act2, call=6, rc=7, status=complete): not running
rsc3_monitor_5000 (node=act3, call=6, rc=7, status=complete): not running
This problem is improved by the correction that Yan made, but it is not
applied to "default" setting.
I want to apply this correction to "default" setting.
And I think that I want Pacemaker-1.0 to apply the same correction.
However , I want to think once again because there are the problem of group
resources reporting according to the present and the problem of colocation
which I do not yet report after including those corrections.
I attach crm_report of the problem work.
Best Regards,
Yuusuke IIDA
(2011/07/05 13:34), Andrew Beekhof wrote:
On Thu, Jun 2, 2011 at 4:59 PM, Gao,Yanwrote:
On 06/01/11 18:51, Yuusuke IIDA wrote:
Hi, Yan
An answer becomes slow, and really I'm sorry.
(2011/05/13 15:06), Gao,Yan wrote:
I understand that you think the improvement for the non-default
placement strategy makes sense to the "default" too. Though the
"default" is somewhat intended not to be affected by any "placement
strategy" so that the behaviors of existing pengine test cases and
users' deployments remain unchanged.
I think that a function dispersed with the number of the start of the
resource has a problem at the time of "default" setting.
This problem is the Pacemaker-1.0 series, but does the same movement.
If it could be settled by this correction, I thought a correction to be
applicable in Pacemaker-1.0.
Should not this problem be revised?
This would affect dozens of existing regression tests, although most of
the changes are just the scores of clone instances, which are due to
different resource allocating orders. Given 1.0 is in such a maintenance
state, I'm not sure we should do that for 1.0.
Andrew, what do you think about it? Perhaps we should fix the
resource-number-balancing for "default" strategy in 1.1 at least?
I think for 1.1 we can do something, I'd just like to understand the
the implications of the patch.
It would help if there was a testcase that illustrated the negative
behaviour.
Is it necessary that both parts of the old if-block are always run?
For "utilization" strategy, load-balancing is still done based on the
number of resources allocated to a node. That might be a choice.
When I do not set capacity by "utilization" setting in Pacemaker-1.1 ,
expected movement is possible!
Best Regards,
Yuusuke IIDA
Regards,
Yan
--
Gao,Yan
Software Engineer
China Server Team, SUSE.
___
Pacemaker mailing list: Pacemaker@oss.
Re: [Pacemaker] Fw: Configuration for FS over DRBD over LVM
One correction: I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote ms_drbd:stop glance-repos:stop I called out the stop of DRBD before the stop of LVM. The syslog attached previously is for this configuration. Thanks, Bob From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Wednesday, July 20, 2011 11:32 AM Subject: [Pacemaker] Fw: Configuration for FS over DRBD over LVM I tried another test based on this thread: http://www.gossamer-threads.com/lists/linuxha/pacemaker/65928?search_string=lvm%20drbd;#65928 I removed the "location" constraint and simply went with this: colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master glance-repos-fs-group order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start order glance-order-fs-after-drbd2 inf: glance-repos-fs-group:stop ms_drbd:demote glance-repos:stop The stop actions were called in this order: stop file system demote DRBD stop LVM * stop DRBD * instead of: stop file system demote DRBD stop DRBD ** stop LVM ** I see these messages in the log which I believe are debug messages based on reading other threads: pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-0-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-promote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-1-demote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd-2-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-0-start-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-demote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-1-promote-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-stop-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-stop-end pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-begin pengine: [21021]: debug: text2task: Unsupported action: glance-order-fs-after-drbd2-2-start-end I have attached a syslog-pacemaker log of the "/etc/init.d/corosync start" through "/etc/init.d/corosync stop" sequence. Thanks, Bob - Forwarded Message - From: Bob Schatz To: "pacemaker@oss.clusterlabs.org" Sent: Tuesday, July 19, 2011 4:38 PM Subject: [Pacemaker] Configuration for FS over DRBD over LVM Hi, I am trying to configure an FS running on top of DRBD on top of LVM or: FS | DRBD | LVM I am using Pacemaker 1.0.8, Ubuntu 10.04 and DRBD 8.3.7. Reviewing all the manuals (Pacemaker Explained 1.0, DRBD 8.4 User Guide, etc) I came up with this Pacemaker configuration: node cnode-1-3-5 node cnode-1-3-6 primitive glance-drbd ocf:linbit:drbd \ params drbd_resource="glance-repos-drbd" \ op start interval="0" timeout="240" \ op stop interval="0" timeout="100" \ op monitor interval="59s" role="Master" timeout="30s" \ op monitor interv
[Pacemaker] Resource Group Questions - Start/Stop Order
Hi group, I am running a 6 node system, 4 of which mount the LUNs for my Lustre file system. I currently have 29 LUNs per server set up in 4 Resource Groups. I understand the default startup/shudown order of the resource but I was wondering if there is a way to override that and have all the resources in the group startup or shutdown at the same time. Ideally what I am looking for is all the resources in the group OSS1group to startup and shutdown at the same time since none of them are dependent on each other, they just belong on the same server. I found this thread here http://www.gossamer-threads.com/lists/linuxha/pacemaker/60893 which talks about non-ordered groups and I think that is what I need but I am at a loss as to how to find the parameters/attributes of the group to set it up. Is it possible to override the default action of the resource group's startup/shutdown order? Can someone point me to some documentation where I can find the available parameters that can be set for groups? I have attached my configuration in case it's needed and I am running Pacemaker 1.0.11 Bobbie Lind Systems Engineer *Solutions Made Simple, Inc (SMSi)* 703-296-3087 (Cell) bl...@sms-fed.com node s02ns030.dsd.net \ attributes standby="off" node s02ns040 node s02ns050 node s02ns060 node s02ns070 \ attributes standby="off" node s02ns090 \ attributes standby="off" primitive resMDT ocf:heartbeat:Filesystem \ operations $id="resMDT-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw_mdt_vg-dsdw_mdt_vol" directory="/lustre/dsdw-MDT" fstype="lustre" \ meta target-role="Started" primitive resMDTLVM ocf:heartbeat:LVM \ params volgrpname="dsdw_mdt_vg" primitive resOST ocf:heartbeat:Filesystem \ operations $id="resOST-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST" directory="/lustre/dsdw-OST" fstype="lustre" primitive resOST0001 ocf:heartbeat:Filesystem \ operations $id="resOST0001-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0001" directory="/lustre/dsdw-OST0001" fstype="lustre" primitive resOST0002 ocf:heartbeat:Filesystem \ operations $id="resOST0002-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0002" directory="/lustre/dsdw-OST0002" fstype="lustre" primitive resOST0003 ocf:heartbeat:Filesystem \ operations $id="resOST0003-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0003" directory="/lustre/dsdw-OST0003" fstype="lustre" primitive resOST0004 ocf:heartbeat:Filesystem \ operations $id="resOST0004-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0004" directory="/lustre/dsdw-OST0004" fstype="lustre" primitive resOST0005 ocf:heartbeat:Filesystem \ operations $id="resOST0005-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0005" directory="/lustre/dsdw-OST0005" fstype="lustre" primitive resOST0006 ocf:heartbeat:Filesystem \ operations $id="resOST0006-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0006" directory="/lustre/dsdw-OST0006" fstype="lustre" primitive resOST0007 ocf:heartbeat:Filesystem \ operations $id="resOST0007-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0007" directory="/lustre/dsdw-OST0007" fstype="lustre" primitive resOST0008 ocf:heartbeat:Filesystem \ operations $id="resOST0008-operations" \ op monitor interval="120" timeout="60" \ op start interval="0" timeout="300" \ op stop interval="0" timeout="300" \ params device="/dev/mapper/dsdw-OST0008" directory="/lustre/dsdw-OST0008" fstype="lustre" primitive resOST0009 ocf:heartbeat:Filesystem \ operations $id="resOST0009-operations" \ op
Re: [Pacemaker] Cluster with DRBD : split brain
On 07/20/2011 11:24 AM, Hugo Deprez wrote:
> Hello Andrew,
>
> in fact DRBD was in standalone mode but the cluster was working :
>
> Here is the syslog of the drbd's split brain :
>
> Jul 15 08:45:34 node1 kernel: [1536023.052245] block drbd0: Handshake
> successful: Agreed network protocol version 91
> Jul 15 08:45:34 node1 kernel: [1536023.052267] block drbd0: conn(
> WFConnection -> WFReportParams )
> Jul 15 08:45:34 node1 kernel: [1536023.066677] block drbd0: Starting
> asender thread (from drbd0_receiver [23281])
> Jul 15 08:45:34 node1 kernel: [1536023.066863] block drbd0:
> data-integrity-alg:
> Jul 15 08:45:34 node1 kernel: [1536023.079182] block drbd0:
> drbd_sync_handshake:
> Jul 15 08:45:34 node1 kernel: [1536023.079190] block drbd0: self
> BBA9B794EDB65CDF:9E8FB52F896EF383:C5FE44742558F9E1:1F9E06135B8E296F
> bits:75338 flags:0
> Jul 15 08:45:34 node1 kernel: [1536023.079196] block drbd0: peer
> 8343B5F30B2BF674:9E8FB52F896EF382:C5FE44742558F9E0:1F9E06135B8E296F
> bits:769 flags:0
> Jul 15 08:45:34 node1 kernel: [1536023.079200] block drbd0:
> uuid_compare()=100 by rule 90
> Jul 15 08:45:34 node1 kernel: [1536023.079203] block drbd0: Split-Brain
> detected, dropping connection!
> Jul 15 08:45:34 node1 kernel: [1536023.079439] block drbd0: helper
> command: /sbin/drbdadm split-brain minor-0
> Jul 15 08:45:34 node1 kernel: [1536023.083955] block drbd0: meta
> connection shut down by peer.
> Jul 15 08:45:34 node1 kernel: [1536023.084163] block drbd0: conn(
> WFReportParams -> NetworkFailure )
> Jul 15 08:45:34 node1 kernel: [1536023.084173] block drbd0: asender
> terminated
> Jul 15 08:45:34 node1 kernel: [1536023.084176] block drbd0: Terminating
> asender thread
> Jul 15 08:45:34 node1 kernel: [1536023.084406] block drbd0: helper
> command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
> Jul 15 08:45:34 node1 kernel: [1536023.084420] block drbd0: conn(
> NetworkFailure -> Disconnecting )
> Jul 15 08:45:34 node1 kernel: [1536023.084430] block drbd0: error
> receiving ReportState, l: 4!
> Jul 15 08:45:34 node1 kernel: [1536023.084789] block drbd0: Connection
> closed
> Jul 15 08:45:34 node1 kernel: [1536023.084813] block drbd0: conn(
> Disconnecting -> StandAlone )
> Jul 15 08:45:34 node1 kernel: [1536023.086345] block drbd0: receiver
> terminated
> Jul 15 08:45:34 node1 kernel: [1536023.086349] block drbd0: Terminating
> receiver thread
This was a DRBD split-brain, not a pacemaker split. I think that might
have been the source of confusion.
The split brain occurs when both DRBD nodes lose contact with one
another and then proceed as StandAlone/Primary/UpToDate. To avoid this,
configure fencing (stonith) in Pacemaker, then use 'crm-fence-peer.sh'
in drbd.conf;
===
disk {
fencing resource-and-stonith;
}
handlers {
outdate-peer"/path/to/crm-fence-peer.sh";
}
===
This will tell DRBD to block (resource) and fence (stonith). DRBD will
not resume IO until either the fence script exits with a success, or
until an admit types 'drbdadm resume-io '.
The CRM script simply calls pacemaker and asks it to fence the other
node. When a node has actually failed, then the lost no is fenced. If
both nodes are up but disconnected, as you had, then only the fastest
node will succeed in calling the fence, and the slower node will be
fenced before it can call a fence.
--
Digimer
E-Mail: digi...@alteeve.com
Freenode handle: digimer
Papers and Projects: http://alteeve.com
Node Assassin: http://nodeassassin.org
"At what point did we forget that the Space Shuttle was, essentially,
a program that strapped human beings to an explosion and tried to stab
through the sky with fire and math?"
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Cluster with DRBD : split brain
Hello Andrew, in fact DRBD was in standalone mode but the cluster was working : Here is the syslog of the drbd's split brain : Jul 15 08:45:34 node1 kernel: [1536023.052245] block drbd0: Handshake successful: Agreed network protocol version 91 Jul 15 08:45:34 node1 kernel: [1536023.052267] block drbd0: conn( WFConnection -> WFReportParams ) Jul 15 08:45:34 node1 kernel: [1536023.066677] block drbd0: Starting asender thread (from drbd0_receiver [23281]) Jul 15 08:45:34 node1 kernel: [1536023.066863] block drbd0: data-integrity-alg: Jul 15 08:45:34 node1 kernel: [1536023.079182] block drbd0: drbd_sync_handshake: Jul 15 08:45:34 node1 kernel: [1536023.079190] block drbd0: self BBA9B794EDB65CDF:9E8FB52F896EF383:C5FE44742558F9E1:1F9E06135B8E296F bits:75338 flags:0 Jul 15 08:45:34 node1 kernel: [1536023.079196] block drbd0: peer 8343B5F30B2BF674:9E8FB52F896EF382:C5FE44742558F9E0:1F9E06135B8E296F bits:769 flags:0 Jul 15 08:45:34 node1 kernel: [1536023.079200] block drbd0: uuid_compare()=100 by rule 90 Jul 15 08:45:34 node1 kernel: [1536023.079203] block drbd0: Split-Brain detected, dropping connection! Jul 15 08:45:34 node1 kernel: [1536023.079439] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 Jul 15 08:45:34 node1 kernel: [1536023.083955] block drbd0: meta connection shut down by peer. Jul 15 08:45:34 node1 kernel: [1536023.084163] block drbd0: conn( WFReportParams -> NetworkFailure ) Jul 15 08:45:34 node1 kernel: [1536023.084173] block drbd0: asender terminated Jul 15 08:45:34 node1 kernel: [1536023.084176] block drbd0: Terminating asender thread Jul 15 08:45:34 node1 kernel: [1536023.084406] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0) Jul 15 08:45:34 node1 kernel: [1536023.084420] block drbd0: conn( NetworkFailure -> Disconnecting ) Jul 15 08:45:34 node1 kernel: [1536023.084430] block drbd0: error receiving ReportState, l: 4! Jul 15 08:45:34 node1 kernel: [1536023.084789] block drbd0: Connection closed Jul 15 08:45:34 node1 kernel: [1536023.084813] block drbd0: conn( Disconnecting -> StandAlone ) Jul 15 08:45:34 node1 kernel: [1536023.086345] block drbd0: receiver terminated Jul 15 08:45:34 node1 kernel: [1536023.086349] block drbd0: Terminating receiver thread On 19 July 2011 02:30, Andrew Beekhof wrote: > On Fri, Jul 15, 2011 at 7:58 PM, Hugo Deprez > wrote: > > Dear community, > > > > I am running on Debian Lenny, a cluster with corosync. I have : > > > > One DRBD partition and 4 resources : > > > > fs-data(ocf::heartbeat:Filesystem): > > mda-ip (ocf::heartbeat:IPaddr2): > > postfix(ocf::heartbeat:postfix): > > apache (ocf::heartbeat:apache): > > > > Last night something happens and DRBD had a 'split brain'. I think the > split > > brain come from > > > > The resources was still running on the node 1. > > > > I checked the corosync logs and seems that something went wrong, I would > > like to understand what happen, in order to improve my cluster > > configuration. > > > > Please find attach the log file. > > I see no evidence of a split-brain. Both nodes appear to be able to > talk to each other. > What exactly is the problem you encountered? > > > > > It seems that the cluster tried to migrate the resources to the other > node > > but didn't succeed ? > > > > Any help appreciated. > > > > Regards, > > > > Hugo > > > > ___ > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > > > Project Home: http://www.clusterlabs.org > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > > Bugs: > > > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker > > > > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker > ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
[Pacemaker] Initial quorum
Hi, I have a cluster where some of the resources cannot run on the same node. All resources must be running to provide a functioning service. This means that a certain amount of nodes needs to be up before it makes sense for the cluster to start any resources. All works fine after enough nodes have been installed and started so that Pacemaker have detected them and updated the expected quorum votes to a high enough number. However, during installation of nodes Pacemaker does not yet know that a certain number of nodes is needed and it will attempt to start resources already after installing the first node. This obviously causes failures since the resources have colocation rules. If I understand correctly Pacemaker automatically detects new nodes and updates the number of expected quorum votes accordingly. My question is: Is it possible to somehow tell Pacemaker that X number of nodes is needed before resources can be started? For example is it possible to manually set the number of expected quorum votes? Or can nodes somehow be manually made known to Pacemaker before it has actually seen them? Right now I have to install enough nodes first and then try to start Pacemaker simultaneously on all nodes to prevent issues. This makes automating the installation much harder since the nodes would need to synchronize the Pacemaker start as opposed to just start it up and when quorum is reached resources would be started. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
