Re: [Pacemaker] create 2 node cluster with clvm starting with one node

[Andrei Borzenkov]

On Thu, Sep 24, 2015 at 3:26 PM, Sven Moeller  wrote:
> Hi,
>
> I have to build a 2 node NFS Cluster based on Pacemaker/Corosync. The Volume 
> used for the filesystem that will be exported by NFS is on a shared storage. 
> I would like to use cLVM on this Volume.

Why do you need cLVM in the first place? Will you be using clustered
filesystem that allows concurrent access from both nodes?

> The challenge is, at the moment just one node is available. That means, I 
> have to create the cluster configuration for a two node cluster by having 
> just one node available.
>
> Is it possible to get cLVM running with just one node? Is this possible by 
> setting quorum policy to ignore?
>
> Is it possible to get cLVM running without enabling Stonith? (as long as i 
> got only one of the two nodes, I'm forced to disable stonith, because the 
> other node is in production).
>
> thank's and kind regards
>
> Sven
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] writing a script

[Andrei Borzenkov]

On 12.08.2015 17:18, Michael Böhm wrote:

2015-08-12 15:45 GMT+02:00 Karl Rößmann :



Hi,

is there an easy way to determine in a script (bash or python) whether
a CRM Resource (Xen Domain) is running or not ?



We query the location of a resource in bash with this:

if [ ! "$(/usr/sbin/crm_resource -Q -r resourcename -W)" = "$(hostname)" ];
then

just have a look at "man crm_resource" i'm pretty sure you'll find
something suitable.



"crm resource status " is also possible. But having some 
more script-friendly output would be good.


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] stonith

[Andrei Borzenkov]

В Sun, 19 Apr 2015 14:23:27 +0200
Andreas Kurz  пишет:

> On 2015-04-17 12:36, Thomas Manninger wrote:
> > Hi list,
> >  
> > i have a pacemaker/corosync2 setup with 4 nodes, stonith configured over
> > ipmi interface.
> >  
> > My problem is, that sometimes, a wrong node is stonithed.
> > As example:
> > I have 4 servers: node1, node2, node3, node4
> >  
> > I start a hardware- reset on node node1, but node1 and node3 will be
> > stonithed.
> 
> You have to tell pacemaker exactly what stonith-resource can fence what
> node if the stonith agent you are using does not support the "list" action.
> 

pacmeker is expected to get this information dynamically from stonith
agent.

> Do this by adding "pcmk_host_check=static-list" and "pcmk_host_list" to
> every stonith-resource like:
> 

Default for pcmk_host_check is "dynamic"; why it does not work in this
case? I use external/ipmi muself and I do not remember ever fiddling
with static list.

> primitive p_stonith_node3 stonith:external/ipmi \
>   op monitor interval=3s timeout=20s \
>   params hostname=node3 ipaddr=10.100.0.6 passwd_method=file
>   passwd="/etc/stonith_ipmi_passwd" userid=stonith interface=lanplus
>   priv=OPERATOR \
>   pcmk_host_check="static-list" pcmk_host_list="node3"
> 
> ... see "man stonithd".
> 
> Best regards,
> Andreas
> 
> >  
> > In the cluster.log, i found following entry:
> > Apr 17 11:02:41 [20473] node2   stonithd:debug:
> > stonith_action_create:   Initiating action reboot for agent
> > fence_legacy (target=node1)
> > Apr 17 11:02:41 [20473] node2   stonithd:debug: make_args:  
> > Performing reboot action for node 'node1' as 'port=node1'
> > Apr 17 11:02:41 [20473] node2   stonithd:debug:
> > internal_stonith_action_execute: forking
> > Apr 17 11:02:41 [20473] node2   stonithd:debug:
> > internal_stonith_action_execute: sending args
> > Apr 17 11:02:41 [20473] node2   stonithd:debug:
> > stonith_device_execute:  Operation reboot for node node1 on
> > p_stonith_node3 now running with pid=113092, timeout=60s
> >  
> > node1 will be reseted with the stonith primitive of node3 ?? Why??
> >  
> > my stonith config:
> > primitive p_stonith_node1 stonith:external/ipmi \
> > params hostname=node1 ipaddr=10.100.0.2 passwd_method=file
> > passwd="/etc/stonith_ipmi_passwd" userid=stonith interface=lanplus
> > priv=OPERATOR \
> > op monitor interval=3s timeout=20s \
> > meta target-role=Started failure-timeout=30s
> > primitive p_stonith_node2 stonith:external/ipmi \
> > op monitor interval=3s timeout=20s \
> > params hostname=node2 ipaddr=10.100.0.4 passwd_method=file
> > passwd="/etc/stonith_ipmi_passwd" userid=stonith interface=lanplus
> > priv=OPERATOR \
> > meta target-role=Started failure-timeout=30s
> > primitive p_stonith_node3 stonith:external/ipmi \
> > op monitor interval=3s timeout=20s \
> > params hostname=node3 ipaddr=10.100.0.6 passwd_method=file
> > passwd="/etc/stonith_ipmi_passwd" userid=stonith interface=lanplus
> > priv=OPERATOR \
> > meta target-role=Started failure-timeout=30s
> > primitive p_stonith_node4 stonith:external/ipmi \
> > op monitor interval=3s timeout=20s \
> > params hostname=node4 ipaddr=10.100.0.8 passwd_method=file
> > passwd="/etc/stonith_ipmi_passwd" userid=stonith interface=lanplus
> > priv=OPERATOR \
> > meta target-role=Started failure-timeout=30s
> >  
> > Somebody can help me??
> > Thanks!
> >  
> > Regards,
> > Thomas
> > 
> > 
> > ___
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> > 
> 
> 



pgpNMWZizhxWA.pgp
Description: OpenPGP digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Question about multiple instance_attributes

[Andrei Borzenkov]

On Mon, Apr 13, 2015 at 12:32 PM, Kazunori INOUE
 wrote:
> 2015-04-10 15:09 GMT+09:00 Andrei Borzenkov :
>> On Fri, Apr 10, 2015 at 8:22 AM, Kazunori INOUE
>>  wrote:
>>> Hi,
>>>
>>> I defined multiple instance_attributes [*].
>>>  * 
>>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/_using_rules_to_control_resource_options.html
>>> Although it expected that start_opt="-p 5441 ..." would be used on vm1
>>> and vm2, it wasn't so.
>>>
>>> # pacemakerd -F
>>> Pacemaker 1.1.12 (Build: 3e93bc1)
>>>
>>> # cibadmin -Q
>>> >> type="pgsql">
>>>   
>>> >> id="prmPostgreSQLDB-instance_attributes-rule">
>>>   >> id="prmPostgreSQLDB-instance_attributes-rule-expression"/>
>>>   >> id="prmPostgreSQLDB-instance_attributes-rule-expression-0"/>
>>> 
>>> >> id="prmPostgreSQLDB-instance_attributes-start_opt"/>
>>> >> id="prmPostgreSQLDB-instance_attributes-pgport"/>
>>>   
>>>   
>>> >> id="prmPostgreSQLDB-instance_attributes-0-start_opt"/>
>>> >> id="prmPostgreSQLDB-instance_attributes-0-pgport"/>
>>>   (snip)
>>>
>>> # hostname
>>> vm1
>>> # ps -ax | grep postgres
>>> 28591 ?S0:00 /usr/pgsql-9.4/bin/postgres (snip) -p 5449 -h
>>> 192.168.201.140
>>>
>>> Is it a design that boolean-op of "rule" doesn't function?
>>>
>>
>> I suspect that your second rule is simply unconditionally applied
>> because it does not have any rules to evaluate.
>>
> That is, is 'boolean-op="or"' invalid?
>

That's not what I said. Your second rule does not have any expressions
(sorry, I meant that) so it is always applied unconditionally.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Configuring fencing with encrypted passwords

[Andrei Borzenkov]

On Fri, Feb 27, 2015 at 12:31 PM, Arjun Pandey  wrote:
> Hi
>
> I am facing some issues while trying out fence_ipmilan on ILO4 setup when
> using encrypted passwords. Which mailing list can i contact for this ?
> Basically if i test out fence_ipmilan/fence_ilo4 from cmd line with
> plaintext password things work fine. However if i specify encrypted password
> and specify auth-type as md5. It doesn't work.
>
> WORKING
> fence_ilo4 -a 10.11.10.21 -l foo -p bar -o status -v
>
> FAILS
> fence_ilo4 -A md5 -fence_ilo4 -A md5 -a 10.11.10.21 -l foo
> -p $1$MGrg5Egf$QQpDePvU1UXtF/3ixMaRS0 -o status -v

You need to quote shell metacharacters on command line.

>
>
> Any pointers would be helpful.
>
> Regards
> Arjun
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Two node cluster and no hardware device for stonith.

[Andrei Borzenkov]

В Tue, 10 Feb 2015 15:58:57 +0100
Dejan Muhamedagic  пишет:

> On Mon, Feb 09, 2015 at 04:41:19PM +0100, Lars Ellenberg wrote:
> > On Fri, Feb 06, 2015 at 04:15:44PM +0100, Dejan Muhamedagic wrote:
> > > Hi,
> > > 
> > > On Thu, Feb 05, 2015 at 09:18:50AM +0100, Digimer wrote:
> > > > That is the problem that makes geo-clustering very hard to nearly
> > > > impossible. You can look at the Booth option for pacemaker, but that
> > > > requires two (or more) full clusters, plus an arbitrator 3rd
> > > 
> > > A full cluster can consist of one node only. Hence, it is
> > > possible to have a kind of stretch two-node [multi-site] cluster
> > > based on tickets and managed by booth.
> > 
> > In theory.
> > 
> > In practice, we rely on "proper behaviour" of "the other site",
> > in case a ticket is revoked, or cannot be renewed.
> > 
> > Relying on a single node for "proper behaviour" does not inspire
> > as much confidence as relying on a multi-node HA-cluster at each site,
> > which we can expect to ensure internal fencing.
> > 
> > With reliable hardware watchdogs, it still should be ok to do
> > "stretched two node HA clusters" in a reliable way.
> > 
> > Be generous with timeouts.
> 
> As always.
> 
> > And document which failure modes you expect to handle,
> > and how to deal with the worst-case scenarios if you end up with some
> > failure case that you are not equipped to handle properly.
> > 
> > There are deployments which favor
> > "rather online with _potential_ split brain" over
> > "rather offline just in case".
> 
> There's an arbitrator which should help in case of split brain.
> 

You can never really differentiate between site down and site cut off
due to (network) infrastructure outage. Arbitrator can mitigate split
brain only to the extent you trust your network. You still have to take
decision what you value more - data availability or data consistency.

Long distance clusters are really for disaster recovery. It is
convenient to have a single button that starts up all resources in
controlled manner, but someone really need to decide to push this
button.

> > Document this, print it out on paper,
> > 
> >"I am aware that this may lead to lost transactions,
> >data divergence, data corruption, or data loss.
> >I am personally willing to take the blame,
> >and live with the consequences."
> > 
> > Have some "boss" sign that ^^^
> > in the real world using a real pen.
> 
> Well, of course running such a "stretch" cluster would be
> rather different from a "normal" one.
> 
> The essential thing is that there's no fencing, unless configured
> as a dead-man switch for the ticket. Given that booth has a
> "sanity" program hook, maybe that could be utilized to verify if
> this side of the cluster is healthy enough.
> 
> Thanks,
> 
> Dejan
> 
> > Lars
> > 
> > -- 
> > : Lars Ellenberg
> > : http://www.LINBIT.com | Your Way to High Availability
> > : DRBD, Linux-HA  and  Pacemaker support and consulting
> > 
> > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> > 
> > ___
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Master-Slave role stickiness

[Andrei Borzenkov]

On Wed, Jan 21, 2015 at 11:06 PM, brook davis  wrote:
> Hi,
>
> I've got a master-slave resource and I'd like to achieve the following
> behavior with it:
>
> * Only ever run (as master or slave) on 2 specific nodes (out of N possible
> nodes).  These nodes are predetermined and are specified at resource
> creation time.
> * Prefer one specific node (of the 2 selected for running the resource) for
> starting in the Master role.
> * Upon failover event, promote the secondary node to master.
> * Do not re-promote the failed node back to master, should it come back
> online.
>
> The last requirement is the one I'm currently struggling with.  I can force
> the resource to run on only the 2 nodes I want (out of 3 possible nodes),
> but I can't get it to "stick" on the secondary node as master after a
> failover and recovery.   That is, when I take the original master offline,
> the resource promotes correctly on the secondary, but if I bring the origin
> node back online, the resource is demoted on the secondary and promotes back
> to master on the origin.   I'd like to avoid that last bit.
>

It sounds like default-resource-stickiness does not kick in; and with
default resource-stickiness=1 it is expected (10 > 6).  Documentation
says default-recource-stickiness is deprecated so may be it is ignored
in your version altogether? What "ptest -L -s" shows?

> Here's the relevant bits of my CRM configuration:
>
> primitive NIMHA-01 ocf:heartbeat:nimha \
> op start interval="0" timeout="60s" \
> op monitor interval="30s" role="Master" \
> op stop interval="0" timeout="60s" \
> op monitor interval="45s" role="Slave" \
> ms NIMMS-01 NIMHA-01 \
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
> notify="true" target-role="Started" is-managed="true"
> location prefer-elmy-inf NIMMS-01 5: elmyra
> location prefer-elmyra-ms NIMMS-01 \
> rule $id="prefer-elmyra-rule" $role="Master" 10: #uname eq elmyra
> location prefer-pres-inf NIMMS-01 5: president
> location prefer-president-ms NIMMS-01 \
> rule $id="prefer-president-rule" $role="Master" 5: #uname eq president
> property $id="cib-bootstrap-options" \
> dc-version="1.1.10-42f2063" \
> cluster-infrastructure="corosync" \
> stonith-enabled="false" \
> no-quorum-policy="ignore" \
> last-lrm-refresh="1421798334" \
> default-resource-stickiness="200" \
> symmetric-cluster="false"
>
>
> I've set symmetric-cluster="false" to achieve an "opt-in" behavior, per the
> corosync docs.  From my understanding, these location constraints should
> direct the resource to be able to run on the two nodes, preferring 'elmyra'
> initially as Master.  My question then becomes, is there a way to apply the
> stickiness to the Master role ??   I've tried adding explicit stickiness
> settings (high numbers and INF) to the default-resource-stickiness, the
> actual "ms" resource, and the primitive, all to no avail.
>
> Anyone have any ideas on how to achieve stickiness on the master role in
> such a configuration ?
>
> Thanks for any and all help in advance,
>
> brook
>
> ps.  please ignore/forgive the no-quorum-policy and stonith-enabled settings
> in my configuration...  I know it's bad and not best practice.  I don't
> think it should affect the answer to the above question, though, based on my
> understanding of the system.
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Corosync fails to start when NIC is absent

[Andrei Borzenkov]

On Tue, Jan 20, 2015 at 11:50 AM, Jan Friesse  wrote:
> Kostiantyn,
>
>
>> One more thing to clarify.
>> You said "rebind can be avoided" - what does it mean?
>
> By that I mean that as long as you don't shutdown interface everything
> will work as expected. Interface shutdown is administrator decision,
> system doesn't do it automagically :)
>

What is possible that e.g. during reboot interface (hardware) fails
and is not detected. This would lead to complete outage of node that
could be avoided.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Unique clone instance is stopped too early on move

[Andrei Borzenkov]

On Tue, Jan 13, 2015 at 10:20 AM, Vladislav Bogdanov
 wrote:
> Hi Andrew, David, all.
>
> I found a little bit strange operation ordering during transition execution.
>
> Could you please look at the following partial configuration (crmsh syntax)?
>
> ===
> ...
> clone cl-broker broker \
> meta interleave=true target-role=Started
> clone cl-broker-vips broker-vips \
> meta clone-node-max=2 globally-unique=true interleave=true 
> resource-stickiness=0 target-role=Started
> clone cl-ctdb ctdb \
> meta interleave=true target-role=Started
> colocation broker-vips-with-broker inf: cl-broker-vips cl-broker
> colocation broker-with-ctdb inf: cl-broker cl-ctdb
> order broker-after-ctdb inf: cl-ctdb cl-broker
> order broker-vips-after-broker 0: cl-broker cl-broker-vips
> ...
> ===
>
> After I put one node to standby and then back to online, I see the following 
> transition (relevant excerpt):
>
> ===
>  * Pseudo action:   cl-broker-vips_stop_0
>  * Resource action: broker-vips:1   stop on c-pa-0
>  * Pseudo action:   cl-broker-vips_stopped_0
>  * Pseudo action:   cl-ctdb_start_0
>  * Resource action: ctdbstart on c-pa-1
>  * Pseudo action:   cl-ctdb_running_0
>  * Pseudo action:   cl-broker_start_0
>  * Resource action: ctdbmonitor=1 on c-pa-1
>  * Resource action: broker  start on c-pa-1
>  * Pseudo action:   cl-broker_running_0
>  * Pseudo action:   cl-broker-vips_start_0
>  * Resource action: broker  monitor=1 on c-pa-1
>  * Resource action: broker-vips:1   start on c-pa-1
>  * Pseudo action:   cl-broker-vips_running_0
>  * Resource action: broker-vips:1   monitor=3 on c-pa-1
> ===
>
> What could be a reason to stop unique clone instance so early for move?
>

Do not take it as definitive answer, but cl-broker-vips cannot run
unless both other resources are started. So if you compute closure of
all required transitions it looks rather logical. Having
cl-broker-vips started while broker is still stopped would violate
constraint.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] CoroSync's UDPu transport for public IP addresses?

[Andrei Borzenkov]

On Mon, Dec 29, 2014 at 1:50 PM, Dejan Muhamedagic  wrote:
> Hi,
>
> On Mon, Dec 29, 2014 at 06:11:49AM +0300, Dmitry Koterov wrote:
>> Hello.
>>
>> I have a geographically distributed cluster, all machines have public IP
>> addresses. No virtual IP subnet exists, so no multicast is available.
>>
>> I thought that UDPu transport can work in such environment, doesn't it?
>>
>> To test everything in advance, I've set up a corosync+pacemaker on Ubuntu
>> 14.04 with the following corosync.conf:
>>
>> totem {
>>   transport: udpu
>>   interface {
>> ringnumber: 0
>> bindnetaddr: ip-address-of-the-current-machine
>> mcastport: 5405
>>   }
>
> You need to add the member directives too. See corosync.conf(5).
>

Are not member directives for corosync 1.x and nodelist directives for
corosync 2.x?

Dmitry, which version do you have?

> Thanks,
>
> Dejan
>
>> }
>> nodelist {
>>   node {
>> ring0_addr: node1
>>   }
>>   node {
>> ring0_addr: node2
>>   }
>> }
>> ...
>>
>> (here node1 and node2 are hostnames from /etc/hosts on both machines).
>> After running "service corosync start; service pacemaker start" logs show
>> no problems, but actually both nodes are always offline:
>>
>> root@node1:/etc/corosync# crm status | grep node
>> OFFLINE: [ node1 node2 ]
>>
>> and "crm node online" (as all other attempts to make crm to do something)
>> are timed out with "communication error".
>>
>> No iptables, selinux, apparmor and other bullshit are active: just pure
>> virtual machines with single public IP addresses on each. Also tcpdump
>> shows that UDB packets on port 5405 are going in and out, and if I e.g.
>> stop corosync at node1, the tcpdump output at node2 changes significantly.
>> So they see each other definitely.
>>
>> And if I attach a gvpe adapter to these 2 machines with a private subnet
>> and switch transport to the default one, corosync + pacemaker begin to work.
>>
>> So my question is: what am I doing wrong? Maybe UDPu is not suitable for
>> communications among machines with public IP addresses only?
>
>> ___
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] [libqb]Unlink of files bound to sockets

[Andrei Borzenkov]

В Mon, 22 Dec 2014 16:25:00 -0500 (EST)
David Vossel  пишет:

> 
> Linux has this non-portable concept of abstract sockets. This lets us create
> sockets without mapping directly to something on the filesystem. Unfortunately
> solaris doesn't have this feature. Somewhere along the line we introduced a
> file leak for solaris.
> 
> Your original patch is actually very close to being correct.
> 
> Take a look at this function.
> 
> https://github.com/davidvossel/libqb/blob/master/lib/ipc_socket.c#L53
> 
> That's where we set the sun_path.
> 
> for the '#if defined(QB_LINUX) || defined(QB_CYGWIN)' sun_path is actually 
> being
> set to ='\0',  which indicates we want to use that abstract socket feature. 
> I'm
> unsure why we continue with what appears to be a useless snprintf to 
> sun_path+1.
> 

For abstract Linux socket you still need unique socket name; it just
starts with '\0' to indicate that it is not file path. 

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] location minus points

[Andrei Borzenkov]

В Wed, 17 Dec 2014 16:38:56 +0100
"Thomas Manninger"  пишет:

> Hi,
>  
> i am using pacemaker on debian in a test environment. pacemaker version is: 
> 1.1.7-1
>  
> I have a problem unterstanding minus points in a location.
>  
> I have 3 nodes in my test environment, node1, node2 and node3
> I created a ressource, named resBind9, which should be preferred started on 
> node1 and node2, if one of the two nodes fails, bind9 should be start on the 
> third node.
>  
> The first crm configuration works correctly:
> node node1
> node node2
> node node3
> primitive resBind9 lsb:bind9 \
> op monitor interval="5"
> clone cloneBind9 resBind9 \
> meta clone-max="2" clone-node-max="1" target-role="Started"
> location locBind9 cloneBind9 100: node1
> location locBind9_2 cloneBind9 100: node2
> property $id="cib-bootstrap-options" \
> dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="3" \
>  
> But, why is the following configuration not working? Bind9 never starts on 
> node3!
> node node1
> node node2
> node node3
> primitive resBind9 lsb:bind9 \
> op monitor interval="5"
> clone cloneBind9 resBind9 \
> meta clone-max="2" clone-node-max="1" target-role="Started"
> location locBind9 cloneBind9 -100: mysqlcluster-slave2
 ^^^ what's this?
> property $id="cib-bootstrap-options" \
> dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="3" \
>  
> "-100" points means, that bind9 only starts on the location, of another node 
> are down, or is this no correct??
>  

No, negative score means node cannot run on this node. Unless some
other rule makes score positive, resource will not run on this node. 

> Thanks for help!


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Failed-over incomplete

[Andrei Borzenkov]

В Thu, 4 Dec 2014 17:41:39 +0700
Teerapatr Kittiratanachai  пишет:

> sorry for my mistyping,
> it's res.vBKN6
> 

pacemaker tried to stop res.vBKN6 but resource agent failed to do it

Dec 03 03:35:57 [2027] node0.ntt.co.th   crmd:   notice: process_lrm_event: 
LRM operation res.vBKN6_stop_0 (call=97, rc=1, cib-update=34, 
confirmed=true) unknown error

This means pacemaker cannot start res.vBKN6 anywhere else, at least
without going via node stonith. 

> --teenigma
> 
> On Thu, Dec 4, 2014 at 4:23 PM, Andrei Borzenkov  wrote:
> > On Thu, Dec 4, 2014 at 9:52 AM, Teerapatr Kittiratanachai
> >  wrote:
> >> Dear Andrei,
> >>
> >> Since the failed over is uncompleted so all the resource isn't failed
> >> over to another node.
> >>
> >> I think this case happened because of the res.vBKN is go into unmanaged 
> >> state.
> >>
> >
> > There is no resource res.vBKN in your logs or configuration snippet
> > you have shown.
> >
> >> But why? Since there is no configuration is changed.
> >>
> >> --teenigma
> >>
> >> On Thu, Dec 4, 2014 at 1:41 PM, Andrei Borzenkov  
> >> wrote:
> >>> On Thu, Dec 4, 2014 at 4:56 AM, Teerapatr Kittiratanachai
> >>>  wrote:
> >>>> Dear List,
> >>>>
> >>>> We are using Pacemaker and Corosync with CMAN as our HA software as
> >>>> below version.
> >>>>
> >>>> OS:CentOS release 6.5 (Final) 64-bit
> >>>> Pacemaker:pacemaker.x86_641.1.10-14.el6_5.3
> >>>> Corosync:corosync.x86_641.4.1-17.el6_5.1
> >>>> CMAN:cman.x86_643.0.12.1-59.el6_5.2
> >>>> Resource-Agent:resource-agents.x86_643.9.5-3.12
> >>>>
> >>>> Topology:2 Nodes with Active/Standby model. (MySQL is
> >>>> Active/Active by clone)
> >>>>
> >>>> All packages are install from CentOS official repository, and the
> >>>> Resource-Agent is only one which be installed from OpenSUSE repository
> >>>> (http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/).
> >>>>
> >>>> The system is work normally for few months until yesterday morning,
> >>>> around 03:35 UTC+0700, we found that one of resource is go into
> >>>> UNMANAGED state without any configuration changed. After another
> >>>> resource is failed, the pacemaker try to failed-over resource to
> >>>> another node but it incomplete after facing this resource.
> >>>>
> >>>> Configuration of some resource is below and the LOG during event is in
> >>>> attached file.
> >>>>
> >>>
> >>> The log just covers resource monitor failure and stopping of
> >>> resources. It does not contain any event related to starting resources
> >>> on another nodes.
> >>>
> >>> You would need to collect crm_report with start time before resource
> >>> failed and stop time after resources were started on another node.
> >>>
> >>>> primitive res.vBKN6 IPv6addr \
> >>>> params ipv6addr="2001:db8:0:f::61a" cidr_netmask=64 nic=eth0 \
> >>>> op monitor interval=10s
> >>>>
> >>>> primitive res.vDMZ6 IPv6addr \
> >>>> params ipv6addr="2001:db8:0:9::61a" cidr_netmask=64 nic=eth1 \
> >>>> op monitor interval=10s
> >>>>
> >>>> group gr.mainService res.vDMZ4 res.vDMZ6 res.vBKN4 res.vBKN6 res.http 
> >>>> res.ftp
> >>>>
> >>>> rsc_defaults rsc_defaults-options: \
> >>>> migration-threshold=1
> >>>>
> >>>> Please help me to solve this problem.
> >>>>
> >>>> --teenigma
> >>>>
> >>>> ___
> >>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >>>>
> >>>> Project Home: http://www.clusterlabs.org
> >>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >>>> Bugs: http://bugs.clusterlabs.org
> >>>>
> >>>
> >>> ___
> >&g

Re: [Pacemaker] Failed-over incomplete

[Andrei Borzenkov]

On Thu, Dec 4, 2014 at 9:52 AM, Teerapatr Kittiratanachai
 wrote:
> Dear Andrei,
>
> Since the failed over is uncompleted so all the resource isn't failed
> over to another node.
>
> I think this case happened because of the res.vBKN is go into unmanaged state.
>

There is no resource res.vBKN in your logs or configuration snippet
you have shown.

> But why? Since there is no configuration is changed.
>
> --teenigma
>
> On Thu, Dec 4, 2014 at 1:41 PM, Andrei Borzenkov  wrote:
>> On Thu, Dec 4, 2014 at 4:56 AM, Teerapatr Kittiratanachai
>>  wrote:
>>> Dear List,
>>>
>>> We are using Pacemaker and Corosync with CMAN as our HA software as
>>> below version.
>>>
>>> OS:CentOS release 6.5 (Final) 64-bit
>>> Pacemaker:pacemaker.x86_641.1.10-14.el6_5.3
>>> Corosync:corosync.x86_641.4.1-17.el6_5.1
>>> CMAN:cman.x86_643.0.12.1-59.el6_5.2
>>> Resource-Agent:resource-agents.x86_643.9.5-3.12
>>>
>>> Topology:2 Nodes with Active/Standby model. (MySQL is
>>> Active/Active by clone)
>>>
>>> All packages are install from CentOS official repository, and the
>>> Resource-Agent is only one which be installed from OpenSUSE repository
>>> (http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/).
>>>
>>> The system is work normally for few months until yesterday morning,
>>> around 03:35 UTC+0700, we found that one of resource is go into
>>> UNMANAGED state without any configuration changed. After another
>>> resource is failed, the pacemaker try to failed-over resource to
>>> another node but it incomplete after facing this resource.
>>>
>>> Configuration of some resource is below and the LOG during event is in
>>> attached file.
>>>
>>
>> The log just covers resource monitor failure and stopping of
>> resources. It does not contain any event related to starting resources
>> on another nodes.
>>
>> You would need to collect crm_report with start time before resource
>> failed and stop time after resources were started on another node.
>>
>>> primitive res.vBKN6 IPv6addr \
>>> params ipv6addr="2001:db8:0:f::61a" cidr_netmask=64 nic=eth0 \
>>> op monitor interval=10s
>>>
>>> primitive res.vDMZ6 IPv6addr \
>>> params ipv6addr="2001:db8:0:9::61a" cidr_netmask=64 nic=eth1 \
>>> op monitor interval=10s
>>>
>>> group gr.mainService res.vDMZ4 res.vDMZ6 res.vBKN4 res.vBKN6 res.http 
>>> res.ftp
>>>
>>> rsc_defaults rsc_defaults-options: \
>>> migration-threshold=1
>>>
>>> Please help me to solve this problem.
>>>
>>> --teenigma
>>>
>>> ___
>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>> ___
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Failed-over incomplete

[Andrei Borzenkov]

On Thu, Dec 4, 2014 at 4:56 AM, Teerapatr Kittiratanachai
 wrote:
> Dear List,
>
> We are using Pacemaker and Corosync with CMAN as our HA software as
> below version.
>
> OS:CentOS release 6.5 (Final) 64-bit
> Pacemaker:pacemaker.x86_641.1.10-14.el6_5.3
> Corosync:corosync.x86_641.4.1-17.el6_5.1
> CMAN:cman.x86_643.0.12.1-59.el6_5.2
> Resource-Agent:resource-agents.x86_643.9.5-3.12
>
> Topology:2 Nodes with Active/Standby model. (MySQL is
> Active/Active by clone)
>
> All packages are install from CentOS official repository, and the
> Resource-Agent is only one which be installed from OpenSUSE repository
> (http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/).
>
> The system is work normally for few months until yesterday morning,
> around 03:35 UTC+0700, we found that one of resource is go into
> UNMANAGED state without any configuration changed. After another
> resource is failed, the pacemaker try to failed-over resource to
> another node but it incomplete after facing this resource.
>
> Configuration of some resource is below and the LOG during event is in
> attached file.
>

The log just covers resource monitor failure and stopping of
resources. It does not contain any event related to starting resources
on another nodes.

You would need to collect crm_report with start time before resource
failed and stop time after resources were started on another node.

> primitive res.vBKN6 IPv6addr \
> params ipv6addr="2001:db8:0:f::61a" cidr_netmask=64 nic=eth0 \
> op monitor interval=10s
>
> primitive res.vDMZ6 IPv6addr \
> params ipv6addr="2001:db8:0:9::61a" cidr_netmask=64 nic=eth1 \
> op monitor interval=10s
>
> group gr.mainService res.vDMZ4 res.vDMZ6 res.vBKN4 res.vBKN6 res.http res.ftp
>
> rsc_defaults rsc_defaults-options: \
> migration-threshold=1
>
> Please help me to solve this problem.
>
> --teenigma
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Suicide fencing and watchdog questions

[Andrei Borzenkov]

В Thu, 27 Nov 2014 08:24:56 +0300
Vladislav Bogdanov  пишет:

> 27.11.2014 03:43, Andrew Beekhof wrote:
> > 
> >> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov  
> >> wrote:
> >>
> >> Hi,
> >>
> >> Is there any information how watchdog integration is intended to work?
> >> What are currently-evaluated use-cases for that?
> >> It seems to be forcibly disabled id SBD is not detected...
> > 
> > Are you referring to no-quorum-policy=suicide?
> 
> That too.
> 
> But main intention was to understand what value that feature can bring
> at all.
> I tried to enable it without SBD or no-quorum-policy=suicide and
> watchdog was not fired up. Then I looked at sources and realized that it
> is enabled only when SBD is detected, and is not actually managed by the
> cluster option.
> 

It is not enough for a node to kill itself, other nodes need to find out
whether it has done it. What are other options besides SBD in this case?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Issues with ocf_run on CentOS7 with pgsql resource agent

[Andrei Borzenkov]

On Fri, Nov 21, 2014 at 12:36 AM, Brendan Reekie  wrote:
> I’m running into an issues with the pgsql resource agent running on CentOS7.
> The issue is when the pgsql resource agent attempts a call to runasowner it
> uses a su as user postgres in a call to ocf_run, this is causing what
> appears to be a new session on CentOS7.
>
>
>
> We’ve narrowed it down to the ocf_run method in ocf-shellfuncs in executing
> the command:
>
>output=`"$@" 2>&1`
>
>
>
> Snippet of /var/log/messages:
>
> Nov 20 21:29:05 ip-10-0-0-4 pengine[2041]: notice: LogActions: Stop
> pgsql:0  (pcmk-1)
>
> Nov 20 21:29:05 ip-10-0-0-4 pengine[2041]: notice: process_pe_message:
> Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-205.bz2
>
> Nov 20 21:29:05 ip-10-0-0-4 crmd[2042]: notice: te_rsc_command: Initiating
> action 47: notify pgsql_pre_notify_stop_0 on pcmk-1 (local)
>
> Nov 20 21:29:05 ip-10-0-0-4 crmd[2042]: notice: te_rsc_command: Initiating
> action 48: notify pgsql_pre_notify_stop_0 on pcmk-2
>
> Nov 20 21:29:05 ip-10-0-0-4 pacemakerd[2412]: notice: crm_add_logfile:
> Additional logging available in /var/log/pacemaker.log
>
> Nov 20 21:29:05 ip-10-0-0-4 su: (to postgres) root on none
>
> Nov 20 21:29:05 ip-10-0-0-4 systemd: Created slice user-26.slice.
>
> Nov 20 21:29:05 ip-10-0-0-4 systemd: Starting Session c4 of user postgres.
>
> Nov 20 21:29:05 ip-10-0-0-4 systemd: Started Session c4 of user postgres.
>
> Nov 20 21:29:30 ip-10-0-0-4 crmd[2042]: notice: process_lrm_event: LRM
> operation pgsql_notify_0 (call=17, rc=0, cib-update=0, confirmed=true) ok
>
> Nov 20 21:29:30 ip-10-0-0-4 crmd[2042]: notice: te_rsc_command: Initiating
> action 1: stop pgsql_stop_0 on pcmk-1 (local)
>
> Nov 20 21:29:30 ip-10-0-0-4 pacemakerd[2464]: notice: crm_add_logfile:
> Additional logging available in /var/log/pacemaker.log
>
> Nov 20 21:29:30 ip-10-0-0-4 su: (to postgres) root on none
>
> Nov 20 21:29:30 ip-10-0-0-4 systemd: Created slice user-26.slice.
>
> Nov 20 21:29:30 ip-10-0-0-4 systemd: Starting Session c5 of user postgres.
>
> Nov 20 21:29:30 ip-10-0-0-4 systemd: Started Session c5 of user postgres.
>
>
>
> Software versions:
>
> pacemaker 1.1.10
>
> resource-agents 3.9.5
>
> corosync 2.3.3-2
>
>
>
> Has anyone else experienced this issue or know of what might be going wrong
> to cause new sessions to be created?
>

Session is created by pam_systemd. What exact problem does it cause?
So far you did not mention any.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] "Hard error" Preventing a service to restart

[Andrei Borzenkov]

В Thu, 20 Nov 2014 14:53:10 +
"Brusq, Jerome"  пишет:

> Dear all,
> 
> We are running corosync-1.4.1-4.el6.x86_64 on RHEL 6.2. We have a cluster 
> with 2 nodes that run as services a Virtual IP and around 20 custom LSB 
> scripts.
> We had some network issue 2 weeks ago and we got a strange  behavior.
> 
> On the logs below we can see that at 06 08:36:29 that the connection between 
> the nodes has been lost, then we see a strange log "pengine: [2621]: ERROR: 
> unpack_rsc_op: Hard error - 5-controller_last_failure_0 failed with rc=3: 
> Preventing 5-controller from re-starting on srv01".
> 
> Only one of our LSB script "5-controller"  got an "Hard error" and this 
> service got the status "unmanaged". This means that once the network got 
> back, the 2 nodes communicate again, and all services have been restarted , 
> excepted "5-controller".
> 
> Do you have an idea about what happened ?
> 

The error is returned by resource agent, pacemaker is just messenger.
You need to check LSB script why it does it.

> Thanks a lot,
> 
> Jerome
> 
> 
> Nov 06 08:31:35 srv01 cib: [2618]: info: cib_stats: Processed 71 operations 
> (5633.00us average, 0% utilization) in the last 10min
> Nov 06 08:36:28 corosync [TOTEM ] A processor failed, forming new 
> configuration.
> Nov 06 08:36:29 corosync [pcmk  ] notice: pcmk_peer_update: Transitional 
> membership event on ring 156: memb=1, new=0, lost=1
> Nov 06 08:36:29 corosync [pcmk  ] info: pcmk_peer_update: memb: srv01 
> 1661074698
> Nov 06 08:36:29 corosync [pcmk  ] info: pcmk_peer_update: lost: srv02 
> 1677851914
> Nov 06 08:36:29 corosync [pcmk  ] notice: pcmk_peer_update: Stable membership 
> event on ring 156: memb=1, new=0, lost=0
> Nov 06 08:36:29 corosync [pcmk  ] info: pcmk_peer_update: MEMB: srv01 
> 1661074698
> Nov 06 08:36:29 corosync [pcmk  ] info: ais_mark_unseen_peer_dead: Node srv02 
> was not seen in the previous transition
> Nov 06 08:36:29 corosync [pcmk  ] info: update_member: Node 1677851914/srv02 
> is now: lost
> Nov 06 08:36:29 corosync [pcmk  ] info: send_member_notification: Sending 
> membership update 156 to 2 children
> Nov 06 08:36:29 srv01 cib: [2618]: notice: ais_dispatch_message: Membership 
> 156: quorum lost
> Nov 06 08:36:29 srv01 cib: [2618]: info: crm_update_peer: Node srv02: 
> id=1677851914 state=lost (new) addr=r(0) ip(10.10.1.100)  votes=1 born=76 
> seen=84 proc=00111312
> Nov 06 08:36:29 corosync [TOTEM ] A processor joined or left the membership 
> and a new membership was formed.
> Nov 06 08:36:29 corosync [CPG   ] chosen downlist: sender r(0) ip(10.10.1.99) 
> ; members(old:2 left:1)
> Nov 06 08:36:29 corosync [MAIN  ] Completed service synchronization, ready to 
> provide service.
> Nov 06 08:36:29 srv01 crmd: [2622]: notice: ais_dispatch_message: Membership 
> 156: quorum lost
> Nov 06 08:36:29 srv01 crmd: [2622]: info: ais_status_callback: status: srv02 
> is now lost (was member)
> Nov 06 08:36:29 srv01 crmd: [2622]: info: crm_update_peer: Node srv02: 
> id=1677851914 state=lost (new) addr=r(0) ip(10.10.1.100)  votes=1 born=76 
> seen=84 proc=00111312
> Nov 06 08:36:29 srv01 crmd: [2622]: WARN: check_dead_member: Our DC node 
> (srv02) left the cluster
> Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: State 
> transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL 
> origin=check_dead_member ]
> Nov 06 08:36:29 srv01 crmd: [2622]: info: update_dc: Unset DC srv02
> Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: State 
> transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC 
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Nov 06 08:36:29 srv01 crmd: [2622]: info: do_te_control: Registering TE UUID: 
> ceab74dd-6a27-4eda-ba66-b63d77beaee5
> Nov 06 08:36:29 srv01 crmd: [2622]: info: set_graph_functions: Setting custom 
> graph functions
> Nov 06 08:36:29 srv01 crmd: [2622]: info: unpack_graph: Unpacked transition 
> -1: 0 actions in 0 synapses
> Nov 06 08:36:29 srv01 crmd: [2622]: info: do_dc_takeover: Taking over DC 
> status for this partition
> Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_readwrite: We are now in 
> R/W mode
> Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation 
> complete: op cib_master for section 'all' (origin=local/crmd/709, 
> version=0.2504.157): ok (rc=0)
> Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation 
> complete: op cib_modify for section cib (origin=local/crmd/710, 
> version=0.2504.158): ok (rc=0)
> Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation 
> complete: op cib_modify for section crm_config (origin=local/crmd/712, 
> version=0.2504.159): ok (rc=0)
> Nov 06 08:36:29 srv01 crmd: [2622]: info: join_make_offer: Making join offers 
> based on membership 156
> Nov 06 08:36:29 srv01 crmd: [2622]: info: do_dc_join_offer_all: join-1: 
> Waiting on 1 outstanding join acks
> Nov 06 08:36:29 srv01 cib: [2618

Re: [Pacemaker] Long failover

[Andrei Borzenkov]

On Mon, Nov 17, 2014 at 9:34 AM, Andrew Beekhof  wrote:
>
>> On 14 Nov 2014, at 10:57 pm, Dmitry Matveichev  
>> wrote:
>>
>> Hello,
>>
>> We have a cluster configured via pacemaker+corosync+crm. The configuration 
>> is:
>>
>> node master
>> node slave
>> primitive HA-VIP1 IPaddr2 \
>> params ip=192.168.22.71 nic=bond0 \
>> op monitor interval=1s
>> primitive HA-variator lsb: variator \
>> op monitor interval=1s \
>> meta migration-threshold=1 failure-timeout=1s
>> group HA-Group HA-VIP1 HA-variator
>> property cib-bootstrap-options: \
>> dc-version=1.1.10-14.el6-368c726 \
>> cluster-infrastructure="classic openais (with plugin)" \
>
> General advice, don't use the plugin. See:
>
> http://blog.clusterlabs.org/blog/2013/pacemaker-and-rhel-6-dot-4/
> http://blog.clusterlabs.org/blog/2013/pacemaker-on-rhel6-dot-4/
>
>> expected-quorum-votes=2 \
>> stonith-enabled=false \
>>no-quorum-policy=ignore \
>> last-lrm-refresh=1383871087
>> rsc_defaults rsc-options: \
>> resource-stickiness=100
>>
>> Firstly I make the variator service down  on the master node (actually I 
>> delete the service binary and kill the variator process, so the variator 
>> fails to restart). Resources very quickly move on the slave node as 
>> expected. Then I return the binary on the master and restart the variator 
>> service. Now I make the same stuff with binary and service on slave node. 
>> The crm status command quickly shows me HA-variator   (lsb: variator):   
>>  Stopped. But it take to much time (for us) before recourses are switched on 
>> the master node (around 1 min).
>
> I see what you mean:
>
> 2013-12-21T07:04:12.230827+04:00 master crmd[14267]:   notice: 
> te_rsc_command: Initiating action 2: monitor HA-variator_monitor_1000 on 
> slave.mfisoft.ru
> 2013-12-21T05:45:09+04:00 slave crmd[7086]:   notice: process_lrm_event: 
> slave.mfisoft.ru-HA-variator_monitor_1000:106 [ variator.x is stopped\n ]
>
> (1 minute goes by)
>
> 2013-12-21T07:05:14.232029+04:00 master crmd[14267]:error: print_synapse: 
> [Action2]: In-flight rsc op HA-variator_monitor_1000 on slave.mfisoft.ru 
> (priority: 0, waiting: none)
> 2013-12-21T07:05:14.232102+04:00 master crmd[14267]:  warning: 
> cib_action_update: rsc_op 2: HA-variator_monitor_1000 on slave.mfisoft.ru 
> timed out
>

Is it possible that pacemaker is confused by time difference on master
and slave?

> Is there a corosync log file configured?  That would have more detail on 
> slave.
>
>> Then line
>> Failed actions:
>> HA- variator _monitor_1000 on slave 'unknown error' (1): call=-1, 
>> status=Timed Out, last-rc-change='Sat Dec 21 03:59:45 2013', queued=0ms, 
>> exec=0ms
>> appears in the crm status and recourses are switched.
>>
>> What is that timeout? Where I can change it?
>>
>> 
>> Kind regards,
>> Dmitriy Matveichev.
>>
>> ___
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Long failover

[Andrei Borzenkov]

On Fri, Nov 14, 2014 at 4:33 PM, Dmitry Matveichev
 wrote:
> We've already tried to set it but it didn't help.
>

I doubt it is possible to say anything without logs.

> 
> Kind regards,
> Dmitriy Matveichev.
>
>
> -Original Message-
> From: Andrei Borzenkov [mailto:arvidj...@gmail.com]
> Sent: Friday, November 14, 2014 4:12 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Long failover
>
> On Fri, Nov 14, 2014 at 2:57 PM, Dmitry Matveichev  
> wrote:
>> Hello,
>>
>>
>>
>> We have a cluster configured via pacemaker+corosync+crm. The
>> configuration
>> is:
>>
>>
>>
>> node master
>>
>> node slave
>>
>> primitive HA-VIP1 IPaddr2 \
>>
>> params ip=192.168.22.71 nic=bond0 \
>>
>> op monitor interval=1s
>>
>> primitive HA-variator lsb: variator \
>>
>> op monitor interval=1s \
>>
>> meta migration-threshold=1 failure-timeout=1s
>>
>> group HA-Group HA-VIP1 HA-variator
>>
>> property cib-bootstrap-options: \
>>
>> dc-version=1.1.10-14.el6-368c726 \
>>
>> cluster-infrastructure="classic openais (with plugin)" \
>>
>> expected-quorum-votes=2 \
>>
>> stonith-enabled=false \
>>
>>no-quorum-policy=ignore \
>>
>> last-lrm-refresh=1383871087
>>
>> rsc_defaults rsc-options: \
>>
>> resource-stickiness=100
>>
>>
>>
>> Firstly I make the variator service down  on the master node (actually
>> I delete the service binary and kill the variator process, so the
>> variator fails to restart). Resources very quickly move on the slave
>> node as expected. Then I return the binary on the master and restart
>> the variator service. Now I make the same stuff with binary and service on 
>> slave node.
>> The crm status command quickly shows me HA-variator   (lsb: variator):
>> Stopped. But it take to much time (for us) before recourses are switched on
>> the master node (around 1 min).   Then line
>>
>> Failed actions:
>>
>> HA- variator _monitor_1000 on slave 'unknown error' (1): call=-1,
>> status=Timed Out, last-rc-change='Sat Dec 21 03:59:45 2013',
>> queued=0ms, exec=0ms
>>
>> appears in the crm status and recourses are switched.
>>
>>
>>
>> What is that timeout? Where I can change it?
>>
>
> This is operation timeout. You can change it in operation definition:
> op monitor interval=1s timeout=5s
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org 
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org Getting started: 
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Long failover

[Andrei Borzenkov]

On Fri, Nov 14, 2014 at 2:57 PM, Dmitry Matveichev
 wrote:
> Hello,
>
>
>
> We have a cluster configured via pacemaker+corosync+crm. The configuration
> is:
>
>
>
> node master
>
> node slave
>
> primitive HA-VIP1 IPaddr2 \
>
> params ip=192.168.22.71 nic=bond0 \
>
> op monitor interval=1s
>
> primitive HA-variator lsb: variator \
>
> op monitor interval=1s \
>
> meta migration-threshold=1 failure-timeout=1s
>
> group HA-Group HA-VIP1 HA-variator
>
> property cib-bootstrap-options: \
>
> dc-version=1.1.10-14.el6-368c726 \
>
> cluster-infrastructure="classic openais (with plugin)" \
>
> expected-quorum-votes=2 \
>
> stonith-enabled=false \
>
>no-quorum-policy=ignore \
>
> last-lrm-refresh=1383871087
>
> rsc_defaults rsc-options: \
>
> resource-stickiness=100
>
>
>
> Firstly I make the variator service down  on the master node (actually I
> delete the service binary and kill the variator process, so the variator
> fails to restart). Resources very quickly move on the slave node as
> expected. Then I return the binary on the master and restart the variator
> service. Now I make the same stuff with binary and service on slave node.
> The crm status command quickly shows me HA-variator   (lsb: variator):
> Stopped. But it take to much time (for us) before recourses are switched on
> the master node (around 1 min).   Then line
>
> Failed actions:
>
> HA- variator _monitor_1000 on slave 'unknown error' (1): call=-1,
> status=Timed Out, last-rc-change='Sat Dec 21 03:59:45 2013', queued=0ms,
> exec=0ms
>
> appears in the crm status and recourses are switched.
>
>
>
> What is that timeout? Where I can change it?
>

This is operation timeout. You can change it in operation definition:
op monitor interval=1s timeout=5s

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Daemon Start attempt on wrong Server

[Andrei Borzenkov]

В Tue, 11 Nov 2014 16:19:56 +0100
Hauke Homburg  пишет:

> Am 11.11.2014 13:34, schrieb Alexandre:
> >
> > You should use an opt out cluster. Set the cluster option  
> > symmetrical=false. This will tell corosync not to place a resource 
> > anywhere on the cluster, unless a location rule explicitly tell the 
> > cluster where it should run.
> >
> > Corosync will still monitor sql resources on www hosts and return rc 5 
> > but this is expected and works.
> >
> > Le 11 nov. 2014 13:22, "Hauke Homburg"  > > a écrit :
> >
> > Hello,
> >
> > I am installing a 6 Node pacemaker CLuster. 3 Nodes for Apache, 3
> > Nodes for Postgres.
> >
> > My Cluster Config is
> >
> > node kvm-node1
> > node sql-node1
> > node sql-node2
> > node sql-node3
> > node www-node1
> > node www-node2
> > node www-node3
> > primitive pri_kvm_ip ocf:heartbeat:IPaddr2 \
> > params ip="10.0.6.41" cidr_netmask="255.255.255.0" \
> > op monitor interval="10s" timeout="20s"
> > primitive pri_sql_ip ocf:heartbeat:IPaddr2 \
> > params ip="10.0.6.31" cidr_netmask="255.255.255.0" \
> > op monitor interval="10s" timeout="20s"
> > primitive pri_www_ip ocf:heartbeat:IPaddr2 \
> > params ip="10.0.6.21" cidr_netmask="255.255.255.0" \
> > op monitor interval="10s" timeout="20s"
> > primitive res_apache ocf:heartbeat:apache \
> > params configfile="/etc/apache2/apache2.conf" \
> > op start interval="0" timeout="40" \
> > op stop interval="0" timeout="60" \
> > op monitor interval="60" timeout="120" start-delay="0" \
> > meta target-role="Started"
> > primitive res_pgsql ocf:heartbeat:pgsql \
> > params pgctl="/usr/lib/postgresql/9.1/bin/pg_ctl"
> > psql="/usr/bin/psql" start_opt=""
> > pgdata="/var/lib/postgresql/9.1/main"
> > config="/etc/postgresql/9.1/main/postgresql.conf" pgdba="postgres" \
> > op start interval="0" timeout="120s" \
> > op stop interval="0" timeout="120s" \
> > op monitor interval="30s" timeout="30s" depth="0"
> > location loc_kvm_ip_node1 pri_kvm_ip 10001: kvm-node1
> > location loc_sql_ip_node1 pri_sql_ip inf: sql-node1
> > location loc_sql_ip_node2 pri_sql_ip inf: sql-node2
> > location loc_sql_ip_node3 pri_sql_ip inf: sql-node3
> > location loc_sql_srv_node1 res_pgsql inf: sql-node1
> > location loc_sql_srv_node2 res_pgsql inf: sql-node2
> > location loc_sql_srv_node3 res_pgsql inf: sql-node3
> > location loc_www_ip_node1 pri_www_ip inf: www-node1
> > location loc_www_ip_node2 pri_www_ip inf: www-node2
> > location loc_www_ip_node3 pri_www_ip inf: www-node3
> > location loc_www_srv_node1 res_apache inf: www-node1
> > location loc_www_srv_node2 res_apache inf: www-node2
> > location loc_www_srv_node3 res_apache inf: www-node3
> > property $id="cib-bootstrap-options" \
> > dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
> > cluster-infrastructurFailed actions:
> >
> > Why do i see in crm_mon the following output?
> >
> > res_pgsql_start_0 (node=www-node1, call=16, rc=5,
> > status=complete): not installed
> > res_pgsql_start_0 (node=www-node2, call=13, rc=5,
> > status=complete): not installed
> > pri_www_ip_monitor_1 (node=www-node3, call=22, rc=7,
> > status=complete): not running
> > res_pgsql_start_0 (node=www-node3, call=13, rc=5,
> > status=complete): not installed
> > res_apache_start_0 (node=sql-node2, call=18, rc=5,
> > status=complete): not installed
> > res_pgsql_start_0 (node=sql-node2, call=12, rc=5,
> > status=complete): not installed
> > res_apache_start_0 (node=sql-node3, call=12, rc=5,
> > status=complete): not installed
> > res_pgsql_start_0 (node=sql-node3, call=10, rc=5,
> > status=complete): not installed
> > res_apache_start_0 (node=kvm-node1, call=12, rc=5,
> > status=complete): not installed
> > res_pgsql_start_0 (node=kvm-node1, call=20, rc=5,
> > status=complete): not installede="openais" \
> > expected-quorum-votes="7" \
> > stonith-enabled="false"
> >
> >
> > I set the infinity for pgsql on all 3 sql nodes, but not! on the
> > www nodes. Why tries Pacemaker to start the Postgres SQL Server on
> > the www Node? In example?
> >
> > Thank for your Help
> >
> > greetings
> >
> > Hauke
> >
> > ___
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > 
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > B

Re: [Pacemaker] Fencing dependency between bare metal host and its VMs guest

[Andrei Borzenkov]

В Mon, 10 Nov 2014 10:07:18 +0100
Tomasz Kontusz  пишет:

> I think the suggestion was to put shooting the host in the fencing path of a 
> VM. This way if you can't get the host to fence the VM (as the host is 
> already dead) you just check if the host was fenced.
> 

Exactly. One thing I do not know how it will behave in case of multiple
VMs on the same host. I.e. will pacemaker try to fence host for every
VM or recognize that all VMs are dead after the first time agent is
invoked.

> Daniel Dehennin  napisał:
> >Andrei Borzenkov  writes:
> >
> >
> >[...]
> >
> >>> Now I have one issue, when the bare metal host on which the VM is
> >>> running die, the VM is lost and can not be fenced.
> >>> 
> >>> Is there a way to make pacemaker ACK the fencing of the VM running
> >on a
> >>> host when the host is fenced itself?
> >>> 
> >>
> >> Yes, you can define multiple stonith agents and priority between
> >them.
> >>
> >> http://clusterlabs.org/wiki/Fencing_topology
> >
> >Hello,
> >
> >If I understand correctly, fencing topology is the way to have several
> >fencing devices for a node and try them consecutively until one works.
> >
> >In my configuration, I group the VM stonith agents with the
> >corresponding VM resource, to make them move together[1].
> >
> >Here is my use case:
> >
> >1. Resource ONE-Frontend-Group runs on nebula1
> >2. nebula1 is fenced
> >3. node one-fronted can not be fenced
> >
> >Is there a way to say that the life on node one-frontend is related to
> >the state of resource ONE-Frontend?
> >
> >In which case when the node nebula1 is fenced, pacemaker should be
> >aware that
> >resource ONE-Frontend is not running any more, so node one-frontend is
> >OFFLINE and not UNCLEAN.
> >
> >Regards.
> >
> >Footnotes: 
> >[1] 
> >http://oss.clusterlabs.org/pipermail/pacemaker/2014-October/022671.html
> >
> >-- 
> >Daniel Dehennin
> >Récupérer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF
> >Fingerprint: 3E69 014E 5C23 50E8 9ED6  2AAD CC1E 9E5B 7A6F E2DF
> >
> >
> >
> >
> >
> >___
> >Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> >http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> >Project Home: http://www.clusterlabs.org
> >Getting started:
> >http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >Bugs: http://bugs.clusterlabs.org
> 


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Fencing dependency between bare metal host and its VMs guest

[Andrei Borzenkov]

В Fri, 07 Nov 2014 17:46:40 +0100
Daniel Dehennin  пишет:

> Hello,
> 
> As I finally manage to integrate my VM to corosync and my dlm/clvm/GFS2
> are running on it.
> 
> Now I have one issue, when the bare metal host on which the VM is
> running die, the VM is lost and can not be fenced.
> 
> Is there a way to make pacemaker ACK the fencing of the VM running on a
> host when the host is fenced itself?
> 

Yes, you can define multiple stonith agents and priority between them.

http://clusterlabs.org/wiki/Fencing_topology

> Regards.
> 



signature.asc
Description: PGP signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] stonith q

[Andrei Borzenkov]

В Mon, 3 Nov 2014 07:07:41 +
Alex Samad - Yieldbroker  пишет:

> {snip}
> > > What I am hearing is that its not available. Is it possible to hook to
> > > a custom script on that event, I can write my own restart
> > >
> > 
> > Sure you can write your own external stonith script.
> 
> 
> Any pointers to a frame work somewhere ?

I do not think there is any formal stonith agent developers guide; take
at any existing agent like external/ipmi and modify to suite your needs.

> Does fenced have any handlers, I notice it logs a message in syslog and 
> cluster log is there a chance to capture the event there ?

I do not have experience with RH CMAN, sorry. But from what I
understand fenced and stonithd agents are compatible.

> 
> A


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] One direction switchover not working

[Andrei Borzenkov]

В Mon, 3 Nov 2014 15:05:39 +0530
Arjun Pandey  пишет:

> Hi
> 
> I have a 2 node CentOS -6.5 based cluster. This being run in active
> standby mode.
> When i want to trigger switchover from blackwidow to elektra (using
> 1.pcs standby blackwidow and 2. pcs unstandby blackwidow) it doesn't
> work.

According to log it was half way through promoting on elektra when
blackwidow returned online

Nov 03 05:45:25 [5780] elektracib:   notice: cib:diff:  --   


and became candidate for master election

Nov 03 05:46:04 [5785] elektra   crmd: info: abort_transition_graph:
te_update_diff:172 - Triggered transition abort (complete=0, node=blackwidow, 
tag=nvpair, id=status-blackwidow-master-foo, name=master-foo, value=5, 
magic=NA, cib=0.484.8) : Transient attribute: update

Unfortunately it does not look like scores are generally logged,
but /var/lib/pacemaker/pengine/pe-input-167.bz2 should contain some
more details.

> The other direction elektra to blackwidow works correctly.
> I was comparing the decision of the DC for each of the cases. Based on
> crm_simulate output it's correct for each case. However after
> pre-promote i don't get a promote on elektra. Eventually blackwidow
> comes up and is promoted.  This has been confirmed based on logs.
> 
> Attaching logs as well as the the corosync log file from elektra. Can
> somebody point out why this is happening. The resource agent script
> for foo does nothing on pre-notify simply logs the notification type
> and the node on which action was being taken.
> 
> I am attaching the pengine comparison output as well as logs for both cases.
> 
> Thanks in advance for any pointers.
> 
> 
> Regards
> Arjun


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Occasional nonsensical resource agent errors, redux

[Andrei Borzenkov]

В Mon, 3 Nov 2014 15:26:34 +0100
Dejan Muhamedagic  пишет:

> Hi,
> 
> On Mon, Nov 03, 2014 at 08:46:00AM +0300, Andrei Borzenkov wrote:
> > В Mon, 3 Nov 2014 13:32:45 +1100
> > Andrew Beekhof  пишет:
> > 
> > > 
> > > > On 1 Nov 2014, at 11:03 pm, Patrick Kane  wrote:
> > > > 
> > > > Hi all:
> > > > 
> > > > In July, list member Ken Gaillot reported occasional nonsensical 
> > > > resource agent errors using Pacemaker 
> > > > (http://oss.clusterlabs.org/pipermail/pacemaker/2014-July/022231.html).
> > > > 
> > > > We're seeing similar issues with our install.  We have a 2 node 
> > > > corosync/pacemaker failover configuration that is using the 
> > > > ocf:heartbeat:IPaddr2 resource agent extensively.  About once a week, 
> > > > we'll get an error like this, out of the blue:
> > > > 
> > > >   Nov  1 05:23:57 lb02 IPaddr2(anon_ip)[32312]: ERROR: Setup problem: 
> > > > couldn't find command: ip
> > > > 
> > > > It goes without saying that the ip command hasn't gone anywhere and all 
> > > > the paths are configured correctly.
> > > > 
> > > > We're currently running 1.1.10-14.el6_5.3-368c726 under CentOS 6 x86_64 
> > > > inside of a xen container.
> > > > 
> > > > Any thoughts from folks on what might be happening or how we can get 
> > > > additional debug information to help figure out what's triggering this?
> > > 
> > > its pretty much in the hands of the agent.
> > 
> > Actually the message seems to be output by check_binary() function
> > which is part of framework.  
> 
> Someone complained in the IRC about this issue (another resource
> agent though, I think Xen) and they said that which(1) was not
> able to find the program. I'd suggest to do strace (or ltrace)
> of which(1) at that point (it's in ocf-shellfuncs).
> 
> The which(1) utility is a simple tool: it splits the PATH
> environment variable and stats the program name appended to each
> of the paths. PATH somehow corrupted or filesystem misbehaving?
> My guess is that it's the former.
> 

As it is called quite often I'd instrument have_binary to dump all
environment and variables on "which" failure for known binary as well as
rerun it under strace. Running it under strace every time would
probably result in too copious output. 

> BTW, was there an upgrade of some kind before this started
> happening?
> 
> Thanks,
> 
> Dejan
> 
> > > you could perhaps find the call that looks for ip and wrap it in a set 
> > > -x/set +x block
> > > that way you'd know exactly why it thinks the binary is missing
> > > ___
> > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > 
> > 
> > ___
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Occasional nonsensical resource agent errors, redux

[Andrei Borzenkov]

В Mon, 3 Nov 2014 13:32:45 +1100
Andrew Beekhof  пишет:

> 
> > On 1 Nov 2014, at 11:03 pm, Patrick Kane  wrote:
> > 
> > Hi all:
> > 
> > In July, list member Ken Gaillot reported occasional nonsensical resource 
> > agent errors using Pacemaker 
> > (http://oss.clusterlabs.org/pipermail/pacemaker/2014-July/022231.html).
> > 
> > We're seeing similar issues with our install.  We have a 2 node 
> > corosync/pacemaker failover configuration that is using the 
> > ocf:heartbeat:IPaddr2 resource agent extensively.  About once a week, we'll 
> > get an error like this, out of the blue:
> > 
> >   Nov  1 05:23:57 lb02 IPaddr2(anon_ip)[32312]: ERROR: Setup problem: 
> > couldn't find command: ip
> > 
> > It goes without saying that the ip command hasn't gone anywhere and all the 
> > paths are configured correctly.
> > 
> > We're currently running 1.1.10-14.el6_5.3-368c726 under CentOS 6 x86_64 
> > inside of a xen container.
> > 
> > Any thoughts from folks on what might be happening or how we can get 
> > additional debug information to help figure out what's triggering this?
> 
> its pretty much in the hands of the agent.

Actually the message seems to be output by check_binary() function
which is part of framework.  

> you could perhaps find the call that looks for ip and wrap it in a set -x/set 
> +x block
> that way you'd know exactly why it thinks the binary is missing
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] stonith q

[Andrei Borzenkov]

В Sun, 2 Nov 2014 20:47:22 +
Alex Samad - Yieldbroker  пишет:

> 
> 
> > -Original Message-
> > From: Digimer [mailto:li...@alteeve.ca]
> > Sent: Monday, 3 November 2014 3:26 AM
> > To: The Pacemaker cluster resource manager; Alex Samad - Yieldbroker
> > Subject: Re: [Pacemaker] stonith q
> > 
> > On 02/11/14 06:45 AM, Andrei Borzenkov wrote:
> > > В Sun, 2 Nov 2014 10:01:59 +
> > > Alex Samad - Yieldbroker  пишет:
> > >
> > >>
> > >>
> > >>> -Original Message-
> > >>> From: Digimer [mailto:li...@alteeve.ca]
> > >>> Sent: Sunday, 2 November 2014 9:49 AM
> > >>> To: The Pacemaker cluster resource manager
> > >>> Subject: Re: [Pacemaker] stonith q
> > >>>
> > >>> On 01/11/14 06:27 PM, Alex Samad - Yieldbroker wrote:
> > >>>> Hi
> {snip}
> > >
> > > That hardly makes sense except in pure test environment. Stonith is
> > > needed when you do not know state of partner node, in which case you
> > > cannot be sure your reboot/shutdown command will be executed, nor that
> > > you can reach your partner at all.
> > >
> > > If you are running under Vmware, use stonith/vmware or stonith/vcenter.
> > 
> > Andrei is correct. A stonith method must be external to the node and work
> > regardless of the state of a node. Try this; 'echo c > /proc/sysrq-trigger' 
> > will
> > crash the node. Any stonith method that requires the OS to respond will fail
> > and your cluster will hang.
> 
> Yes but vmware will restart the node in that circumstance.
> I have had issues with my 2 node cluster where 1 node will remove itself 
> because of lack of communication. Very hard to track down when it happens 
> every now and then and only at 1am, I believe because of backup traffic or 
> because its starved of cpu cycles.
> 
> What I would like to see happen in that situation if for a reboot to be 
> issued, I know that the node would respond and I know that it would reconnect.
> 
> I read that there was a suicide option module that did what I wanted but its 
> not available.
> 
> I don't want to setup useid for vmware for each node and configure that. I 
> just want fenced to do a reboot via the os of the node instead of just 
> killing cman 
> 
> What I am hearing is that its not available. Is it possible to hook to a 
> custom script on that event, I can write my own restart
>  

Sure you can write your own external stonith script.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] stonith q

[Andrei Borzenkov]

В Sun, 2 Nov 2014 10:01:59 +
Alex Samad - Yieldbroker  пишет:

> 
> 
> > -Original Message-
> > From: Digimer [mailto:li...@alteeve.ca]
> > Sent: Sunday, 2 November 2014 9:49 AM
> > To: The Pacemaker cluster resource manager
> > Subject: Re: [Pacemaker] stonith q
> > 
> > On 01/11/14 06:27 PM, Alex Samad - Yieldbroker wrote:
> > > Hi
> > >
> > > 2 node cluster, running under vmware
> {snip}
> > >
> > > Alex
> > 
> > In cman's cluster.conf, you configure the fence device 'fence_pcmk', as you
> > have. That is a dummy/hook fence agent that simply passes fence requests
> > up to pacemaker to actually perform. Pacemaker will then tell cman whether
> > the fence succeeded or failed.
> > 
> > To make sure you have cluster.conf configured properly, it should look
> > something like this;
> > 
> > 
> > ccs -f /etc/cluster/cluster.conf --createcluster an-anvil-04 ccs -f
> > /etc/cluster/cluster.conf --setcman two_node="1" expected_votes="1"
> > ccs -f /etc/cluster/cluster.conf --addnode an-a04n01.alteeve.ca ccs -f
> > /etc/cluster/cluster.conf --addnode an-a04n02.alteeve.ca ccs -f
> > /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk ccs -f
> > /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n01.alteeve.ca
> > ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-
> > a04n02.alteeve.ca ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-
> > a04n01.alteeve.ca pcmk-redirect port=an-a04n01.alteeve.ca ccs -f
> > /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n02.alteeve.ca pcmk-
> > redirect port=an-a04n02.alteeve.ca ccs -f /etc/cluster/cluster.conf --
> > setfencedaemon post_join_delay="30"
> > cat /etc/cluster/cluster.conf
> > 
> > 
> >
> >
> >  
> >
> >  
> >
> >  
> >
> >  
> >  
> >
> >  
> >
> >  
> >
> >  
> >
> >
> >
> >  
> >
> >
> >  
> >  
> >
> > 
> > 
> 
> [Alex Samad - Yieldbroker] 
> Actually I do, I just put the initial bit in there.
> 
> > 
> > Then you move over to pacemaker and configure stonith there. How you do
> > this will vary a bit for your fence agent. I use IPMI fencing, which looks 
> > like
> > this:
> > 
> > 
> > pcs cluster cib stonith_cfg
> > pcs -f stonith_cfg stonith create fence_n01_ipmi fence_ipmilan
> > pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi"
> > action="reboot" login="admin" passwd="Initial1" delay=15 op monitor
> > interval=10s pcs -f stonith_cfg stonith create fence_n02_ipmi fence_ipmilan
> > pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi"
> > action="reboot" login="admin" passwd="Initial1" op monitor interval=10s
> > pcs cluster cib-push stonith_cfg pcs property set stonith-enabled=true 
> [Alex Samad - Yieldbroker] 
> 
> Problem is that uses ipmi I don't want to use that.
> 
> I would like to translater a stonight shutdown/reboot into a OS reboot 
> commend.
> 

That hardly makes sense except in pure test environment. Stonith is
needed when you do not know state of partner node, in which case you
cannot be sure your reboot/shutdown command will be executed, nor that
you can reach your partner at all.

If you are running under Vmware, use stonith/vmware or stonith/vcenter.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] pacemaker counts probe failure twice

[Andrei Borzenkov]

В Thu, 30 Oct 2014 08:32:24 +1100
Andrew Beekhof  пишет:

> 
> > On 29 Oct 2014, at 10:01 pm, Andrei Borzenkov  wrote:
> > 
> > I observe strange behavior that I cannot understand. Pacemaker 
> > 1.1.11-3ca8c3b.
> > 
> > There is master/slave resource running. Maintenance-mode was set,
> > pacemaker restarted, maintenance-mode reset. This specific RA returns
> > Slave instead of Master for the first probe. But what happens later is
> > rather strange.
> > 
> > So on startup pacemaker initiates probe:
> > 
> > Oct 29 11:25:59 n1 crmd[2879]:   notice: te_rsc_command: Initiating
> > action 3: monitor rsc_SAPHana_HDB_HDB00:0_monitor_0 on n1 (local)
> > Oct 29 11:25:59 n1 crmd[2879]:   notice: te_rsc_command: Initiating
> > action 4: monitor rsc_SAPHanaTopology_HDB_HDB00:0_monitor_0 on n1
> > (local)
> > 
> > That's fine. Agents return OK (correctly for clone
> > rsc_SAPHanaTopology_HDB, may be incorrectly for master/slave
> > rsc_SAPHana_HDB):
> > 
> > Oct 29 11:26:03 n1 crmd[2879]:   notice: process_lrm_event: LRM
> > operation rsc_SAPHanaTopology_HDB_HDB00_monitor_0 (call=12, rc=0,
> > cib-update=49, confirmed=true) ok
> > Oct 29 11:26:03 n1 crmd[2879]:   notice: process_lrm_event: LRM
> > operation rsc_SAPHana_HDB_HDB00_monitor_0 (call=7, rc=0,
> > cib-update=50, confirmed=true) ok
> > 
> > But pacemaker suddenly counts each of them twice? Why?
> 
> From the looks of it, the crmd is getting the updates from the CIB twice.
> It looks like this is a SUSE install right? 
> 
> Probably best to poke them directly about this, they will know exactly what 
> went into their build, will be able to compare it to the current upstream and 
> backport patches as needed.

Well, the question was more whether this is expected behavior that I do
not understand or it is something unexpected. I take it from your
answer that it is unexpected?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] pacemaker counts probe failure twice

[Andrei Borzenkov]

I observe strange behavior that I cannot understand. Pacemaker 1.1.11-3ca8c3b.

There is master/slave resource running. Maintenance-mode was set,
pacemaker restarted, maintenance-mode reset. This specific RA returns
Slave instead of Master for the first probe. But what happens later is
rather strange.

So on startup pacemaker initiates probe:

Oct 29 11:25:59 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 3: monitor rsc_SAPHana_HDB_HDB00:0_monitor_0 on n1 (local)
Oct 29 11:25:59 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 4: monitor rsc_SAPHanaTopology_HDB_HDB00:0_monitor_0 on n1
(local)

That's fine. Agents return OK (correctly for clone
rsc_SAPHanaTopology_HDB, may be incorrectly for master/slave
rsc_SAPHana_HDB):

Oct 29 11:26:03 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHanaTopology_HDB_HDB00_monitor_0 (call=12, rc=0,
cib-update=49, confirmed=true) ok
Oct 29 11:26:03 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_monitor_0 (call=7, rc=0,
cib-update=50, confirmed=true) ok

But pacemaker suddenly counts each of them twice? Why?

Oct 29 11:26:03 n1 crmd[2879]:  warning: status_from_rc: Action 4
(rsc_SAPHanaTopology_HDB_HDB00:0_monitor_0) on n1 failed (target: 7
vs. rc: 0): Error
Oct 29 11:26:03 n1 crmd[2879]:  warning: status_from_rc: Action 4
(rsc_SAPHanaTopology_HDB_HDB00:0_monitor_0) on n1 failed (target: 7
vs. rc: 0): Error

Oct 29 11:26:03 n1 crmd[2879]:  warning: status_from_rc: Action 3
(rsc_SAPHana_HDB_HDB00:0_monitor_0) on n1 failed (target: 7 vs. rc:
0): Error
Oct 29 11:26:03 n1 crmd[2879]:  warning: status_from_rc: Action 3
(rsc_SAPHana_HDB_HDB00:0_monitor_0) on n1 failed (target: 7 vs. rc:
0): Error

The maintenance mode gets reset and pacemaker starts monitor

Oct 29 11:32:06 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 29: monitor rsc_SAPHana_HDB_HDB00_monitor_61000 on n1 (local)

which now properly indicates Master status

Oct 29 11:32:09 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_monitor_61000 (call=55, rc=8,
cib-update=110, confirmed=false) master

which is being interpreted as error due to unexpectedness?

Oct 29 11:32:09 n1 crmd[2879]:  warning: status_from_rc: Action 29
(rsc_SAPHana_HDB_HDB00_monitor_61000) on n1 failed (target: 0 vs. rc:
8): Error
Oct 29 11:32:09 n1 crmd[2879]:  warning: update_failcount: Updating
failcount for rsc_SAPHana_HDB_HDB00 on n1 after failed monitor: rc=8
(update=value++, time=1414571529)
Oct 29 11:32:09 n1 attrd[2877]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-rsc_SAPHana_HDB_HDB00
(1)

Fine, so far it is logical. But where second error and increase
failure count suddenly comes from? We started just one monitor so far
...


Oct 29 11:32:09 n1 attrd[2877]:   notice: attrd_perform_update: Sent
update 33: fail-count-rsc_SAPHana_HDB_HDB00=1
Oct 29 11:32:09 n1 attrd[2877]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-rsc_SAPHana_HDB_HDB00
(1414571529)
Oct 29 11:32:09 n1 attrd[2877]:   notice: attrd_perform_update: Sent
update 35: last-failure-rsc_SAPHana_HDB_HDB00=1414571529
Oct 29 11:32:09 n1 attrd[2877]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-rsc_SAPHana_HDB_HDB00
(2)

Now it starts recovery by stopping and then starting master.

Oct 29 11:32:09 n1 pengine[2878]:   notice: LogActions: Recover
rsc_SAPHana_HDB_HDB00:0 (Master n1)
Oct 29 11:32:09 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 38: demote rsc_SAPHana_HDB_HDB00_demote_0 on n1 (local)
Oct 29 11:32:10 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_demote_0 (call=60, rc=0,
cib-update=116, confirmed=true) ok
Oct 29 11:32:10 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 6: stop rsc_SAPHana_HDB_HDB00_stop_0 on n1 (local)
Oct 29 11:32:21 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_stop_0 (call=61, rc=0, cib-update=118,
confirmed=true) ok
Oct 29 11:32:21 n1 pengine[2878]:   notice: LogActions: Start
rsc_SAPHana_HDB_HDB00:1 (n1)
Oct 29 11:32:21 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 38: start rsc_SAPHana_HDB_HDB00:1_start_0 on n1 (local)
Oct 29 11:33:12 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_start_0 (call=62, rc=0,
cib-update=121, confirmed=true) ok
Oct 29 11:33:12 n1 pengine[2878]:   notice: LogActions: Promote
rsc_SAPHana_HDB_HDB00:0 (Slave -> Master n1)
Oct 29 11:33:12 n1 crmd[2879]:   notice: te_rsc_command: Initiating
action 38: promote rsc_SAPHana_HDB_HDB00_promote_0 on n1 (local
Oct 29 11:33:14 n1 crmd[2879]:   notice: process_lrm_event: LRM
operation rsc_SAPHana_HDB_HDB00_promote_0 (call=63, rc=0,
cib-update=124, confirmed=true) ok

So we successfully restarted and promoted local instance. And suddenly ...

Oct 29 11:33:14 n1 pengine[2878]:  warning: unpack_rsc_op_failure:
Processing failed op monitor for rsc

Re: [Pacemaker] Split brain and STONITH behavior (VMware fencing)

[Andrei Borzenkov]

On Wed, Oct 29, 2014 at 10:46 AM, Ariel S  wrote:
> Hello,
>
> I'm trying to understand how this STONITH works.
>
> I have 2 VMware VMs (moon1a, moon1b) on two different hosts. Each have 2 nic
> assigned: eth0 for heartbeat while eth1 used for everything else.
>
> This is my testing configuration:
>
> node $id="168428034" moon1a
> node $id="168428035" moon1b
> primitive Foo ocf:heartbeat:Dummy
> primitive stonith_moon1a stonith:fence_vmware_soap \
> params ipaddr="192.168.1.134" login="foo" \
> uuid="42053b22-d3fd-25fe-6fb3-7cb2c7cd2c63" \
> action="off" verbose="true" passwd="bar" \
> ssl="true" \
> op monitor interval="60s"
> primitive stonith_moon1b stonith:fence_vmware_soap \
> params ipaddr="192.168.1.134" login="foo" \
> uuid="4205b986-4426-5de4-1069-b10a77123bc4" \
> action="off" verbose="true" passwd="bar" \
> ssl="true" \
> op monitor interval="60s"
> clone FooClones Foo
> location loc_stonith_moon1a stonith_moon1a -inf: moon1a
> location loc_stonith_moon1b stonith_moon1b -inf: moon1b
> property $id="cib-bootstrap-options" \
> dc-version="1.1.10-42f2063" \
> cluster-infrastructure="corosync" \
> stonith-enabled="true" \
> last-lrm-refresh="1414565715"
> rsc_defaults $id="rsc-options" \
> resource-stickiness="200"
>
>
> The vCenter is at 192.168.1.134 and the uuids taken from a list generated by
> fence_vmware_soap.
>
> When I do fencing manually using:
>
> # fence_vmware_soap -z -a 192.168.1.134 \
> -l foo -p bar \
> -U 4205b986-4426-5de4-1069-b10a77123bc4 \
> -o off
>
> from moon1a, as expected the moon1b (4205b986-4426-5de4-1069-b10a77123bc4)
> VM
> died, so the configuration should be right, I think.
>
> But so far I cant emulate split brain by killing corosync like this:
>
> # killall -9 corosync
>

Killing corosync is not strictly speaking split-brain, it is emulation
of (partial) node failure.

>
> My questions:
>
> 1.Is my configuration correct?

On two node cluster you also need no-quorum-policy=ignore, otherwise
remaining node won't initiate fencing.

> 2.How one cause a split-brain to trigger the expected stonith
> behavior?
>
>
>
> Thank you,
> Ariel
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Notification messages - 1150184

[Andrei Borzenkov]

В Tue, 28 Oct 2014 14:31:17 +1100
Andrew Beekhof  пишет:

> 
> > On 28 Oct 2014, at 7:36 am, Neale Ferguson  wrote:
> > 
> > I am getting loads of these types of messages in syslog:
> > 
> > Oct 27 16:23:47 rh7cn1 systemd: pacemaker.service: Got notification
> > message from PID 43007, but reception only permitted for PID 17131
> > 
> > 
> > PID 43007 refers to the https resource and 17131 to pacemakerd.
> > 
> > A search on this type of message shows a similar question on this list
> > from March 2014 but no resolution. The RH Bugzilla system has 1150184 in
> > the system that was last updated Oct 7 but it is restricted, so I was
> > wondering if anyone knows of the progress on this issue?
> 
> httpd.service uses 'Type=notify'
> 
> 
> Type=
>   Configures the process start-up type for this service unit. One of simple, 
> forking, oneshot, dbus, notify or idle.
> 
> 
>   Behavior of notify is similar to simple; however, it is expected that the 
> daemon sends a notification message via sd_notify(3) or an equivalent call 
> when it has finished starting up. systemd will proceed with starting 
> follow-up units after this notification message has been sent. If this option 
> is used, NotifyAccess= (see below) should be set to open access to the 
> notification socket provided by systemd. If NotifyAccess= is not set, it will 
> be implicitly set to main.
> 
>   If set to simple (the default value if neither Type= nor BusName= are 
> specified), it is expected that the process configured with ExecStart= is the 
> main process of the service. In this mode, if the process offers 
> functionality to other processes on the system, its communication channels 
> should be installed before the daemon is started up (e.g. sockets set up by 
> systemd, via socket activation), as systemd will immediately proceed starting 
> follow-up units.
> 
> 
> Looking at the code, the message seems to be an older version of this:
> 
> ./src/core/service.c-2553-if (s->notify_access == NOTIFY_MAIN && pid 
> != s->main_pid) {
> ./src/core/service.c-2554-if (s->main_pid != 0)
> ./src/core/service.c:2555:log_warning_unit(u->id, 
> "%s: Got notification message from PID "PID_FMT", but reception only 
> permitted for main PID "PID_FMT, u->id, pid, s->main_pid);
> 
> So rephrasing:
> 
>Got notification message from httpd, but reception only permitted for 
> pacemakerd
> 
> So it seems that systemd is calculating the wrong value for main_pid
> 

It's also possible that httpd is defined as OCF resource instead of
systemd ...

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] How to find out why pacemaker skipped action?

[Andrei Borzenkov]

On Mon, Oct 27, 2014 at 12:40 PM, Andrew Beekhof  wrote:
>
>> On 27 Oct 2014, at 7:36 pm, Andrei Borzenkov  wrote:
>>
>> On Wed, Oct 22, 2014 at 8:59 AM, Andrew Beekhof  wrote:
>>>
>>>> On 22 Oct 2014, at 4:34 pm, Andrei Borzenkov  wrote:
>>>>
>>>> On Wed, Oct 22, 2014 at 8:01 AM, Andrew Beekhof  wrote:
>>>>>
>>>>>> On 21 Oct 2014, at 11:15 pm, Andrei Borzenkov  
>>>>>> wrote:
>>>>>>
>>>>>> Pacemaker 1.1.11. I see in engine logs that it is going to restart 
>>>>>> resource:
>>>>>>
>>>>>> Oct 21 12:34:50 n2 pengine[19748]:   notice: LogActions: Restart
>>>>>> rsc_SAPHana_HDB_HDB00:0 (Master n2)
>>>>>>
>>>>>> But I never see actual stop/start action being executed and in summary I 
>>>>>> get
>>>>>>
>>>>>> Oct 21 12:35:11 n2 crmd[19749]:   notice: run_graph: Transition 32
>>>>>> (Complete=10, Pending=0, Fired=0, Skipped=13, Incomplete=3,
>>>>>> Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Stopped
>>>>>>
>>>>>> So 13 actions were skipped and I presume restart was among them.
>>>>>>
>>>>>> In which logs can I find explanation why actions were skipped? I do
>>>>>> not see anything obvious.
>>>>>
>>>>> Do you see any actions failing?
>>>>
>>>> Yes
>>>>
>>>> Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
>>>> (rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
>>>> vs. rc: 0): Error
>>>> Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
>>>> (rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
>>>> vs. rc: 0): Error
>>>>
>>>> Now there is the following ordering:
>>>>
>>>> order ord_SAPHana_HDB_HDB00 2000: cln_SAPHanaTopology_HDB_HDB00
>>>> msl_SAPHana_HDB_HDB00
>>>>
>>>>> Further up the crmd should have said why the transaction is being aborted
>>>>>
>>>>
>>>> If it says it, I do not yet understand it.
>>>>
>>>> Am I right that if any action during transaction returns unexpected
>>>> result, transaction is aborted?
>>>
>>> Yes, and a new one calculated
>>
>> Do CIB updates also abort running transaction?
>
> Unexpected ones do, yes.
>

OK, monitoring script does "crm_attribute -N $(uname -n) -n foo -v
bar". Is it expected?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] How to find out why pacemaker skipped action?

[Andrei Borzenkov]

On Wed, Oct 22, 2014 at 8:59 AM, Andrew Beekhof  wrote:
>
>> On 22 Oct 2014, at 4:34 pm, Andrei Borzenkov  wrote:
>>
>> On Wed, Oct 22, 2014 at 8:01 AM, Andrew Beekhof  wrote:
>>>
>>>> On 21 Oct 2014, at 11:15 pm, Andrei Borzenkov  wrote:
>>>>
>>>> Pacemaker 1.1.11. I see in engine logs that it is going to restart 
>>>> resource:
>>>>
>>>> Oct 21 12:34:50 n2 pengine[19748]:   notice: LogActions: Restart
>>>> rsc_SAPHana_HDB_HDB00:0 (Master n2)
>>>>
>>>> But I never see actual stop/start action being executed and in summary I 
>>>> get
>>>>
>>>> Oct 21 12:35:11 n2 crmd[19749]:   notice: run_graph: Transition 32
>>>> (Complete=10, Pending=0, Fired=0, Skipped=13, Incomplete=3,
>>>> Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Stopped
>>>>
>>>> So 13 actions were skipped and I presume restart was among them.
>>>>
>>>> In which logs can I find explanation why actions were skipped? I do
>>>> not see anything obvious.
>>>
>>> Do you see any actions failing?
>>
>> Yes
>>
>> Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
>> (rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
>> vs. rc: 0): Error
>> Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
>> (rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
>> vs. rc: 0): Error
>>
>> Now there is the following ordering:
>>
>> order ord_SAPHana_HDB_HDB00 2000: cln_SAPHanaTopology_HDB_HDB00
>> msl_SAPHana_HDB_HDB00
>>
>>> Further up the crmd should have said why the transaction is being aborted
>>>
>>
>> If it says it, I do not yet understand it.
>>
>> Am I right that if any action during transaction returns unexpected
>> result, transaction is aborted?
>
> Yes, and a new one calculated

Do CIB updates also abort running transaction? It looks like it, but
I'd like to be sure.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Stopping/restarting pacemaker without stopping resources?

[Andrei Borzenkov]

On Mon, Oct 27, 2014 at 6:34 AM, Andrew Beekhof  wrote:
>
>> On 27 Oct 2014, at 2:30 pm, Andrei Borzenkov  wrote:
>>
>> В Mon, 27 Oct 2014 11:09:08 +1100
>> Andrew Beekhof  пишет:
>>
>>>
>>>> On 25 Oct 2014, at 12:38 am, Andrei Borzenkov  wrote:
>>>>
>>>> On Fri, Oct 24, 2014 at 9:17 AM, Andrew Beekhof  wrote:
>>>>>
>>>>>> On 16 Oct 2014, at 9:31 pm, Andrei Borzenkov  wrote:
>>>>>>
>>>>>> The primary goal is to transparently update software in cluster. I
>>>>>> just did HA suite update using simple RPM and observed that RPM
>>>>>> attempts to restart stack (rcopenais try-restart). So
>>>>>>
>>>>>> a) if it worked, it would mean resources had been migrated from this
>>>>>> node - interruption
>>>>>>
>>>>>> b) it did not work - apparently new versions of installed utils were
>>>>>> incompatible with running pacemaker so request to shutdown crm fails
>>>>>> and openais hung forever.
>>>>>>
>>>>>> The usual workflow with one cluster products I worked before was -
>>>>>> stop cluster processes without stopping resources; update; restart
>>>>>> cluster processes. They would detect that resources are started and
>>>>>> return to the same state as before stopping. Is something like this
>>>>>> possible with pacemaker?
>>>>>
>>>>> absolutely.  this should be of some help:
>>>>>
>>>>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_disconnect_and_reattach.html
>>>>>
>>>>
>>>> Did not work. It ended up moving master to another node and leaving
>>>> slave on original node stopped after that.
>>>
>>> When you stopped the cluster or when you started it after an upgrade?
>>
>> When I started it
>>
>> crm_attribute -t crm_config -n is-managed-default -v false
>> rcopenais stop on both nodes
>> rcopenais start on both node; wait for them to stabilize
>> crm_attribute -t crm_config -n is-managed-default -v true
>>
>> It stopped running master/slave, moved master and left slave stopped.
>
> What did crm_mon say before you set is-managed-default back to true?
> Did the resource agent properly detect it as running in the master state?

You are right, it returned 0, not 8.

> Did the resource agent properly (re)set a preference for being promoted 
> during the initial monitor operation?
>

It did, but it was too late - after it had already been demoted.

> Pacemaker can do it, but it is dependant on the resources behaving correctly.
>

I see.

Well, this would be a problem ... RA keeps track of current
promoted/demoted status in CIB as transient attribute which gets reset
after reboot. This would entail quite a bit of redesign ...

But what got me confused were these errors during initial probing, like

Oct 24 17:26:54 n1 crmd[32425]:  warning: status_from_rc: Action 9
(rsc_ip_VIP_monitor_0) on n2 failed (target: 7 vs. rc: 0): Error

This looks like pacemaker does expect resource to be in stopped state
and "running" state would be interpreted as error? I mean, normal
response to such monitor response would be to stop resource to bring
it in target state, no?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Stopping/restarting pacemaker without stopping resources?

[Andrei Borzenkov]

В Mon, 27 Oct 2014 11:09:08 +1100
Andrew Beekhof  пишет:

> 
> > On 25 Oct 2014, at 12:38 am, Andrei Borzenkov  wrote:
> > 
> > On Fri, Oct 24, 2014 at 9:17 AM, Andrew Beekhof  wrote:
> >> 
> >>> On 16 Oct 2014, at 9:31 pm, Andrei Borzenkov  wrote:
> >>> 
> >>> The primary goal is to transparently update software in cluster. I
> >>> just did HA suite update using simple RPM and observed that RPM
> >>> attempts to restart stack (rcopenais try-restart). So
> >>> 
> >>> a) if it worked, it would mean resources had been migrated from this
> >>> node - interruption
> >>> 
> >>> b) it did not work - apparently new versions of installed utils were
> >>> incompatible with running pacemaker so request to shutdown crm fails
> >>> and openais hung forever.
> >>> 
> >>> The usual workflow with one cluster products I worked before was -
> >>> stop cluster processes without stopping resources; update; restart
> >>> cluster processes. They would detect that resources are started and
> >>> return to the same state as before stopping. Is something like this
> >>> possible with pacemaker?
> >> 
> >> absolutely.  this should be of some help:
> >> 
> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_disconnect_and_reattach.html
> >> 
> > 
> > Did not work. It ended up moving master to another node and leaving
> > slave on original node stopped after that.
> 
> When you stopped the cluster or when you started it after an upgrade?

When I started it

crm_attribute -t crm_config -n is-managed-default -v false
rcopenais stop on both nodes
rcopenais start on both node; wait for them to stabilize
crm_attribute -t crm_config -n is-managed-default -v true

It stopped running master/slave, moved master and left slave stopped.

> 
> > I have crm_report if this
> > is not expected ...
> > 
> > pacemaker Version: 1.1.11-3ca8c3b
> > 
> > ___
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] split brain - after network recovery - resources can still be migrated

[Andrei Borzenkov]

В Sun, 26 Oct 2014 12:01:03 +0100
Vladimir  пишет:

> On Sat, 25 Oct 2014 19:11:02 -0400
> Digimer  wrote:
> 
> > On 25/10/14 06:35 PM, Vladimir wrote:
> > > On Sat, 25 Oct 2014 17:30:07 -0400
> > > Digimer  wrote:
> > >
> > >> On 25/10/14 05:09 PM, Vladimir wrote:
> > >>> Hi,
> > >>>
> > >>> currently I'm testing a 2 node setup using ubuntu trusty.
> > >>>
> > >>> # The scenario:
> > >>>
> > >>> All communication links betwenn the 2 nodes are cut off. This
> > >>> results in a split brain situation and both nodes take their
> > >>> resources online.
> > >>>
> > >>> When the communication links get back, I see following behaviour:
> > >>>
> > >>> On drbd level the split brain is detected and the device is
> > >>> disconnected on both nodes because of "after-sb-2pri disconnect"
> > >>> and then it goes to StandAlone ConnectionState.
> > >>>
> > >>> I'm wondering why pacemaker does not let the resources fail.
> > >>> It is still possible to migrate resources between the nodes
> > >>> although they're in StandAlone ConnectionState. After a split
> > >>> brain that's not what I want.
> > >>>
> > >>> Is this the expected behaviour? Is it possible to let the
> > >>> resources fail after the network recovery to avoid fürther data
> > >>> corruption.
> > >>>
> > >>> (At the moment I can't use resource or node level fencing in my
> > >>> setup.)
> > >>>
> > >>> Here the main part of my config:
> > >>>
> > >>> #> dpkg -l | awk '$2 ~ /^(pacem|coro|drbd|libqb)/{print $2,$3}'
> > >>> corosync 2.3.3-1ubuntu1
> > >>> drbd8-utils 2:8.4.4-1ubuntu1
> > >>> libqb-dev 0.16.0.real-1ubuntu3
> > >>> libqb0 0.16.0.real-1ubuntu3
> > >>> pacemaker 1.1.10+git20130802-1ubuntu2.1
> > >>> pacemaker-cli-utils 1.1.10+git20130802-1ubuntu2.1
> > >>>
> > >>> # pacemaker
> > >>> primitive drbd-mysql ocf:linbit:drbd \
> > >>> params drbd_resource="mysql" \
> > >>> op monitor interval="29s" role="Master" \
> > >>> op monitor interval="30s" role="Slave"
> > >>>
> > >>> ms ms-drbd-mysql drbd-mysql \
> > >>> meta master-max="1" master-node-max="1" clone-max="2"
> > >>> clone-node-max="1" notify="true"
> > >>
> > >> Split-brains are prevented by using reliable fencing (aka stonith).
> > >> You configure stonith in pacemaker (using IPMI/iRMC/iLO/etc,
> > >> switched PDUs, etc). Then you configure DRBD to use the
> > >> crm-fence-peer.sh fence-handler and you set the fencing policy to
> > >> 'resource-and-stonith;'.
> > >>
> > >> This way, if all links fail, both nodes block and call a fence. The
> > >> faster one fences (powers off) the slower, and then it begins
> > >> recovery, assured that the peer is not doing the same.
> > >>
> > >> Without stonith/fencing, then there is no defined behaviour. You
> > >> will get split-brains and that is that. Consider; Both nodes lose
> > >> contact with it's peer. Without fencing, both must assume the peer
> > >> is dead and thus take over resources.
> > >
> > > That split brains can occur in such a setup that's clear. But I
> > > would expect pacemaker to stop the drbd resource when the link
> > > between the cluster nodes is reestablished instead of continue
> > > running it.
> > 
> > DRBD will refuse to reconnect until it is told which node's data to 
> > delete. This is data loss and can not be safely automated.
> 
> Sorry if maybe described it unclear but I don't want pacemaker to do an
> automatic split brain recovery. That would not make any sense to me
> either. This decission has to be taken by an administrator.
> 
> But is it possible to configure pacemaker to do the following?
>  
> - if there are 2 Nodes which can see and communicate with each other
>   AND
> - if their disk state is not UpToDate/UpToDate (typically after a split
>   brain)
> - then let drbd resource fail because something is obviously broken and
>   an administrator has to decide how to continue.
> 

This would require resource agent support. But it looks like current
resource agent relies on fencing to resolve split brain situation. As
long as resource agent itself does not indicate resource failure,
there is nothing pacemaker can do. 


> > >> This is why stonith is required in clusters. Even with quorum, you
> > >> can't assume anything about the state of the peer until it is
> > >> fenced, so it would only give you a false sense of security.
> > >
> > > Maybe I'll can use resource level fencing.
> > 
> > You need node-level fencing.
> 
> I know node level fencing is more secure. But shouldn't resource level
> fencing also work here? e.g.
> (http://www.drbd.org/users-guide/s-pacemaker-fencing.html) 
> 
> Currently I can't use ipmi, apc switches or a shared storage device
> for fencing at most fencing via ssh. But what I've read this is also not
> recommended for production setups.
> 

You could try meatware stonith agent. This does exactly what you want -
it freezes further processing unless administrator manually declares one
node as down. 

___
Pacemaker mailing list: Pacemaker@oss

Re: [Pacemaker] MySQL, Percona replication manager - split brain

[Andrei Borzenkov]

В Sun, 26 Oct 2014 10:51:13 +0200
Andrew  пишет:

> 26.10.2014 08:32, Andrei Borzenkov пишет:
> > В Sat, 25 Oct 2014 23:34:54 +0300
> > Andrew  пишет:
> >
> >> 25.10.2014 22:34, Digimer пишет:
> >>> On 25/10/14 03:32 PM, Andrew wrote:
> >>>> Hi all.
> >>>>
> >>>> I use Percona as RA on cluster (nothing mission-critical, currently -
> >>>> just zabbix data); today after restarting MySQL resource (crm resource
> >>>> restart p_mysql) I've got a split brain state - MySQL for some reason
> >>>> started first at ex-slave node, ex-master starts later (possibly I've
> >>>> set too small timeout to shutdown - only 120s, but I'm not sure).
> >>>>

Your logs do not show resource restart - they show pacemaker restart on
node2.

> >>>> After restart resource on both nodes it seems like mysql replication was
> >>>> ok - but then after ~50min it fails in split brain again for unknown
> >>>> reason (no resource restart was noticed).
> >>>>
> >>>> In 'show replication status' there is an error in table caused by unique
> >>>> index dup.
> >>>>
> >>>> So I have a questions:
> >>>> 1) Which thing causes split brain, and how to avoid it in future?
> >>> Cause:
> >>>
> >>> Logs?
> >> ct 25 13:54:13 node2 crmd[29248]:   notice: do_state_transition: State
> >> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> >> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> >> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_config: On loss
> >> of CCM Quorum: Ignore
> >> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_rsc_op: Operation
> >> monitor found resource p_pgsql:0 active in master mode on node1.cluster
> >> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_rsc_op: Operation
> >> monitor found resource p_mysql:1 active in master mode on node2.cluster
> > That seems too late. The real cause is that resource was reported as
> > being in master state on both nodes and this happened earlier.
> This is a different resources (pgsql and mysql)/
> 
> >>> Prevent:
> >>>
> >>> Fencing (aka stonith). This is why fencing is required.
> >> No node failure. Just daemon was restarted.
> >>
> > "Split brain" == loss of communication. It does not matter whether
> > communication was lost because node failed or because daemon was not
> > running. There is no way for surviving node to know, *why*
> > communication was lost.
> >
> So how stonith will help in this case? Daemon will be restarted after 
> it's death if it occures during restart, and stonith will see alive 
> daemon...
> 
> So what is the easiest split-brain solution? Just to stop daemons, and 
> copy all mysql data from good node to bad one?

There is no split-brain visible in your log. Pacemaker on node2 was
restarted, cleanly as far as I can tell, and reintegrated back in
cluster. May be node1 "lost" node2, but that needs logs from node1.

You probably misuse "split brain" in this case. Split-brain means -
nodes lost communication with each other, so each node is unaware of
in which state resources on other node are. Here "nodes" means
corosync/pacemaker. Not individual resources.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] MySQL, Percona replication manager - split brain

[Andrei Borzenkov]

В Sat, 25 Oct 2014 23:34:54 +0300
Andrew  пишет:

> 25.10.2014 22:34, Digimer пишет:
> > On 25/10/14 03:32 PM, Andrew wrote:
> >> Hi all.
> >>
> >> I use Percona as RA on cluster (nothing mission-critical, currently -
> >> just zabbix data); today after restarting MySQL resource (crm resource
> >> restart p_mysql) I've got a split brain state - MySQL for some reason
> >> started first at ex-slave node, ex-master starts later (possibly I've
> >> set too small timeout to shutdown - only 120s, but I'm not sure).
> >>
> >> After restart resource on both nodes it seems like mysql replication was
> >> ok - but then after ~50min it fails in split brain again for unknown
> >> reason (no resource restart was noticed).
> >>
> >> In 'show replication status' there is an error in table caused by unique
> >> index dup.
> >>
> >> So I have a questions:
> >> 1) Which thing causes split brain, and how to avoid it in future?
> >
> > Cause:
> >
> > Logs?
> ct 25 13:54:13 node2 crmd[29248]:   notice: do_state_transition: State 
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_config: On loss 
> of CCM Quorum: Ignore
> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_rsc_op: Operation 
> monitor found resource p_pgsql:0 active in master mode on node1.cluster
> Oct 25 13:54:13 node2 pengine[29247]:   notice: unpack_rsc_op: Operation 
> monitor found resource p_mysql:1 active in master mode on node2.cluster

That seems too late. The real cause is that resource was reported as
being in master state on both nodes and this happened earlier.
> 
> >
> > Prevent:
> >
> > Fencing (aka stonith). This is why fencing is required.
> No node failure. Just daemon was restarted.
> 

"Split brain" == loss of communication. It does not matter whether
communication was lost because node failed or because daemon was not
running. There is no way for surviving node to know, *why*
communication was lost.

> >
> >> 2) How to resolve split brain state? Is it enough just to wait for
> >> failure, then - restart mysql by hand and clean row with dup index in
> >> slave db, and then run resource again? Or there is some automation for
> >> such cases?
> >
> > How are you sharing data? Can you give us a better understanding of 
> > your setup?
> >
> Semi-synchronous MySQL replication, if you mean sharing DB log between 
> nodes.
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Stopping/restarting pacemaker without stopping resources?

[Andrei Borzenkov]

On Fri, Oct 24, 2014 at 9:17 AM, Andrew Beekhof  wrote:
>
>> On 16 Oct 2014, at 9:31 pm, Andrei Borzenkov  wrote:
>>
>> The primary goal is to transparently update software in cluster. I
>> just did HA suite update using simple RPM and observed that RPM
>> attempts to restart stack (rcopenais try-restart). So
>>
>> a) if it worked, it would mean resources had been migrated from this
>> node - interruption
>>
>> b) it did not work - apparently new versions of installed utils were
>> incompatible with running pacemaker so request to shutdown crm fails
>> and openais hung forever.
>>
>> The usual workflow with one cluster products I worked before was -
>> stop cluster processes without stopping resources; update; restart
>> cluster processes. They would detect that resources are started and
>> return to the same state as before stopping. Is something like this
>> possible with pacemaker?
>
> absolutely.  this should be of some help:
>
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_disconnect_and_reattach.html
>

Did not work. It ended up moving master to another node and leaving
slave on original node stopped after that. I have crm_report if this
is not expected ...

pacemaker Version: 1.1.11-3ca8c3b

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Master-slave master not promoted on Corosync restart

[Andrei Borzenkov]

On Fri, Oct 24, 2014 at 2:00 PM, Sékine Coulibaly  wrote:
> Hi Andrew,
>
> Yep, forgot the attachments. I did reproduce the issue, please find
> the bz2 files attached. Please tell if you need hb_report being used.
>

You forgot to attach log matching these files.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] IPaddr resource agent on Illumos

[Andrei Borzenkov]

В Thu, 23 Oct 2014 17:51:24 +0200
Vincenzo Pii  пишет:

> I am trying to run the IPaddr resource agent on an active/passive cluster
> on Illumos nodes (pacemaker, corosync, crm... built from updated sources).
> 
> By reading the example from Saso here
> http://zfs-create.blogspot.ch/2013/06/building-zfs-storage-appliance-part-1.html,
> this would seem straightforward and this makes me think that I am doing
> something wrong :)!
> 
> I patched the IPaddr script to use /usr/bin/gnu/sh and to avoid finding a
> free interface with \" grep "^$NIC:[0-9]" \" as that is just not the case,
> but now I am stuck at trying to configure the ip address.
> 
> This, in the script, is done with ifconfig (something like
> 
> ifconfig e1000g2 inet 10.0.100.4 && ifconfig e1000g2 netmask
> 255.255.255.0 && ifconfig e1000g2 up
> 
> ).
> 
> However, the script is run by the hacluster user, which cannot write
> network configuration settings.
> 

Unless I'm completely confused, resource scripts are launched by lrmd
which runs as root.

> To solve this problem, I am now looking at profiles, roles and
> authorizations, which seems to be a very "user friendly" way to handle
> permissions in Solaris.
> 
> My question is: there is no mention of this in Saso's post, or other
> discussions (even thought old ones) that I've come across today; am I
> missing something obvious, or this is just the way it has to be?
> 
> This is how I configure the IPaddr prmitive:
> 
> # ipadm create-if e1000g2
> # crm configure primitive frontend_IP ocf:heartbeat:IPaddr params
> ip="10.0.100.4" cidr_netmask="255.255.255.0" nic="e1000g2"
> 
> Many thanks,
> Vincenzo.
> 


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] meta failure-timeout: crashed resource is assumed to be Started?

[Andrei Borzenkov]

В Thu, 23 Oct 2014 13:46:00 +0200
Carsten Otto  пишет:

> Dear all,
> 
> I did not get any response so far. Could you please find the time and
> tell me how the "meta failure-timeout" is supposed to work, in
> combination with monitor operations?
> 

If you attach unedited logs from the point of the FIRST failure as
well as your configuration you probably will get more chances.
Failure-timeout should have no relation to monitor operation; most
likely monitor actually indicates FIRST is running even when it is not. 

> Thanks,
> Carsten
> 
> On Thu, Oct 16, 2014 at 05:06:41PM +0200, Carsten Otto wrote:
> > Dear all,
> > 
> > I configured meta failure-timeout=60sec on all of my resources. For the
> > sake of simplicity, assume I have a group of two resources FIRST and
> > SECOND (where SECOND is started after FIRST, surprise!).
> > 
> > If now FIRST crashes, I see a failure, as expected. I also see that
> > SECOND is stopped, as expected.
> > 
> > Sadly, SECOND needs more than 60 seconds to stop. Thus, it can happen
> > that the "failure-timeout" for FIRST is reached, and its failure is
> > cleaned. This also is expected.
> > 
> > The problem now is that after the 60sec timeout pacemaker assumes that
> > FIRST is in the Started state. There is no indication about that in the
> > log files, and the last monitor operation which ran just a few seconds
> > before also indicated that FIRST is actually not running.
> > 
> > As a consequence of the bug, pacemaker tries to re-start SECOND on the
> > same system, which fails to start (as it depends on FIRST, which
> > actually is not running). Only then the resources are started on the
> > other system.
> > 
> > So, my question is:
> > Why does pacemaker assume that a previously failed resource is "Started"
> > when the "meta failure-timeout" is triggered? Why is the monitor
> > operation not invoked to determine the correct state?
> > 
> > The corresponding lines of the log file, about a minute after FIRST
> > crashed and the stop operation for SECOND was triggered:
> > 
> > Oct 16 16:27:20 [2100] HOSTNAME [...] (monitor operation indicating that 
> > FIRST is not running)
> > [...]
> > Oct 16 16:27:23 [2104] HOSTNAME   lrmd: info: log_finished: 
> > finished - rsc:SECOND action:stop call_id:123 pid:29314 exit-code:0 
> > exec-time:62827ms queue-time:0ms
> > Oct 16 16:27:23 [2107] HOSTNAME   crmd:   notice: process_lrm_event:
> > LRM operation SECOND_stop_0 (call=123, rc=0, cib-update=225, 
> > confirmed=true) ok
> > Oct 16 16:27:23 [2107] HOSTNAME   crmd: info: match_graph_event:
> > Action SECOND_stop_0 (74) confirmed on HOSTNAME (rc=0)
> > Oct 16 16:27:23 [2107] HOSTNAME   crmd:   notice: run_graph:
> > Transition 40 (Complete=5, Pending=0, Fired=0, Skipped=31, Incomplete=10, 
> > Source=/var/lib/pacemaker/pengine/pe-input-2937.bz2): Stopped
> > Oct 16 16:27:23 [2107] HOSTNAME   crmd: info: do_state_transition:  
> > State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC 
> > cause=C_FSA_INTERNAL origin=notify_crmd ]
> > Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request:  
> > Completed cib_modify operation for section status: OK (rc=0, 
> > origin=local/crmd/225, version=0.1450.89)
> > Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request:  
> > Completed cib_query operation for section 'all': OK (rc=0, 
> > origin=local/crmd/226, version=0.1450.89)
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine:   notice: unpack_config:
> > On loss of CCM Quorum: Ignore
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: 
> > determine_online_status_fencing:  Node HOSTNAME is active
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: 
> > determine_online_status:  Node HOSTNAME is online
> > [...]
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full:   
> > FIRST has failed 1 times on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine:   notice: unpack_rsc_op:
> > Clearing expired failcount for FIRST on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full:   
> > FIRST has failed 1 times on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine:   notice: unpack_rsc_op:
> > Clearing expired failcount for FIRST on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full:   
> > FIRST has failed 1 times on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine:   notice: unpack_rsc_op:
> > Clearing expired failcount for FIRST on HOSTNAME
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine:   notice: unpack_rsc_op:
> > Re-initiated expired calculated failure FIRST_last_failure_0 (rc=7, 
> > magic=0:7;68:31:0:28c68203-6990-48fd-96cc-09f86e2b21f9) on HOSTNAME
> > [...]
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: group_print:   
> > Resource Group: GROUP
> > Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: 

Re: [Pacemaker] How to find out why pacemaker skipped action?

[Andrei Borzenkov]

On Wed, Oct 22, 2014 at 8:01 AM, Andrew Beekhof  wrote:
>
>> On 21 Oct 2014, at 11:15 pm, Andrei Borzenkov  wrote:
>>
>> Pacemaker 1.1.11. I see in engine logs that it is going to restart resource:
>>
>> Oct 21 12:34:50 n2 pengine[19748]:   notice: LogActions: Restart
>> rsc_SAPHana_HDB_HDB00:0 (Master n2)
>>
>> But I never see actual stop/start action being executed and in summary I get
>>
>> Oct 21 12:35:11 n2 crmd[19749]:   notice: run_graph: Transition 32
>> (Complete=10, Pending=0, Fired=0, Skipped=13, Incomplete=3,
>> Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Stopped
>>
>> So 13 actions were skipped and I presume restart was among them.
>>
>> In which logs can I find explanation why actions were skipped? I do
>> not see anything obvious.
>
> Do you see any actions failing?

Yes

Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
(rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
vs. rc: 0): Error
Oct 21 12:35:10 n2 crmd[19749]:  warning: status_from_rc: Action 11
(rsc_SAPHanaTopology_HDB_HDB00:1_monitor_0) on n1 failed (target: 7
vs. rc: 0): Error

Now there is the following ordering:

order ord_SAPHana_HDB_HDB00 2000: cln_SAPHanaTopology_HDB_HDB00
msl_SAPHana_HDB_HDB00

> Further up the crmd should have said why the transaction is being aborted
>

If it says it, I do not yet understand it.

Am I right that if any action during transaction returns unexpected
result, transaction is aborted?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] How to find out why pacemaker skipped action?

[Andrei Borzenkov]

Pacemaker 1.1.11. I see in engine logs that it is going to restart resource:

Oct 21 12:34:50 n2 pengine[19748]:   notice: LogActions: Restart
rsc_SAPHana_HDB_HDB00:0 (Master n2)

But I never see actual stop/start action being executed and in summary I get

Oct 21 12:35:11 n2 crmd[19749]:   notice: run_graph: Transition 32
(Complete=10, Pending=0, Fired=0, Skipped=13, Incomplete=3,
Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Stopped

So 13 actions were skipped and I presume restart was among them.

In which logs can I find explanation why actions were skipped? I do
not see anything obvious.

TIA

-andrei

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Y should pacemaker be started simultaneously.

[Andrei Borzenkov]

В Mon, 06 Oct 2014 10:27:49 -0400
Digimer  пишет:

> On 06/10/14 02:11 AM, Andrei Borzenkov wrote:
> > On Mon, Oct 6, 2014 at 9:03 AM, Digimer  wrote:
> >> If stonith was configured, after the time out, the first node would fence
> >> the second node ("unable to reach" != "off").
> >>
> >> Alternatively, you can set corosync to 'wait_for_all' and have the first
> >> node do nothing until it sees the peer.
> >>
> >
> > Am I right that wait_for_all is available only in corosync 2.x and not in 
> > 1.x?
> 
> You are correct, yes.
> 
> >> To do otherwise would be to risk a split-brain. Each node needs to know the
> >> state of the peer in order to run services safely. By having both start at
> >> the same time, then they know what the other is doing. By disabling quorum,
> >> you allow one node to continue to operate when the other leaves, but it
> >> needs that initial connection to know for sure what it's doing.
> >>
> >
> > Does it apply to both corosync 1.x and 2.x or only to 2.x with
> > wait_for_all? Because I actually also was confused about precise
> > meaning of disabling quorum in pacemaker (setting no-quorum-policy:
> > ignore). So if I have two node cluster with pacemaker 1.x and corosync
> > 1.x with no-quorum-policy=ignore and no fencing - what happens when
> > one single node starts?
> 
> Quorum tells the cluster that if a peer leaves (gracefully or was 
> fenced), the remaining node is allowed to continue providing services.
> 
> Stonith is needed to put a node that is in an unknown state into a known 
> state; Be it because it couldn't reach the node when starting or because 
> the node stopped responding.
> 
> So quorum and stonith play rather different roles.
> 
> Without stonith, regardless of quorum, you risk split-brains and/or data 
> corruption. Operating a cluster without stonith is to operate a cluster 
> in an undermined state and should never be done.
> 

OK I try to rephrase. Is it possible to achieve the same effect as
wait_for_all in corosync 2.x with combination of pacemaker 1.1.x and
corosync 1.x? I.e. ensure that cluster does not come up *on the
first startup* until all nodes are present? So just make cluster nodes
wait for others to join instead of trying to stonith them?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Show all resource properties with crmsh

[Andrei Borzenkov]

В Tue, 14 Oct 2014 11:06:28 +0200
Dejan Muhamedagic  пишет:

> 
> Do you have any specific meta attributes in mind? Taking a
> cursory look doesn't reveal anything very interesting.
> 

I was recently bitten by "interleave". Having not much experience so
far, I do not know if there are more attributes that can influence
pacemaker decisions. I assume there are or that they may be added. 

> > >
> > > But why would you want to have this?
> > >
> > 
> > Because otherwise I must somehow find what defaults are and this may
> > change from release to release and from vendor to vendor. I simply
> > want to make sure I have all necessary information that may influence
> > cluster behavior.
> 
> We try hard to keep the defaults the same between releases, even
> when they're in retrospective considered to be less than optimal.
> Otherwise, the existing cluster configurations may fail. But
> there's no guarantee.
> 

Exactly :)

> It would be possible to implement a small utility which would
> compare defaults of two RA meta-data. There are also meta-data in
> crmd, pengine, cib, and stonithd, representing various global
> cluster properties.
> 

I'm not sure I understand, but probably simple use case. I get report
of a problem and get logs + cib. I need to understand why pacemaker
made this decision. For this I obviously must know exact configuration.
I simply want to avoid "assuming" anything.

So I'm not interesting in comparing defaults (although this would be
useful for update anyway - as pre-flight check to ensure configuration
won't change in incompatible way). I'm simply interested in getting
actual values without need to dig into sources of this specific version
I happen to work with. Something similar to corosync-cmapctl that dumps
full configuration.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Stopping/restarting pacemaker without stopping resources?

[Andrei Borzenkov]

The primary goal is to transparently update software in cluster. I
just did HA suite update using simple RPM and observed that RPM
attempts to restart stack (rcopenais try-restart). So

a) if it worked, it would mean resources had been migrated from this
node - interruption

b) it did not work - apparently new versions of installed utils were
incompatible with running pacemaker so request to shutdown crm fails
and openais hung forever.

The usual workflow with one cluster products I worked before was -
stop cluster processes without stopping resources; update; restart
cluster processes. They would detect that resources are started and
return to the same state as before stopping. Is something like this
possible with pacemaker?

TIA

-andrei

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Show all resource properties with crmsh

[Andrei Borzenkov]

On Thu, Oct 2, 2014 at 6:59 PM, Саша Александров  wrote:
> Andrei,
>
> I suspect that you think in a way 'if there is a default monitor
> interval value of 60s monitor operation should occur every 60
> seconds', correct?
>

Somewhere in this direction, yes, but not limited to operations.

> Well, this is not true: you have to define (add) all operations manually.
>

You mean - unless operation is explicitly defined in CIB pacemaker
won't try to use it for resource? That's not what I observed ...

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Show all resource properties with crmsh

[Andrei Borzenkov]

On Thu, Oct 2, 2014 at 6:31 PM, Dejan Muhamedagic  wrote:
> Hi,
>
> On Thu, Oct 02, 2014 at 12:22:35PM +0400, Andrei Borzenkov wrote:
>> Is it possible to display values for all resource properties,
>> including those set to default values?
>
> What do you consider a "property"? Instance attributes or meta
> attributes? Or both?

I was referring mostly to meta attributes, but having both would be
quite useful.

>   The defaults for the former live in the RA
> meta-data and are used only by crmsh (probably some other tools
> too) to display in brackets when showing the RA info. Note also
> that the default in the meta-data may not actually match the
> default used by the RA (it should, but there's no mechanism to
> make sure).
>

Yes, I understand that.

>
> But why would you want to have this?
>

Because otherwise I must somehow find what defaults are and this may
change from release to release and from vendor to vendor. I simply
want to make sure I have all necessary information that may influence
cluster behavior.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Y should pacemaker be started simultaneously.

[Andrei Borzenkov]

On Mon, Oct 6, 2014 at 9:03 AM, Digimer  wrote:
> If stonith was configured, after the time out, the first node would fence
> the second node ("unable to reach" != "off").
>
> Alternatively, you can set corosync to 'wait_for_all' and have the first
> node do nothing until it sees the peer.
>

Am I right that wait_for_all is available only in corosync 2.x and not in 1.x?

> To do otherwise would be to risk a split-brain. Each node needs to know the
> state of the peer in order to run services safely. By having both start at
> the same time, then they know what the other is doing. By disabling quorum,
> you allow one node to continue to operate when the other leaves, but it
> needs that initial connection to know for sure what it's doing.
>

Does it apply to both corosync 1.x and 2.x or only to 2.x with
wait_for_all? Because I actually also was confused about precise
meaning of disabling quorum in pacemaker (setting no-quorum-policy:
ignore). So if I have two node cluster with pacemaker 1.x and corosync
1.x with no-quorum-policy=ignore and no fencing - what happens when
one single node starts?

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] When pacemaker expects resource to go directly to Master after start?

[Andrei Borzenkov]

On Thu, Oct 2, 2014 at 2:36 PM, emmanuel segura  wrote:
> I don't know if you can use Dummy primitivi as MS
>
> egrep "promote|demote" /usr/lib/ocf/resource.d/pacemaker/Dummy
> echo $?
> 1
>

Yes, I know I'm not bright, but still not *that* stupid :)

cn1:/usr/lib/ocf/resource.d # grep -E 'promote|demote' test/Dummy


promote)echo MASTER > ${OCF_RESKEY_state};;
demote) echo SLAVE > ${OCF_RESKEY_state};;
cn1:/usr/lib/ocf/resource.d # ocf-tester -n XXX $PWD/test/Dummy
Beginning tests for /usr/lib/ocf/resource.d/test/Dummy...
* Your agent does not support the notify action (optional)
/usr/lib/ocf/resource.d/test/Dummy passed all tests
cn1:/usr/lib/ocf/resource.d #


>
>
>
> 2014-10-02 12:02 GMT+02:00 Andrei Borzenkov :
>> According to documentation (Pacemaker 1.1.x explained) "when
>> [Master/Slave] the resource is started, it must come up in the
>> mode called Slave". But what I observe here - in some cases pacemaker
>> treats Slave state as error. As example (pacemaker 1.1.9):
>>
>> Oct  2 13:23:34 cn1 pengine[9446]:   notice: unpack_rsc_op: Operation
>> monitor found resource test_Dummy:0 active in master mode on cn1
>>
>> So resource currently is Master on node cn1. Second node boots and
>> starts pacemaker which now decides to restart it on the first node (I
>> know why it happens, so it is not relevant to this question :) )
>>
>> Oct  2 13:23:34 cn1 pengine[9446]:   notice: LogActions: Restart
>> test_Dummy:0  (Master cn1)
>> Oct  2 13:23:34 cn1 pengine[9446]:   notice: LogActions: Start
>> test_Dummy:1  (cn2)
>> Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 31: monitor test_Dummy:1_monitor_0 on cn2
>> Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 84: demote test_Dummy_demote_0 on cn1 (local)
>> Oct  2 13:23:34 cn1 crmd[9447]:   notice: process_lrm_event: LRM
>> operation test_Dummy_demote_0 (call=1227, rc=0, cib-update=7826,
>> confirmed=true) ok
>> Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 85: stop test_Dummy_stop_0 on cn1 (local)
>> Oct  2 13:23:34 cn1 crmd[9447]:   notice: process_lrm_event: LRM
>> operation test_Dummy_stop_0 (call=1234, rc=0, cib-update=7827,
>> confirmed=true) ok
>>
>> As expected it calls demote first and stop next. At this point
>> resource is stopped.
>>
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 83: start test_Dummy_start_0 on cn1 (local)
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 87: start test_Dummy:1_start_0 on cn2
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: process_lrm_event: LRM
>> operation test_Dummy_start_0 (call=1244, rc=0, cib-update=7830,
>> confirmed=true) ok
>>
>> Resource is started again. In full conformance with requirement above,
>> it is now slave.
>>
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 88: monitor test_Dummy:1_monitor_11000 on cn2
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
>> action 3: monitor test_Dummy_monitor_1 on cn1 (local)
>> Oct  2 13:23:35 cn1 crmd[9447]:   notice: process_lrm_event: LRM
>> operation test_Dummy_monitor_1 (call=1247, rc=0, cib-update=7831,
>> confirmed=false) ok
>> Oct  2 13:23:35 cn1 crmd[9447]:  warning: status_from_rc: Action 3
>> (test_Dummy_monitor_1) on cn1 failed (target: 8 vs. rc: 0): Error
>>
>> Oops! Why pacemaker expects resource to be Master on cn1? It had been
>> stopped, it was started, it was not promoted yet. Only after recovery
>> from above "error" does it get promoted:
>>
>> Oct  2 13:23:41 cn1 pengine[9446]:   notice: LogActions: Promote
>> test_Dummy:0  (Slave -> Master cn1)
>>
>> primitive pcm_Dummy ocf:pacemaker:Dummy
>> primitive test_Dummy ocf:test:Dummy \
>> op monitor interval="10" role="Master" \
>> op monitor interval="11" \
>> op start interval="0" timeout="30" \
>> op stop interval="0" timeout="120" \
>> op promote interval="0" timeout="20" \
>> op demote interval="0" timeout="20"
>> ms ms_Dummy test_Dummy \
>> meta target-role="Master"
>> clone cln_Dummy pcm_Dummy
>> order ms_Dummy-after-cln_Dummy 2000: cln_Dummy ms_Dummy
>>
>> ___
>

Re: [Pacemaker] Master is restarted when other node comes online

[Andrei Borzenkov]

On Sun, Sep 28, 2014 at 3:03 PM, emmanuel segura  wrote:
> try to use interleave meta attribute in your clone definition
>

Yes, you were right. I missed the fact that it was split between two
clones. Sorry, just starting with pacemaker , still learning ...

Thank you!

> http://www.hastexo.com/resources/hints-and-kinks/interleaving-pacemaker-clones
>
> 2014-09-28 9:56 GMT+02:00 Andrei Borzenkov :
>> I have two node cluster with single master/slave resource (replicated
>> database) using pacemaker+openais on SLES11 SP3 (pacemaker
>> 1.1.11-3ca8c3b). I hit weird situation that I did not see before, and
>> I cannot really understand it. Assuming master runs on node A and
>> slave runs on node B. If I stop cluster stack on B (rcopenais stop)
>> and start it again (rcopenais start) master is restarted. Of course
>> this means service interruption. Same happens if I reboot node B. I
>> have crm_report and can provide logs which are required but I wanted
>> first to quickly make sure this is not expected behavior.
>>
>> I have not seen it before, but now when I recall what I tested, it was
>> always simulation of node failure. I did not really tried above
>> scenario.
>>
>> Assuming this is correct behavior - what is the correct procedure to
>> shutdown single node then? It makes it impossible to do any
>> maintenance on slave node.
>>
>> Configuration below:
>>
>> node msksaphana1 \
>> attributes hana_hdb_vhost="msksaphana1" hana_hdb_site="SITE1"
>> hana_hdb_remoteHost="msksaphana2" hana_hdb_srmode="sync"
>> lpa_hdb_lpt="1411732740"
>> node msksaphana2 \
>> attributes hana_hdb_vhost="msksaphana2" hana_hdb_site="SITE2"
>> hana_hdb_srmode="sync" hana_hdb_remoteHost="msksaphana1"
>> lpa_hdb_lpt="30"
>> primitive rsc_SAPHanaTopology_HDB_HDB00 ocf:suse:SAPHanaTopology \
>> params SID="HDB" InstanceNumber="00" \
>> op monitor interval="10" timeout="600" \
>> op start interval="0" timeout="600" \
>> op stop interval="0" timeout="300"
>> primitive rsc_SAPHana_HDB_HDB00 ocf:suse:SAPHana \
>> params SID="HDB" InstanceNumber="00" PREFER_SITE_TAKEOVER="true"
>> AUTOMATED_REGISTER="true" DUPLICATE_PRIMARY_TIMEOUT="7200" \
>> op start timeout="3600" interval="0" \
>> op stop timeout="3600" interval="0" \
>> op promote timeout="3600" interval="0" \
>> op monitor timeout="700" role="Master" interval="60" \
>> op monitor timeout="700" role="Slave" interval="61"
>> primitive rsc_ip_HDB_HDB00 ocf:heartbeat:IPaddr2 \
>> params ip="10.72.10.64" \
>> op start timeout="20" interval="0" \
>> op stop timeout="20" interval="0" \
>> op monitor interval="10" timeout="20"
>> primitive stonith_IPMI_msksaphana1 stonith:external/ipmi \
>> params ipmitool="/usr/bin/ipmitool" hostname="msksaphana1"
>> passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.47" \
>> op stop timeout="15" interval="0" \
>> op monitor timeout="20" interval="3600" \
>> op start timeout="20" interval="0" \
>> meta target-role="Started"
>> primitive stonith_IPMI_msksaphana2 stonith:external/ipmi \
>> params ipmitool="/usr/bin/ipmitool" hostname="msksaphana2"
>> passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.48" \
>> op stop timeout="15" interval="0" \
>> op monitor timeout="20" interval="3600" \
>> op start timeout="20" interval="0" \
>> meta target-role="Started"
>> ms msl_SAPHana_HDB_HDB00 rsc_SAPHana_HDB_HDB00 \
>> meta clone-max="2" clone-node-max="1" target-role="Started"
>> clone cln_SAPHanaTopology_HDB_HDB00 rsc_SAPHanaTopology_HDB_HDB00 \
>> meta is-managed="true" clone-node-max="1" target-role="Started"
>> location stonoth_IPMI_msksaphana1_on_msksaphana2
>> stonith_IPMI_msksaphana1 -inf: msksaphana1
>> location stonoth_IPMI_msksaphana2_on_msksaphana1
>> stonith_IPMI_msksaphana2 -inf: msksaphana2
>> colo

[Pacemaker] When pacemaker expects resource to go directly to Master after start?

[Andrei Borzenkov]

According to documentation (Pacemaker 1.1.x explained) "when
[Master/Slave] the resource is started, it must come up in the
mode called Slave". But what I observe here - in some cases pacemaker
treats Slave state as error. As example (pacemaker 1.1.9):

Oct  2 13:23:34 cn1 pengine[9446]:   notice: unpack_rsc_op: Operation
monitor found resource test_Dummy:0 active in master mode on cn1

So resource currently is Master on node cn1. Second node boots and
starts pacemaker which now decides to restart it on the first node (I
know why it happens, so it is not relevant to this question :) )

Oct  2 13:23:34 cn1 pengine[9446]:   notice: LogActions: Restart
test_Dummy:0  (Master cn1)
Oct  2 13:23:34 cn1 pengine[9446]:   notice: LogActions: Start
test_Dummy:1  (cn2)
Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 31: monitor test_Dummy:1_monitor_0 on cn2
Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 84: demote test_Dummy_demote_0 on cn1 (local)
Oct  2 13:23:34 cn1 crmd[9447]:   notice: process_lrm_event: LRM
operation test_Dummy_demote_0 (call=1227, rc=0, cib-update=7826,
confirmed=true) ok
Oct  2 13:23:34 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 85: stop test_Dummy_stop_0 on cn1 (local)
Oct  2 13:23:34 cn1 crmd[9447]:   notice: process_lrm_event: LRM
operation test_Dummy_stop_0 (call=1234, rc=0, cib-update=7827,
confirmed=true) ok

As expected it calls demote first and stop next. At this point
resource is stopped.

Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 83: start test_Dummy_start_0 on cn1 (local)
Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 87: start test_Dummy:1_start_0 on cn2
Oct  2 13:23:35 cn1 crmd[9447]:   notice: process_lrm_event: LRM
operation test_Dummy_start_0 (call=1244, rc=0, cib-update=7830,
confirmed=true) ok

Resource is started again. In full conformance with requirement above,
it is now slave.

Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 88: monitor test_Dummy:1_monitor_11000 on cn2
Oct  2 13:23:35 cn1 crmd[9447]:   notice: te_rsc_command: Initiating
action 3: monitor test_Dummy_monitor_1 on cn1 (local)
Oct  2 13:23:35 cn1 crmd[9447]:   notice: process_lrm_event: LRM
operation test_Dummy_monitor_1 (call=1247, rc=0, cib-update=7831,
confirmed=false) ok
Oct  2 13:23:35 cn1 crmd[9447]:  warning: status_from_rc: Action 3
(test_Dummy_monitor_1) on cn1 failed (target: 8 vs. rc: 0): Error

Oops! Why pacemaker expects resource to be Master on cn1? It had been
stopped, it was started, it was not promoted yet. Only after recovery
from above "error" does it get promoted:

Oct  2 13:23:41 cn1 pengine[9446]:   notice: LogActions: Promote
test_Dummy:0  (Slave -> Master cn1)

primitive pcm_Dummy ocf:pacemaker:Dummy
primitive test_Dummy ocf:test:Dummy \
op monitor interval="10" role="Master" \
op monitor interval="11" \
op start interval="0" timeout="30" \
op stop interval="0" timeout="120" \
op promote interval="0" timeout="20" \
op demote interval="0" timeout="20"
ms ms_Dummy test_Dummy \
meta target-role="Master"
clone cln_Dummy pcm_Dummy
order ms_Dummy-after-cln_Dummy 2000: cln_Dummy ms_Dummy

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Show all resource properties with crmsh

[Andrei Borzenkov]

Is it possible to display values for all resource properties,
including those set to default values? cibadmin or "crm configure
show" display only explicitly set properties, and crm_resource or "crm
resource meta" work with single property only. Ideally I'd like to get
actual values of all resource properties in configuration.

This is pacemaker 1.19 with crmsh 1.2.5 on SLES.

TIA

-andrei

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Master is restarted when other node comes online

[Andrei Borzenkov]

В Sun, 28 Sep 2014 13:03:08 +0200
emmanuel segura  пишет:

> try to use interleave meta attribute in your clone definition
> 
> http://www.hastexo.com/resources/hints-and-kinks/interleaving-pacemaker-clones
> 

I appreciate if you could elaborate a bit more here. The above link
explains dependencies between two cloned resources. I have single
cloned resource, without any external dependency. I'm afraid I miss how
interleave helps here.

Thank you!

> 2014-09-28 9:56 GMT+02:00 Andrei Borzenkov :
> > I have two node cluster with single master/slave resource (replicated
> > database) using pacemaker+openais on SLES11 SP3 (pacemaker
> > 1.1.11-3ca8c3b). I hit weird situation that I did not see before, and
> > I cannot really understand it. Assuming master runs on node A and
> > slave runs on node B. If I stop cluster stack on B (rcopenais stop)
> > and start it again (rcopenais start) master is restarted. Of course
> > this means service interruption. Same happens if I reboot node B. I
> > have crm_report and can provide logs which are required but I wanted
> > first to quickly make sure this is not expected behavior.
> >
> > I have not seen it before, but now when I recall what I tested, it was
> > always simulation of node failure. I did not really tried above
> > scenario.
> >
> > Assuming this is correct behavior - what is the correct procedure to
> > shutdown single node then? It makes it impossible to do any
> > maintenance on slave node.
> >
> > Configuration below:
> >
> > node msksaphana1 \
> > attributes hana_hdb_vhost="msksaphana1" hana_hdb_site="SITE1"
> > hana_hdb_remoteHost="msksaphana2" hana_hdb_srmode="sync"
> > lpa_hdb_lpt="1411732740"
> > node msksaphana2 \
> > attributes hana_hdb_vhost="msksaphana2" hana_hdb_site="SITE2"
> > hana_hdb_srmode="sync" hana_hdb_remoteHost="msksaphana1"
> > lpa_hdb_lpt="30"
> > primitive rsc_SAPHanaTopology_HDB_HDB00 ocf:suse:SAPHanaTopology \
> > params SID="HDB" InstanceNumber="00" \
> > op monitor interval="10" timeout="600" \
> > op start interval="0" timeout="600" \
> > op stop interval="0" timeout="300"
> > primitive rsc_SAPHana_HDB_HDB00 ocf:suse:SAPHana \
> > params SID="HDB" InstanceNumber="00" PREFER_SITE_TAKEOVER="true"
> > AUTOMATED_REGISTER="true" DUPLICATE_PRIMARY_TIMEOUT="7200" \
> > op start timeout="3600" interval="0" \
> > op stop timeout="3600" interval="0" \
> > op promote timeout="3600" interval="0" \
> > op monitor timeout="700" role="Master" interval="60" \
> > op monitor timeout="700" role="Slave" interval="61"
> > primitive rsc_ip_HDB_HDB00 ocf:heartbeat:IPaddr2 \
> > params ip="10.72.10.64" \
> > op start timeout="20" interval="0" \
> > op stop timeout="20" interval="0" \
> > op monitor interval="10" timeout="20"
> > primitive stonith_IPMI_msksaphana1 stonith:external/ipmi \
> > params ipmitool="/usr/bin/ipmitool" hostname="msksaphana1"
> > passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.47" \
> > op stop timeout="15" interval="0" \
> > op monitor timeout="20" interval="3600" \
> > op start timeout="20" interval="0" \
> > meta target-role="Started"
> > primitive stonith_IPMI_msksaphana2 stonith:external/ipmi \
> > params ipmitool="/usr/bin/ipmitool" hostname="msksaphana2"
> > passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.48" \
> > op stop timeout="15" interval="0" \
> > op monitor timeout="20" interval="3600" \
> > op start timeout="20" interval="0" \
> > meta target-role="Started"
> > ms msl_SAPHana_HDB_HDB00 rsc_SAPHana_HDB_HDB00 \
> > meta clone-max="2" clone-node-max="1" target-role="Started"
> > clone cln_SAPHanaTopology_HDB_HDB00 rsc_SAPHanaTopology_HDB_HDB00 \
> > meta is-managed="true" clone-node-max="1" target-role="Started"
> > location stonoth_IPMI_msksaphana1_on_msksaphana2
>

[Pacemaker] Master is restarted when other node comes online

[Andrei Borzenkov]

I have two node cluster with single master/slave resource (replicated
database) using pacemaker+openais on SLES11 SP3 (pacemaker
1.1.11-3ca8c3b). I hit weird situation that I did not see before, and
I cannot really understand it. Assuming master runs on node A and
slave runs on node B. If I stop cluster stack on B (rcopenais stop)
and start it again (rcopenais start) master is restarted. Of course
this means service interruption. Same happens if I reboot node B. I
have crm_report and can provide logs which are required but I wanted
first to quickly make sure this is not expected behavior.

I have not seen it before, but now when I recall what I tested, it was
always simulation of node failure. I did not really tried above
scenario.

Assuming this is correct behavior - what is the correct procedure to
shutdown single node then? It makes it impossible to do any
maintenance on slave node.

Configuration below:

node msksaphana1 \
attributes hana_hdb_vhost="msksaphana1" hana_hdb_site="SITE1"
hana_hdb_remoteHost="msksaphana2" hana_hdb_srmode="sync"
lpa_hdb_lpt="1411732740"
node msksaphana2 \
attributes hana_hdb_vhost="msksaphana2" hana_hdb_site="SITE2"
hana_hdb_srmode="sync" hana_hdb_remoteHost="msksaphana1"
lpa_hdb_lpt="30"
primitive rsc_SAPHanaTopology_HDB_HDB00 ocf:suse:SAPHanaTopology \
params SID="HDB" InstanceNumber="00" \
op monitor interval="10" timeout="600" \
op start interval="0" timeout="600" \
op stop interval="0" timeout="300"
primitive rsc_SAPHana_HDB_HDB00 ocf:suse:SAPHana \
params SID="HDB" InstanceNumber="00" PREFER_SITE_TAKEOVER="true"
AUTOMATED_REGISTER="true" DUPLICATE_PRIMARY_TIMEOUT="7200" \
op start timeout="3600" interval="0" \
op stop timeout="3600" interval="0" \
op promote timeout="3600" interval="0" \
op monitor timeout="700" role="Master" interval="60" \
op monitor timeout="700" role="Slave" interval="61"
primitive rsc_ip_HDB_HDB00 ocf:heartbeat:IPaddr2 \
params ip="10.72.10.64" \
op start timeout="20" interval="0" \
op stop timeout="20" interval="0" \
op monitor interval="10" timeout="20"
primitive stonith_IPMI_msksaphana1 stonith:external/ipmi \
params ipmitool="/usr/bin/ipmitool" hostname="msksaphana1"
passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.47" \
op stop timeout="15" interval="0" \
op monitor timeout="20" interval="3600" \
op start timeout="20" interval="0" \
meta target-role="Started"
primitive stonith_IPMI_msksaphana2 stonith:external/ipmi \
params ipmitool="/usr/bin/ipmitool" hostname="msksaphana2"
passwd="P@ssw0rd" userid="hacluster" ipaddr="10.72.5.48" \
op stop timeout="15" interval="0" \
op monitor timeout="20" interval="3600" \
op start timeout="20" interval="0" \
meta target-role="Started"
ms msl_SAPHana_HDB_HDB00 rsc_SAPHana_HDB_HDB00 \
meta clone-max="2" clone-node-max="1" target-role="Started"
clone cln_SAPHanaTopology_HDB_HDB00 rsc_SAPHanaTopology_HDB_HDB00 \
meta is-managed="true" clone-node-max="1" target-role="Started"
location stonoth_IPMI_msksaphana1_on_msksaphana2
stonith_IPMI_msksaphana1 -inf: msksaphana1
location stonoth_IPMI_msksaphana2_on_msksaphana1
stonith_IPMI_msksaphana2 -inf: msksaphana2
colocation col_saphana_ip_HDB_HDB00 2000: rsc_ip_HDB_HDB00:Started
msl_SAPHana_HDB_HDB00:Master
order ord_SAPHana_HDB_HDB00 2000: cln_SAPHanaTopology_HDB_HDB00
msl_SAPHana_HDB_HDB00
property $id="cib-bootstrap-options" \
stonith-enabled="true" \
placement-strategy="balanced" \
dc-version="1.1.11-3ca8c3b" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-action="reboot" \
no-quorum-policy="ignore" \
last-lrm-refresh="1411730405"
rsc_defaults $id="rsc-options" \
resource-stickiness="1" \
migration-threshold="3"
op_defaults $id="op-options" \
timeout="600" \
record-pending="true"

Thank you!

-andrei

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Favor one node during stonith?

[Andrei Borzenkov]

В Thu, 14 Aug 2014 12:45:27 +1000
Andrew Beekhof  пишет:

> > 
> > It statically assigns priorities to cluster nodes. I need to
> > dynamically assign higher priority (lower delay) to a node that is
> > currently running application to ensure that application survives. It
> > was relatively easy in other cluster products I worked with, but I
> > cannot figure out how to do it in pacemaker so far.
> 
> Its not something we have traditionally supported as it only really makes 
> sense in a strict active/passive environment.

Not really - second node may run non-production applications that we do
not really care about in case of split brain.

But thank you for confirming it is not me :)

> As soon as you have more than one node hosting resources, or hosting more 
> than one resource, determining the priority quickly gets complicated.
> 
> Better to avoid split-brain in the first place.
> Add a third node?  (Even if you don't allow it to run services).
> That way when a node disappears, its almost certainly because its dead/failed.
> 

I have to work with what I get ...


signature.asc
Description: PGP signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org