Re: [ClusterLabs] Change disk

[Ken Gaillot]

On 09/14/2016 09:30 AM, NetLink wrote:
> My two node email server cluster uses corosync 1.4.2.
> 
> The device /dev/drbd3, which only holds the email data in a separated
> disk, is running out of space.
> 
> To change the two disks for bigger ones I’m thinking to use the
> following strategy:
> 
> 1.Put node 2 in standby
> 
> 2.Change and configure the new bigger disk on node 2
> 
> 3.Put node 2 back online and wait for syncing.
> 
> 4.Put node 1 in standby and repeat the procedure
> 
> Would this approach work?
> 
> I’m concerned if the meta data of the new disk created on node 2 with
> drdbadm create-md r3 would conflict with node 1.

I think you need to invalidate the data on the new disk and do a full
sync, just as if you were adding the node for the first time. Exact
syntax depends on drbd version. It's probably best to stop the cluster
on node 2 altogether, after putting it in standby, and bring up drbd
manually for the initial sync.

>From your list there, I'm thinking you might have hot swappable disks
... I always loved that feature :)

> Or even better, is there a 'how to' specific for this case?
> 
>  
> 
> Best regards
> 
> Wolfgang
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/22/2016 05:58 PM, Andrew Beekhof wrote:
> 
> 
> On Fri, Sep 23, 2016 at 1:58 AM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 09/22/2016 09:53 AM, Jan Pokorný wrote:
> > On 22/09/16 08:42 +0200, Kristoffer Grönlund wrote:
> >> Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>> writes:
> >>
> >>> I'm not saying it's a bad idea, just that it's more complicated than 
> it
> >>> first sounds, so it's worth thinking through the implications.
> >>
> >> Thinking about it and looking at how complicated it gets, maybe what
> >> you'd really want, to make it clearer for the user, is the ability to
> >> explicitly configure the behavior, either globally or per-resource. So
> >> instead of having to tweak a set of variables that interact in complex
> >> ways, you'd configure something like rule expressions,
> >>
> >> 
> >>   
> >>   
> >>   
> >> 
> >>
> >> So, try to restart the service 3 times, if that fails migrate the
> >> service, if it still fails, fence the node.
> >>
> >> (obviously the details and XML syntax are just an example)
> >>
> >> This would then replace on-fail, migration-threshold, etc.
> >
> > I must admit that in previous emails in this thread, I wasn't able to
> > follow during the first pass, which is not the case with this procedural
> > (sequence-ordered) approach.  Though someone can argue it doesn't take
> > type of operation into account, which might again open the door for
> > non-obvious interactions.
> 
> "restart" is the only on-fail value that it makes sense to escalate.
> 
> block/stop/fence/standby are final. Block means "don't touch the
> resource again", so there can't be any further response to failures.
> Stop/fence/standby move the resource off the local node, so failure
> handling is reset (there are 0 failures on the new node to begin with).
> 
> "Ignore" is theoretically possible to escalate, e.g. "ignore 3 failures
> then migrate", but I can't think of a real-world situation where that
> makes sense, 
> 
> 
> really?
> 
> it is not uncommon to hear "i know its failed, but i dont want the
> cluster to do anything until its _really_ failed"  

Hmm, I guess that would be similar to how monitoring systems such as
nagios can be configured to send an alert only if N checks in a row
fail. That's useful where transient outages (e.g. a webserver hitting
its request limit) are acceptable for a short time.

I'm not sure that's translatable to Pacemaker. Pacemaker's error count
is not "in a row" but "since the count was last cleared".

"Ignore up to three monitor failures if they occur in a row [or, within
10 minutes?], then try soft recovery for the next two monitor failures,
then ban this node for the next monitor failure." Not sure being able to
say that is worth the complexity.

> 
> and it would be a significant re-implementation of "ignore"
> (which currently ignores the state of having failed, as opposed to a
> particular instance of failure).
> 
> 
> agreed
>  
> 
> 
> What the interface needs to express is: "If this operation fails,
> optionally try a soft recovery [always stop+start], but if  failures
> occur on the same node, proceed to a [configurable] hard recovery".
> 
> And of course the interface will need to be different depending on how
> certain details are decided, e.g. whether any failures count toward 
> or just failures of one particular operation type, and whether the hard
> recovery type can vary depending on what operation failed.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 10/02/2016 10:02 PM, Andrew Beekhof wrote:
>> Take a
>> look at all of nagios' options for deciding when a failure becomes "real".
> 
> I used to take a very hard line on this: if you don't want the cluster
> to do anything about an error, don't tell us about it.
> However I'm slowly changing my position... the reality is that many
> people do want a heads up in advance and we have been forcing that
> policy (when does an error become real) into the agents where one size
> must fit all.
> 
> So I'm now generally in favour of having the PE handle this "somehow".

Nagios is a useful comparison:

check_interval - like pacemaker's monitor interval

retry_interval - if a check returns failure, switch to this interval
(i.e. check more frequently when trying to decide whether it's a "hard"
failure)

max_check_attempts - if a check fails this many times in a row, it's a
hard failure. Before this is reached, it's considered a soft failure.
Nagios will call event handlers (comparable to pacemaker's alert agents)
for both soft and hard failures (distinguishing the two). A service is
also considered to have a "hard failure" if its host goes down.

high_flap_threshold/low_flap_threshold - a service is considered to be
flapping when its percent of state changes (ok <-> not ok) in the last
21 checks (= max. 20 state changes) reaches high_flap_threshold, and
stable again once the percentage drops to low_flap_threshold. To put it
another way, a service that passes every monitor is 0% flapping, and a
service that fails every other monitor is 100% flapping. With these,
even if a service never reaches max_check_attempts failures in a row, an
alert can be sent if it's repeatedly failing and recovering.

>> If you clear failures after a success, you can't detect/recover a
>> resource that is flapping.
> 
> Ah, but you can if the thing you're clearing only applies to other
> failures of the same action.
> A completed start doesn't clear a previously failed monitor.

Nope -- a monitor can alternately succeed and fail repeatedly, and that
indicates a problem, but wouldn't trip an "N-failures-in-a-row" system.

>> It only makes sense to escalate from ignore -> restart -> hard, so maybe
>> something like:
>>
>>   op monitor ignore-fail=3 soft-fail=2 on-hard-fail=ban
>>
> I would favour something more concrete than 'soft' and 'hard' here.
> Do they have a sufficiently obvious meaning outside of us developers?
> 
> Perhaps (with or without a "failures-" prefix) :
> 
>ignore-count
>recover-count
>escalation-policy

I think the "soft" vs "hard" terminology is somewhat familiar to
sysadmins -- there's at least nagios, email (SPF failures and bounces),
and ECC RAM. But throwing "ignore" into the mix does confuse things.

How about ... max-fail-ignore=3 max-fail-restart=2 fail-escalation=ban


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] stonithd/fenced filling up logs

[Ken Gaillot]

On 10/04/2016 11:31 AM, Israel Brewster wrote:
> I sent this a week ago, but never got a response, so I'm sending it
> again in the hopes that it just slipped through the cracks. It seems to
> me that this should just be a simple mis-configuration on my part
> causing the issue, but I suppose it could be a bug as well.
> 
> I have two two-node clusters set up using corosync/pacemaker on CentOS
> 6.8. One cluster is simply sharing an IP, while the other one has
> numerous services and IP's set up between the two machines in the
> cluster. Both appear to be working fine. However, I was poking around
> today, and I noticed that on the single IP cluster, corosync, stonithd,
> and fenced were using "significant" amounts of processing power - 25%
> for corosync on the current primary node, with fenced and stonithd often
> showing 1-2% (not horrible, but more than any other process). In looking
> at my logs, I see that they are dumping messages like the following to
> the messages log every second or two:
> 
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:  warning: get_xpath_object:
> No match for //@st_delegate in /st-reply
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:   notice: remote_op_done:
> Operation reboot of fai-dbs1 by fai-dbs2 for
> stonith_admin.cman.15835@fai-dbs2.c5161517: No such device
> Sep 27 08:51:50 fai-dbs1 crmd[4855]:   notice: tengine_stonith_notify:
> Peer fai-dbs1 was not terminated (reboot) by fai-dbs2 for fai-dbs2: No
> such device (ref=c5161517-c0cc-42e5-ac11-1d55f7749b05) by client
> stonith_admin.cman.15835
> Sep 27 08:51:50 fai-dbs1 fence_pcmk[15393]: Requesting Pacemaker fence
> fai-dbs2 (reset)

The above shows that CMAN is asking pacemaker to fence a node. Even
though fencing is disabled in pacemaker itself, CMAN is configured to
use pacemaker for fencing (fence_pcmk).

> Sep 27 08:51:50 fai-dbs1 stonith_admin[15394]:   notice: crm_log_args:
> Invoked: stonith_admin --reboot fai-dbs2 --tolerance 5s --tag cman 
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:   notice: handle_request:
> Client stonith_admin.cman.15394.2a97d89d wants to fence (reboot)
> 'fai-dbs2' with device '(any)'
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:   notice:
> initiate_remote_stonith_op: Initiating remote operation reboot for
> fai-dbs2: bc3f5d73-57bd-4aff-a94c-f9978aa5c3ae (0)
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:   notice:
> stonith_choose_peer: Couldn't find anyone to fence fai-dbs2 with 
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:  warning: get_xpath_object:
> No match for //@st_delegate in /st-reply
> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:error: remote_op_done:
> Operation reboot of fai-dbs2 by fai-dbs1 for
> stonith_admin.cman.15394@fai-dbs1.bc3f5d73: No such device
> Sep 27 08:51:50 fai-dbs1 crmd[4855]:   notice: tengine_stonith_notify:
> Peer fai-dbs2 was not terminated (reboot) by fai-dbs1 for fai-dbs1: No
> such device (ref=bc3f5d73-57bd-4aff-a94c-f9978aa5c3ae) by client
> stonith_admin.cman.15394
> Sep 27 08:51:50 fai-dbs1 fence_pcmk[15393]: Call to fence fai-dbs2
> (reset) failed with rc=237
> 
> After seeing this one the one cluster, I checked the logs on the other
> and sure enough I'm seeing the same thing there. As I mentioned, both
> nodes in both clusters *appear* to be operating correctly. For example,
> the output of "pcs status" on the small cluster is this:
> 
> [root@fai-dbs1 ~]# pcs status
> Cluster name: dbs_cluster
> Last updated: Tue Sep 27 08:59:44 2016
> Last change: Thu Mar  3 06:11:00 2016
> Stack: cman
> Current DC: fai-dbs1 - partition with quorum
> Version: 1.1.11-97629de
> 2 Nodes configured
> 1 Resources configured
> 
> 
> Online: [ fai-dbs1 fai-dbs2 ]
> 
> Full list of resources:
> 
>  virtual_ip(ocf::heartbeat:IPaddr2):Started fai-dbs1
> 
> And on the larger cluster, it has services running across both nodes of
> the cluster, and I've been able to move stuff back and forth without
> issue. Both nodes have the stonith-enabled property set to false, and
> no-quorum-policy set to ignore (since they are only two nodes in the
> cluster).
> 
> What could be causing the log messages? Is the CPU usage normal, or
> might there be something I can do about that as well? Thanks.

It's not normal; most likely, the failed fencing is being retried endlessly.

You'll want to figure out why CMAN is asking for fencing. You may have
some sort of communication problem between the nodes (that might be a
factor in corosync's CPU usage, too).

Once that's straightened out, it's a good idea to actually configure and
enable fencing :)


> 
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: 

Re: [ClusterLabs] stonithd/fenced filling up logs

[Ken Gaillot]

On 10/05/2016 11:56 AM, Israel Brewster wrote:
>> On Oct 4, 2016, at 4:06 PM, Digimer <li...@alteeve.ca
>> <mailto:li...@alteeve.ca>> wrote:
>>
>> On 04/10/16 07:50 PM, Israel Brewster wrote:
>>> On Oct 4, 2016, at 3:38 PM, Digimer <li...@alteeve.ca
>>> <mailto:li...@alteeve.ca>> wrote:
>>>>
>>>> On 04/10/16 07:09 PM, Israel Brewster wrote:
>>>>> On Oct 4, 2016, at 3:03 PM, Digimer <li...@alteeve.ca
>>>>> <mailto:li...@alteeve.ca>> wrote:
>>>>>>
>>>>>> On 04/10/16 06:50 PM, Israel Brewster wrote:
>>>>>>> On Oct 4, 2016, at 2:26 PM, Ken Gaillot <kgail...@redhat.com
>>>>>>> <mailto:kgail...@redhat.com>
>>>>>>> <mailto:kgail...@redhat.com>> wrote:
>>>>>>>>
>>>>>>>> On 10/04/2016 11:31 AM, Israel Brewster wrote:
>>>>>>>>> I sent this a week ago, but never got a response, so I'm sending it
>>>>>>>>> again in the hopes that it just slipped through the cracks. It
>>>>>>>>> seems to
>>>>>>>>> me that this should just be a simple mis-configuration on my part
>>>>>>>>> causing the issue, but I suppose it could be a bug as well.
>>>>>>>>>
>>>>>>>>> I have two two-node clusters set up using corosync/pacemaker on
>>>>>>>>> CentOS
>>>>>>>>> 6.8. One cluster is simply sharing an IP, while the other one has
>>>>>>>>> numerous services and IP's set up between the two machines in the
>>>>>>>>> cluster. Both appear to be working fine. However, I was poking
>>>>>>>>> around
>>>>>>>>> today, and I noticed that on the single IP cluster, corosync,
>>>>>>>>> stonithd,
>>>>>>>>> and fenced were using "significant" amounts of processing power
>>>>>>>>> - 25%
>>>>>>>>> for corosync on the current primary node, with fenced and
>>>>>>>>> stonithd often
>>>>>>>>> showing 1-2% (not horrible, but more than any other process).
>>>>>>>>> In looking
>>>>>>>>> at my logs, I see that they are dumping messages like the
>>>>>>>>> following to
>>>>>>>>> the messages log every second or two:
>>>>>>>>>
>>>>>>>>> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:  warning:
>>>>>>>>> get_xpath_object:
>>>>>>>>> No match for //@st_delegate in /st-reply
>>>>>>>>> Sep 27 08:51:50 fai-dbs1 stonith-ng[4851]:   notice:
>>>>>>>>> remote_op_done:
>>>>>>>>> Operation reboot of fai-dbs1 by fai-dbs2 for
>>>>>>>>> stonith_admin.cman.15835@fai-dbs2.c5161517: No such device
>>>>>>>>> Sep 27 08:51:50 fai-dbs1 crmd[4855]:   notice:
>>>>>>>>> tengine_stonith_notify:
>>>>>>>>> Peer fai-dbs1 was not terminated (reboot) by fai-dbs2 for
>>>>>>>>> fai-dbs2: No
>>>>>>>>> such device (ref=c5161517-c0cc-42e5-ac11-1d55f7749b05) by client
>>>>>>>>> stonith_admin.cman.15835
>>>>>>>>> Sep 27 08:51:50 fai-dbs1 fence_pcmk[15393]: Requesting
>>>>>>>>> Pacemaker fence
>>>>>>>>> fai-dbs2 (reset)
>>>>>>>>
>>>>>>>> The above shows that CMAN is asking pacemaker to fence a node. Even
>>>>>>>> though fencing is disabled in pacemaker itself, CMAN is
>>>>>>>> configured to
>>>>>>>> use pacemaker for fencing (fence_pcmk).
>>>>>>>
>>>>>>> I never did any specific configuring of CMAN, Perhaps that's the
>>>>>>> problem? I missed some configuration steps on setup? I just
>>>>>>> followed the
>>>>>>> directions
>>>>>>> here:
>>>>>>> http://jensd.be/156/linux/building-a-high-available-failover-cluster-with-pacemaker-corosync-pcs,
>>>>>>> which disabled stonith in pacemaker via the
>>>>>>> "pcs property set stonith-enabled=fal

Re: [ClusterLabs] No DRBD resource promoted to master in Active/Passive setup

[Ken Gaillot]

t; Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: State transition S_NOT_DC -> 
> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK 
> origin=peer_update_callback ]
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: State transition S_ELECTION 
> -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED 
> origin=election_timeout_popped ]
> Sep 20 12:08:04 MDA1PFP-S02 attrd[2351]:  notice: crm_update_peer_proc: Node 
> MDA1PFP-PCS01[1] - state is now lost (was member)
> Sep 20 12:08:04 MDA1PFP-S02 attrd[2351]:  notice: Removing all MDA1PFP-PCS01 
> attributes for attrd_peer_change_cb
> Sep 20 12:08:04 MDA1PFP-S02 attrd[2351]:  notice: Lost attribute writer 
> MDA1PFP-PCS01
> Sep 20 12:08:04 MDA1PFP-S02 attrd[2351]:  notice: Removing MDA1PFP-PCS01/1 
> from the membership list
> Sep 20 12:08:04 MDA1PFP-S02 attrd[2351]:  notice: Purged 1 peers with id=1 
> and/or uname=MDA1PFP-PCS01 from the membership cache
> Sep 20 12:08:04 MDA1PFP-S02 stonith-ng[2349]:  notice: crm_update_peer_proc: 
> Node MDA1PFP-PCS01[1] - state is now lost (was member)
> Sep 20 12:08:04 MDA1PFP-S02 stonith-ng[2349]:  notice: Removing 
> MDA1PFP-PCS01/1 from the membership list
> Sep 20 12:08:04 MDA1PFP-S02 stonith-ng[2349]:  notice: Purged 1 peers with 
> id=1 and/or uname=MDA1PFP-PCS01 from the membership cache
> Sep 20 12:08:04 MDA1PFP-S02 cib[2348]:  notice: crm_update_peer_proc: Node 
> MDA1PFP-PCS01[1] - state is now lost (was member)
> Sep 20 12:08:04 MDA1PFP-S02 cib[2348]:  notice: Removing MDA1PFP-PCS01/1 from 
> the membership list
> Sep 20 12:08:04 MDA1PFP-S02 cib[2348]:  notice: Purged 1 peers with id=1 
> and/or uname=MDA1PFP-PCS01 from the membership cache
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]: warning: FSA: Input I_ELECTION_DC 
> from do_election_check() received in state S_INTEGRATION
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: Notifications disabled
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:   error: pcmkRegisterNode: Triggered 
> assert at xml.c:594 : node->type == XML_ELEMENT_NODE
> Sep 20 12:08:04 MDA1PFP-S02 pengine[2353]:  notice: On loss of CCM Quorum: 
> Ignore
> Sep 20 12:08:04 MDA1PFP-S02 pengine[2353]:  notice: Demote  drbd1:0   (Master 
> -> Slave MDA1PFP-PCS02)
> Sep 20 12:08:04 MDA1PFP-S02 pengine[2353]:  notice: Calculated Transition 0: 
> /var/lib/pacemaker/pengine/pe-input-1813.bz2
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: Initiating action 55: notify 
> drbd1_pre_notify_demote_0 on MDA1PFP-PCS02 (local)
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: Operation drbd1_notify_0: ok 
> (node=MDA1PFP-PCS02, call=39, rc=0, cib-update=0, confirmed=true)
> Sep 20 12:08:04 MDA1PFP-S02 crmd[2354]:  notice: Initiating action 18: demote 
> drbd1_demote_0 on MDA1PFP-PCS02 (local)

The demote is requested by Pacemaker.

You can get more info from the pe-input-1813.bz2 file referenced above,
e.g. "crm_simulate -Ssx /var/lib/pacemaker/pengine/pe-input-1813.bz2"
should show the scores and planned actions. It's not the easiest to read
but it has some good info.

> 
> Best wishes,
>   Jens
> 
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter 
> de.cgi.com/pflichtangaben.
> 
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI 
> Group Inc. and its affiliates may be contained in this message. If you are 
> not a recipient indicated or intended in this message (or responsible for 
> delivery of this message to such person), or you think for any reason that 
> this message may have been addressed to you in error, you may not use or copy 
> or deliver this message to anyone else. In such case, you should destroy this 
> message and are asked to notify the sender by reply e-mail.
> 
> 
> Von: Ken Gaillot [kgail...@redhat.com]
> Gesendet: Montag, 19. September 2016 17:27
> An: Auer, Jens; Cluster Labs - All topics related to open-source clustering 
> welcomed
> Betreff: Re: AW: [ClusterLabs] No DRBD resource promoted to master in 
> Active/Passive setup
> 
> On 09/19/2016 09:48 AM, Auer, Jens wrote:
>> Hi,
>>
>>> Is the network interface being taken down here used for corosync
>>> communication? If so, that is a node-level failure, and pacemaker will
>>> fence.
>>
>> We have different connections on each server:
>> - A bonded 10GB network card for data traffic that will be accessed via a 
>> virtual ip managed by pacemaker in 192.168.120.1/24. In the cluster nodes 
>> MDA1PFP-S01 and MDA1PFP-S02 are assigned to 192.168.120.10 and 
>> 

Re: [ClusterLabs] best practice fencing with ipmi in 2node-setups / cloneresource/monitor/timeout

[Ken Gaillot]

On 09/21/2016 01:51 AM, Stefan Bauer wrote:
> Hi Ken,
> 
> let met sum it up:
> 
> Pacemaker in recent versions is smart enough to run (trigger, execute) the 
> fence operation on the node, that is not the target.
> 
> If i have an external stonith device that can fence multiple nodes, a single 
> primitive is enough in pacemaker.
> 
> If with external/ipmi i can only address a single node, i need to have 
> multiple primitives - one for each node.
> 
> In this case it's recommended to let the primitive always run on the opposite 
> node - right?

Yes, exactly :-)

In terms of implementation, I'd use a +INFINITY location constraint to
tie the device to the opposite node. This approach (as opposed to a
-INFINITY constraint on the target node) allows the target node to run
the fence device when the opposite node is unavailable.

> thank you.
> 
> Stefan
>  
> -Ursprüngliche Nachricht-
>> Von:Ken Gaillot 
>> Gesendet: Die 20 September 2016 16:49
>> An: users@clusterlabs.org
>> Betreff: Re: [ClusterLabs] best practice fencing with ipmi in 2node-setups / 
>> cloneresource/monitor/timeout
>>
>> On 09/20/2016 06:42 AM, Digimer wrote:
>>> On 20/09/16 06:59 AM, Stefan Bauer wrote:
 Hi,

 i run a 2 node cluster and want to be save in split-brain scenarios. For
 this i setup external/ipmi to stonith the other node.
>>>
>>> Please use 'fence_ipmilan'. I believe that the older external/ipmi are
>>> deprecated (someone correct me if I am wrong on this).
>>
>> It's just an alternative. The "external/" agents come with the
>> cluster-glue package, which isn't provided by some distributions (such
>> as RHEL and its derivatives), so it's "deprecated" on those only.
>>
 Some possible issues jumped to my mind and i would ike to find the best
 practice solution:

 - I have a primitive for each node to stonith. Many documents and guides
 recommend to never let them run on the host it should fence. I would
 setup clone resources to avoid dealing with locations that would also
 influence scoring. Does that make sense?
>>>
>>> Since v1.1.10 of pacemaker, you don't have to worry about this.
>>> Pacemaker is smart enough to know where to run a fence call from in
>>> order to terminate a target.
>>
>> Right, fence devices can run anywhere now, and in fact they don't even
>> have to be "running" for pacemaker to use them -- as long as they are
>> configured and not intentionally disabled, pacemaker will use them.
>>
>> There is still a slight advantage to not running a fence device on a
>> node it can fence. "Running" a fence device in pacemaker really means
>> running the recurring monitor for it. Since the node that runs the
>> monitor has "verified" access to the device, pacemaker will prefer to
>> use it to execute that device. However, pacemaker will not use a node to
>> fence itself, except as a last resort if no other node is available. So,
>> running a fence device on a node it can fence means that the preference
>> is lost.
>>
>> That's a very minor detail, not worth worrying about. It's more a matter
>> of personal preference.
>>
>> In this particular case, a more relevant concern is that you need
>> different configurations for the different targets (the IPMI address is
>> different).
>>
>> One approach is to define two different fence devices, each with one
>> IPMI address. In that case, it makes sense to use the location
>> constraints to ensure the device prefers the node that's not its target.
>>
>> Another approach (if the fence agent supports it) is to use
>> pcmk_host_map to provide a different "port" (IPMI address) depending on
>> which host is being fenced. In this case, you need only one fence device
>> to be able to fence both hosts. You don't need a clone. (Remember, the
>> node "running" the device merely refers to its monitor, so the cluster
>> can still use the fence device, even if that node crashes.)
>>
 - Monitoring operation on the stonith primitive is dangerous. I read
 that if monitor operations fail for the stonith device, stonith action
 is triggered. I think its not clever to give the cluster the option to
 fence a node just because it has an issue to monitor a fence device.
 That should not be a reason to shutdown a node. What is your opinion on
 this? Can i just set the primitive monitor operation to disabled?
>>>
>>> Monitoring is how you will detect that, for example, the IPMI cable
>>> failed or was unplugged. I do not believe the node will get fenced on
>>> fence agent monitor failing... At least not by default.
>>
>> I am not aware of any situation in which a failing fence monitor
>> triggers a fence. Monitoring is good -- it verifies that the fence
>> device is still working.
>>
>> One concern particular to on-board IPMI devices is that they typically
>> share the same power supply as their host. So if the machine loses
>> power, the cluster can't contact the IPMI to fence it -- which means it
>> 

Re: [ClusterLabs] kind=Optional order constraint not working at startup

[Ken Gaillot]

On 09/21/2016 09:00 AM, Auer, Jens wrote:
> Hi,
> 
> could this be issue 5039 (http://bugs.clusterlabs.org/show_bug.cgi?id=5039)? 
> It sounds similar.

Correct -- "Optional" means honor the constraint only if both resources
are starting *in the same transition*.

shared_fs has to wait for the DRBD promotion, but the other resources
have no such limitation, so they are free to start before shared_fs.

The problem is "... only impacts the startup procedure". Pacemaker
doesn't distinguish start-up from any other state of the cluster. Nodes
(and entire partitions of nodes) can come and go at any time, and any or
all resources can be stopped and started again at any time, so
"start-up" is not really as meaningful as it sounds.

Maybe try an optional constraint of the other resources on the DRBD
promotion. That would make it more likely that all the resources end up
starting in the same transition.

> Cheers,
>   Jens
> 
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter 
> de.cgi.com/pflichtangaben.
> 
> 
> 
> Von: Auer, Jens [jens.a...@cgi.com]
> Gesendet: Mittwoch, 21. September 2016 15:10
> An: users@clusterlabs.org
> Betreff: [ClusterLabs] kind=Optional order constraint not working at startup
> 
> Hi,
> 
> in my cluster setup I have a couple of resources from which I need to start 
> some in specific order. Basically I have two cloned resources that should 
> start after mounting a DRBD filesystem on all nodes plus one resource that 
> start after the clone sets. It is important that this only impacts the 
> startup procedure. Once the system is running stopping or starting one of the 
> clone resources should not impact the other resource's state. From reading 
> the manual, this should be what a local constraint with kind=Optional 
> implements. However, when I start the cluster the filesystem is started after 
> the otehr resources ignoring the ordering constraint.
> 
> My cluster configuration:
> pcs cluster setup --name MDA1PFP MDA1PFP-PCS01,MDA1PFP-S01 
> MDA1PFP-PCS02,MDA1PFP-S02
> pcs cluster start --all
> sleep 5
> crm_attribute --type nodes --node MDA1PFP-PCS01 --name ServerRole --update 
> PRIME
> crm_attribute --type nodes --node MDA1PFP-PCS02 --name ServerRole --update 
> BACKUP
> pcs property set stonith-enabled=false
> pcs resource defaults resource-stickiness=100
> 
> rm -f mda; pcs cluster cib mda
> pcs -f mda property set no-quorum-policy=ignore
> 
> pcs -f mda resource create mda-ip ocf:heartbeat:IPaddr2 ip=192.168.120.20 
> cidr_netmask=24 nic=bond0 op monitor interval=1s
> pcs -f mda constraint location mda-ip prefers MDA1PFP-PCS01=50
> pcs -f mda resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 
> host_list=pf-pep-dev-1  params timeout=1 attempts=3  op monitor interval=1 
> --clone
> pcs -f mda constraint location mda-ip rule score=-INFINITY pingd lt 1 or 
> not_defined pingd
> 
> pcs -f mda resource create ACTIVE ocf:heartbeat:dummy
> pcs -f mda constraint colocation add ACTIVE with mda-ip score=INFINITY
> 
> pcs -f mda resource create drbd1 ocf:linbit:drbd drbd_resource=shared_fs op 
> monitor interval=60s
> pcs -f mda resource master drbd1_sync drbd1 master-max=1 master-node-max=1 
> clone-max=2 clone-node-max=1 notify=true
> pcs -f mda constraint colocation add master drbd1_sync with mda-ip 
> score=INFINITY
> 
> pcs -f mda resource create shared_fs Filesystem device="/dev/drbd1" 
> directory=/shared_fs fstype="xfs"
> pcs -f mda constraint order promote drbd1_sync then start shared_fs
> pcs -f mda constraint colocation add shared_fs with master drbd1_sync 
> score=INFINITY
> 
> pcs -f mda resource create supervisor ocf:pfpep:supervisor params 
> config="/shared_fs/pfpep.ini" --clone
> pcs -f mda resource create snmpAgent ocf:pfpep:snmpAgent params 
> config="/shared_fs/pfpep.ini" --clone
> pcs -f mda resource create clusterSwitchNotification ocf:pfpep:clusterSwitch 
> params config="/shared_fs/pfpep.ini"
> 
> pcs -f mda constraint order start shared_fs then snmpAgent-clone  
> kind=Optional
> pcs -f mda constraint order start shared_fs then supervisor-clone 
> kind=Optional
> pcs -f mda constraint order start snmpAgent-clone then supervisor-clone 
> kind=Optional
> pcs -f mda constraint order start supervisor-clone then 
> clusterSwitchNotification kind=Optional
> pcs -f mda constraint colocation add clusterSwitchNotification with shared_fs 
> score=INFINITY
> 
> pcs cluster cib-push mda
> 
> The order of resource startup in the log file is:
> Sep 21 13:01:21 MDA1PFP-S01 crmd[2760]:  notice: Operation snmpAgent_start_0: 
> ok (node=MDA1PFP-PCS01, call=40, rc=0, cib-update=82, confirmed=true)
> Sep 21 13:01:21 MDA1PFP-S01 crmd[2760]:  notice: Operation drbd1_start_0: ok 
> (node=MDA1PFP-PCS01, call=39, rc=0, cib-update=83, confirmed=true)

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/21/2016 02:23 AM, Kristoffer Grönlund wrote:
> First of all, is there a use case for when fence-after-3-failures is a
> useful behavior? I seem to recall some case where someone expected that
> to be the behavior and were surprised by how pacemaker works, but that
> problem wouldn't be helped by adding another option for them not to know
> about.

I think I've most often encountered it with ignore/block. Sometimes
users have one particular service that's buggy and not really important,
so they ignore errors (or block). But they would like to try restarting
it a few times first.

I think fence-after-3-failures would make as much sense as
fence-immediately. The idea behind restarting a few times then taking a
more drastic action is that restarting is for the case where the service
crashed or is in a buggy state, and if that doesn't work, maybe
something's wrong with the node.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/20/2016 07:51 PM, Andrew Beekhof wrote:
> 
> 
> On Wed, Sep 21, 2016 at 6:25 AM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> Hi everybody,
> 
> Currently, Pacemaker's on-fail property allows you to configure how the
> cluster reacts to operation failures. The default "restart" means try to
> restart on the same node, optionally moving to another node once
> migration-threshold is reached. Other possibilities are "ignore",
> "block", "stop", "fence", and "standby".
> 
> Occasionally, we get requests to have something like migration-threshold
> for values besides restart. For example, try restarting the resource on
> the same node 3 times, then fence.
> 
> I'd like to get your feedback on two alternative approaches we're
> considering.
> 
> ###
> 
> Our first proposed approach would add a new hard-fail-threshold
> operation property. If specified, the cluster would first try restarting
> the resource on the same node, 
> 
> 
> Well, just as now, it would be _allowed_ to start on the same node, but
> this is not guaranteed.
>  
> 
> before doing the on-fail handling.
> 
> For example, you could configure a promote operation with
> hard-fail-threshold=3 and on-fail=fence, to fence the node after 3
> failures.
> 
> 
> One point that's not settled is whether failures of *any* operation
> would count toward the 3 failures (which is how migration-threshold
> works now), or only failures of the specified operation.
> 
> 
> I think if hard-fail-threshold is per-op, then only failures of that
> operation should count.
>  
> 
> 
> Currently, if a start fails (but is retried successfully), then a
> promote fails (but is retried successfully), then a monitor fails, the
> resource will move to another node if migration-threshold=3. We could
> keep that behavior with hard-fail-threshold, or only count monitor
> failures toward monitor's hard-fail-threshold. Each alternative has
> advantages and disadvantages.
> 
> ###
> 
> The second proposed approach would add a new on-restart-fail resource
> property.
> 
> Same as now, on-fail set to anything but restart would be done
> immediately after the first failure. A new value, "ban", would
> immediately move the resource to another node. (on-fail=ban would behave
> like on-fail=restart with migration-threshold=1.)
> 
> When on-fail=restart, and restarting on the same node doesn't work, the
> cluster would do the on-restart-fail handling. on-restart-fail would
> allow the same values as on-fail (minus "restart"), and would default to
> "ban". 
> 
> 
> I do wish you well tracking "is this a restart" across demote -> stop ->
> start -> promote in 4 different transitions :-)
>  
> 
> 
> So, if you want to fence immediately after any promote failure, you
> would still configure on-fail=fence; if you want to try restarting a few
> times first, you would configure on-fail=restart and
> on-restart-fail=fence.
> 
> This approach keeps the current threshold behavior -- failures of any
> operation count toward the threshold. We'd rename migration-threshold to
> something like hard-fail-threshold, since it would apply to more than
> just migration, but unlike the first approach, it would stay a resource
> property.
> 
> ###
> 
> Comparing the two approaches, the first is more flexible, but also more
> complex and potentially confusing.
> 
> 
> More complex to implement or more complex to configure?

I was thinking more complex in behavior, so perhaps harder to follow /
expect.

For example, "After two start failures, fence this node; after three
promote failures, put the node in standby; but if a monitor failure is
the third operation failure of any type, then move the resource to
another node."

Granted, someone would have to inflict that on themselves :) but another
sysadmin / support tech / etc. who had to deal with the config later
might have trouble following it.

To keep the current default behavior, the default would be complicated,
too: "1 for start and stop operations, and 0 for other operations" where
"0 is equivalent to 1 except when on-fail=restart, in which case
migration-threshold will be used instead".

And then add to that tracking fail-count per node+resource+operation
combination, with the associated status output and cleanup options.
"crm_mon -f" currently shows failures like:

* Node node1:
   rs

Re: [ClusterLabs] Virtual ip resource restarted on node with down network device

[Ken Gaillot]

On 09/19/2016 10:04 AM, Jan Pokorný wrote:
> On 19/09/16 10:18 +, Auer, Jens wrote:
>> Ok, after reading the log files again I found 
>>
>> Sep 19 10:03:45 MDA1PFP-S01 crmd[7797]:  notice: Initiating action 3: stop 
>> mda-ip_stop_0 on MDA1PFP-PCS01 (local)
>> Sep 19 10:03:45 MDA1PFP-S01 crmd[7797]:  notice: 
>> MDA1PFP-PCS01-mda-ip_monitor_1000:14 [ ocf-exit-reason:Unknown interface 
>> [bond0] No such device.\n ]
>> Sep 19 10:03:45 MDA1PFP-S01 IPaddr2(mda-ip)[8745]: ERROR: Unknown interface 
>> [bond0] No such device.
>> Sep 19 10:03:45 MDA1PFP-S01 IPaddr2(mda-ip)[8745]: WARNING: [findif] failed
>> Sep 19 10:03:45 MDA1PFP-S01 lrmd[7794]:  notice: mda-ip_stop_0:8745:stderr [ 
>> ocf-exit-reason:Unknown interface [bond0] No such device. ]
>> Sep 19 10:03:45 MDA1PFP-S01 crmd[7797]:  notice: Operation mda-ip_stop_0: ok 
>> (node=MDA1PFP-PCS01, call=16, rc=0, cib-update=49, confirmed=true)
>> Sep 19 10:03:46 MDA1PFP-S01 crmd[7797]:  notice: Transition 3 (Complete=2, 
>> Pending=0, Fired=0, Skipped=0, Incomplete=0, 
>> Source=/var/lib/pacemaker/pengine/pe-input-501.bz2): Complete
>> Sep 19 10:03:46 MDA1PFP-S01 crmd[7797]:  notice: State transition 
>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL 
>> origin=notify_crmd ]
>> Sep 19 10:03:46 MDA1PFP-S01 crmd[7797]:  notice: State transition S_IDLE -> 
>> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL 
>> origin=abort_transition_graph ]
>> Sep 19 10:03:46 MDA1PFP-S01 pengine[7796]:  notice: On loss of CCM Quorum: 
>> Ignore
>> Sep 19 10:03:46 MDA1PFP-S01 pengine[7796]: warning: Processing failed op 
>> monitor for mda-ip on MDA1PFP-PCS01: not configured (6)
>> Sep 19 10:03:46 MDA1PFP-S01 pengine[7796]:   error: Preventing mda-ip from 
>> re-starting anywhere: operation monitor failed 'not configured' (6)
>>
>> I think that explains why the resource is not started on the other
>> node, but I am not sure this is a good decision. It seems to be a
>> little harsh to prevent the resource from starting anywhere,
>> especially considering that the other node will be able to start the
>> resource. 

The resource agent is supposed to return "not configured" only when the
*pacemaker* configuration of the resource is inherently invalid, so
there's no chance of it starting anywhere.

As Jan suggested, make sure you've applied any resource-agents updates.
If that doesn't fix it, it sounds like a bug in the agent, or something
really is wrong with your pacemaker resource config.

> 
> The problem to start with is that based on 
> 
>> Sep 19 10:03:45 MDA1PFP-S01 IPaddr2(mda-ip)[8745]: ERROR: Unknown interface 
>> [bond0] No such device.
>> Sep 19 10:03:45 MDA1PFP-S01 IPaddr2(mda-ip)[8745]: WARNING: [findif] failed
> 
> you may be using too ancient version resource-agents:
> 
> https://github.com/ClusterLabs/resource-agents/pull/320
> 
> so until you update, the troubleshooting would be quite moot.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] No DRBD resource promoted to master in Active/Passive setup

[Ken Gaillot]

On 09/19/2016 09:48 AM, Auer, Jens wrote:
> Hi,
> 
>> Is the network interface being taken down here used for corosync
>> communication? If so, that is a node-level failure, and pacemaker will
>> fence.
> 
> We have different connections on each server:
> - A bonded 10GB network card for data traffic that will be accessed via a 
> virtual ip managed by pacemaker in 192.168.120.1/24. In the cluster nodes 
> MDA1PFP-S01 and MDA1PFP-S02 are assigned to 192.168.120.10 and 192.168.120.11.
> 
> - A dedicated back-to-back connection for corosync heartbeats in 
> 192.168.121.1/24. MDA1PFP-PCS01 and MDA1PFP-S02 are assigned to 
> 192.168.121.10 and 192.168.121.11. When the cluster is created, we use these 
> as primary node names and use the 10GB device as a second backup connection 
> for increased reliability: pcs cluster setup --name MDA1PFP 
> MDA1PFP-PCS01,MDA1PFP-S01 MDA1PFP-PCS02,MDA1PFP-S02
> 
> - A dedicated back-to-back connection for drbd in 192.168.122.1/24. Hosts 
> MDA1PFP-DRBD01 and MDA1PFP-DRBD02 are assigned 192.168.23.10 and 
> 192.168.123.11.

Ah, nice.

> Given that I think it is not a node-level failure. pcs status also reports 
> the nodes as online. I think this should not trigger fencing from pacemaker.
> 
>> When DRBD is configured with 'fencing resource-only' and 'fence-peer
>> "/usr/lib/drbd/crm-fence-peer.sh";', and DRBD detects a network outage,
>> it will try to add a constraint that prevents the other node from
>> becoming master. It removes the constraint when connectivity is restored.
> 
>> I am not familiar with all the under-the-hood details, but IIUC, if
>> pacemaker actually fences the node, then the other node can still take
>> over the DRBD. But if there is a network outage and no pacemaker
>> fencing, then you'll see the behavior you describe -- DRBD prevents
>> master takeover, to avoid stale data being used.
> 
> This is my understanding as well, but there should be no network outage for 
> DRBD. I can reproduce the behavior by stopping cluster nodes which DRBD seems 
> to interpret as network outages since it cannot communicate with the shutdown 
> node anymore. Maybe I should ask on the DRBD mailing list?

OK, I think I follow you now: you're ifdown'ing the data traffic
interface, but the interfaces for both corosync and DRBD traffic are
still up. So, pacemaker detects the virtual IP failure on the traffic
interface, and correctly recovers the IP on the other node, but the DRBD
master role is not recovered.

If the behavior goes away when you remove the DRBD fencing config, then
it sounds like DRBD is seeing it as a network outage, and is adding the
constraint to prevent a stale master. Yes, I think that would be worth
bringing up on the DRBD list, though there might be some DRBD users here
who can chime in, too.

> Cheers,
>   Jens
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter 
> de.cgi.com/pflichtangaben.
> 
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI 
> Group Inc. and its affiliates may be contained in this message. If you are 
> not a recipient indicated or intended in this message (or responsible for 
> delivery of this message to such person), or you think for any reason that 
> this message may have been addressed to you in error, you may not use or copy 
> or deliver this message to anyone else. In such case, you should destroy this 
> message and are asked to notify the sender by reply e-mail.
> 
> 
> Von: Ken Gaillot [kgail...@redhat.com]
> Gesendet: Montag, 19. September 2016 16:28
> An: Auer, Jens; Cluster Labs - All topics related to open-source clustering 
> welcomed
> Betreff: Re: [ClusterLabs] No DRBD resource promoted to master in 
> Active/Passive setup
> 
> On 09/19/2016 02:31 AM, Auer, Jens wrote:
>> Hi,
>>
>> I am not sure that pacemaker should do any fencing here. In my setting, 
>> corosync is configured to use a back-to-back connection for heartbeats. This 
>> is different subnet then used by the ping resource that checks the network 
>> connectivity and detects a failure. In my test, I bring down the network 
>> device used by ping and this triggers the failover. The node status is known 
>> by pacemaker since it receives heartbeats and it only a resource failure. I 
>> asked for fencing conditions a few days ago, and basically was asserted that 
>> resource failure should not trigger STONITH actions if not explicitly 
>> configured.
> 
> Is the network i

Re: [ClusterLabs] [Linux-ha-dev] Announcing crmsh release 2.1.7

[Ken Gaillot]

On 09/23/2016 06:59 AM, Kostiantyn Ponomarenko wrote:
>>> Out of curiosity: What do you use it for, where the two_node option
> is not sufficient?
> 
> Alongside with starting the cluster with two nodes I need that
> possibility of starting the cluster with only one node.
> "two_node" option doesn't provide that.

Actually it can, if you use "two_node: 1" with "wait_for_all: 0".

The risk with that configuration is that both nodes can start without
seeing each other, and both start resources.

> 
> Thank you,
> Kostia
> 
> On Fri, Sep 2, 2016 at 11:33 AM, Kristoffer Grönlund  > wrote:
> 
> Kostiantyn Ponomarenko  > writes:
> 
> > Hi,
> >
> >>> If "scripts: no-quorum-policy=ignore" is becoming depreciated
> > Are there any plans to get rid of this option?
> > Am I missing something?
> 
> The above is talking about crmsh cluster configuration scripts, not core
> Pacemaker. As far as I know, no-quorum-policy=ignore is not being
> deprecated in Pacemaker.
> 
> However, it is no longer the recommended configuration for two node
> clusters.
> 
> >
> > PS: this option is very useful (vital) to me. And "two_node" option 
> won't
> > replace it.
> >
> 
> Out of curiosity: What do you use it for, where the two_node option is
> not sufficient?
> 
> Cheers,
> Kristoffer
> 
> --
> // Kristoffer Grönlund
> // kgronl...@suse.com 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/22/2016 09:53 AM, Jan Pokorný wrote:
> On 22/09/16 08:42 +0200, Kristoffer Grönlund wrote:
>> Ken Gaillot <kgail...@redhat.com> writes:
>>
>>> I'm not saying it's a bad idea, just that it's more complicated than it
>>> first sounds, so it's worth thinking through the implications.
>>
>> Thinking about it and looking at how complicated it gets, maybe what
>> you'd really want, to make it clearer for the user, is the ability to
>> explicitly configure the behavior, either globally or per-resource. So
>> instead of having to tweak a set of variables that interact in complex
>> ways, you'd configure something like rule expressions,
>>
>> 
>>   
>>   
>>   
>> 
>>
>> So, try to restart the service 3 times, if that fails migrate the
>> service, if it still fails, fence the node.
>>
>> (obviously the details and XML syntax are just an example)
>>
>> This would then replace on-fail, migration-threshold, etc.
> 
> I must admit that in previous emails in this thread, I wasn't able to
> follow during the first pass, which is not the case with this procedural
> (sequence-ordered) approach.  Though someone can argue it doesn't take
> type of operation into account, which might again open the door for
> non-obvious interactions.

"restart" is the only on-fail value that it makes sense to escalate.

block/stop/fence/standby are final. Block means "don't touch the
resource again", so there can't be any further response to failures.
Stop/fence/standby move the resource off the local node, so failure
handling is reset (there are 0 failures on the new node to begin with).

"Ignore" is theoretically possible to escalate, e.g. "ignore 3 failures
then migrate", but I can't think of a real-world situation where that
makes sense, and it would be a significant re-implementation of "ignore"
(which currently ignores the state of having failed, as opposed to a
particular instance of failure).

What the interface needs to express is: "If this operation fails,
optionally try a soft recovery [always stop+start], but if  failures
occur on the same node, proceed to a [configurable] hard recovery".

And of course the interface will need to be different depending on how
certain details are decided, e.g. whether any failures count toward 
or just failures of one particular operation type, and whether the hard
recovery type can vary depending on what operation failed.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/22/2016 10:43 AM, Jan Pokorný wrote:
> On 21/09/16 10:51 +1000, Andrew Beekhof wrote:
>> On Wed, Sep 21, 2016 at 6:25 AM, Ken Gaillot <kgail...@redhat.com> wrote:
>>> Our first proposed approach would add a new hard-fail-threshold
>>> operation property. If specified, the cluster would first try restarting
>>> the resource on the same node,
>>
>>
>> Well, just as now, it would be _allowed_ to start on the same node, but
>> this is not guaranteed.
> 
> Yeah, I should attend doublethink classes to understand "the same
> node" term better:
> 
> https://github.com/ClusterLabs/pacemaker/pull/1146/commits/3b3fc1fd8f2c95d8ab757711cf096cf231f27941

"Same node" is really a shorthand to hand-wave some details, because
that's what will typically happen.

The exact behavior is: "If the fail-count on this node reaches , ban
this node from running the resource."

That's not the same as *requiring* the resource to restart on the same
node before  is reached. As in any situation, Pacemaker will
re-evaluate the current state of the cluster, and choose the best node
to try starting the resource on.

For example, consider if the failed resource with on-fail=restart is
colocated with another resource with on-fail=standby that also failed,
then the whole node will be put in standby, and the original resource
will of course move away. It will be restarted, but the start will
happen on another node.

There are endless such scenarios, so "try restarting on the same node"
is not really accurate. To be accurate, I should have said something
like "try restarting without banning the node with the failure".

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/22/2016 12:58 PM, Kristoffer Grönlund wrote:
> Ken Gaillot <kgail...@redhat.com> writes:
>>
>> "restart" is the only on-fail value that it makes sense to escalate.
>>
>> block/stop/fence/standby are final. Block means "don't touch the
>> resource again", so there can't be any further response to failures.
>> Stop/fence/standby move the resource off the local node, so failure
>> handling is reset (there are 0 failures on the new node to begin with).
> 
> Hrm. If a restart potentially migrates the resource to a different node,
> is the failcount reset then as well? If so, wouldn't that complicate the
> hard-fail-threshold variable too, since potentially, the resource could
> keep migrating between nodes and since the failcount is reset on each
> migration, it would never reach the hard-fail-threshold. (or am I
> missing something?)

The failure count is specific to each node. By "failure handling is
reset" I mean that when the resource moves to a different node, the
failure count of the original node no longer matters -- the new node's
failure count is now what matters.

A node's failure count is reset only when the user manually clears it,
or the node is rebooted. Also, resources may have a failure-timeout
configured, in which case the count will go down as failures expire.

So, a resource with on-fail=restart would never go back to a node where
it had previously reached the threshold, unless that node's fail count
were cleared in one of those ways.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] best practice fencing with ipmi in 2node-setups / cloneresource/monitor/timeout

[Ken Gaillot]

On 09/20/2016 06:42 AM, Digimer wrote:
> On 20/09/16 06:59 AM, Stefan Bauer wrote:
>> Hi,
>>
>> i run a 2 node cluster and want to be save in split-brain scenarios. For
>> this i setup external/ipmi to stonith the other node.
> 
> Please use 'fence_ipmilan'. I believe that the older external/ipmi are
> deprecated (someone correct me if I am wrong on this).

It's just an alternative. The "external/" agents come with the
cluster-glue package, which isn't provided by some distributions (such
as RHEL and its derivatives), so it's "deprecated" on those only.

>> Some possible issues jumped to my mind and i would ike to find the best
>> practice solution:
>>
>> - I have a primitive for each node to stonith. Many documents and guides
>> recommend to never let them run on the host it should fence. I would
>> setup clone resources to avoid dealing with locations that would also
>> influence scoring. Does that make sense?
> 
> Since v1.1.10 of pacemaker, you don't have to worry about this.
> Pacemaker is smart enough to know where to run a fence call from in
> order to terminate a target.

Right, fence devices can run anywhere now, and in fact they don't even
have to be "running" for pacemaker to use them -- as long as they are
configured and not intentionally disabled, pacemaker will use them.

There is still a slight advantage to not running a fence device on a
node it can fence. "Running" a fence device in pacemaker really means
running the recurring monitor for it. Since the node that runs the
monitor has "verified" access to the device, pacemaker will prefer to
use it to execute that device. However, pacemaker will not use a node to
fence itself, except as a last resort if no other node is available. So,
running a fence device on a node it can fence means that the preference
is lost.

That's a very minor detail, not worth worrying about. It's more a matter
of personal preference.

In this particular case, a more relevant concern is that you need
different configurations for the different targets (the IPMI address is
different).

One approach is to define two different fence devices, each with one
IPMI address. In that case, it makes sense to use the location
constraints to ensure the device prefers the node that's not its target.

Another approach (if the fence agent supports it) is to use
pcmk_host_map to provide a different "port" (IPMI address) depending on
which host is being fenced. In this case, you need only one fence device
to be able to fence both hosts. You don't need a clone. (Remember, the
node "running" the device merely refers to its monitor, so the cluster
can still use the fence device, even if that node crashes.)

>> - Monitoring operation on the stonith primitive is dangerous. I read
>> that if monitor operations fail for the stonith device, stonith action
>> is triggered. I think its not clever to give the cluster the option to
>> fence a node just because it has an issue to monitor a fence device.
>> That should not be a reason to shutdown a node. What is your opinion on
>> this? Can i just set the primitive monitor operation to disabled?
> 
> Monitoring is how you will detect that, for example, the IPMI cable
> failed or was unplugged. I do not believe the node will get fenced on
> fence agent monitor failing... At least not by default.

I am not aware of any situation in which a failing fence monitor
triggers a fence. Monitoring is good -- it verifies that the fence
device is still working.

One concern particular to on-board IPMI devices is that they typically
share the same power supply as their host. So if the machine loses
power, the cluster can't contact the IPMI to fence it -- which means it
will be unable to recover any resources from the lost node. (It can't
assume the node lost power -- it's possible just network connectivity
between the two nodes was lost.)

The only way around that is to have a second fence device (such as an
intelligent power switch). If the cluster can't reach the IPMI, it will
try the second device.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Virtual ip resource restarted on node with down network device

[Ken Gaillot]

On 09/16/2016 10:43 AM, Auer, Jens wrote:
> Hi,
> 
> thanks for the help.
> 
>> I'm not sure what you mean by "the device the virtual ip is attached
>> to", but a separate question is why the resource agent reported that
>> restarting the IP was successful, even though that device was
>> unavailable. If the monitor failed when the device was made unavailable,
>> I would expect the restart to fail as well.
> 
> I created the virtual ip with parameter nic=bond0, and this is the device I 
> am bringing down
> and was referring to in my question. I think the current behavior is a little 
> inconsistent. I bring 
> down the device and pacemaker recognizes this and restarts the resource. 
> However, the monitor
> then should fail again, but it just doesn't detect any problems. 

That is odd. Pacemaker is just acting on what the resource agent
reports, so the issue will be in the agent. Agents are usually fairly
simple shell scripts, so you could just look at what it does, and try
running those commands manually and see what the results are.

There are also some pcs commands debug-start, debug-monitor, etc. that
run the agent directly, using the configuration from the cluster.

And you can look in the system log and pacemaker detail log around the
time of the incident for any interesting messages.

> Cheers,
>   Jens
> 
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter 
> de.cgi.com/pflichtangaben.
> 
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI 
> Group Inc. and its affiliates may be contained in this message. If you are 
> not a recipient indicated or intended in this message (or responsible for 
> delivery of this message to such person), or you think for any reason that 
> this message may have been addressed to you in error, you may not use or copy 
> or deliver this message to anyone else. In such case, you should destroy this 
> message and are asked to notify the sender by reply e-mail.
> 
> 
> Von: Ken Gaillot [kgail...@redhat.com]
> Gesendet: Freitag, 16. September 2016 17:27
> An: users@clusterlabs.org
> Betreff: Re: [ClusterLabs] Virtual ip resource restarted on node with down 
> network device
> 
> On 09/16/2016 10:08 AM, Auer, Jens wrote:
>> Hi,
>>
>> I have configured an Active/Passive cluster to host a virtual ip
>> address. To test failovers, I shutdown the device the virtual ip is
>> attached to and expected that it moves to the other node. However, the
>> virtual ip is detected as FAILED, but is then restarted on the same
>> node. I was able to solve this by using a ping resource which we want to
>> do anyway, but I am wondering why the resource is restarted on the node
>> and no failure is detected anymore.
> 
> If a *node* fails, pacemaker will recover all its resources elsewhere,
> if possible.
> 
> If a *resource* fails but the node is OK, the response is configurable,
> via the "on-fail" operation option and "migration-threshold" resource
> option.
> 
> By default, on-fail=restart for monitor operations, and
> migration-threshold=INFINITY. This means that if a monitor fails,
> pacemaker will attempt to restart the resource on the same node.
> 
> To get an immediate failover of the resource, set migration-threshold=1
> on the resource.
> 
> I'm not sure what you mean by "the device the virtual ip is attached
> to", but a separate question is why the resource agent reported that
> restarting the IP was successful, even though that device was
> unavailable. If the monitor failed when the device was made unavailable,
> I would expect the restart to fail as well.
> 
>>
>> On my setup, this is very easy to reproduce:
>> 1. Start cluster with virtual ip
>> 2. On the node hosting the virtual ip, bring down the network device
>> with ifdown
>> => The resource is detected as failed
>> => The resource is restarted
>> => No failures are dected from now on
>>
>> Best wishes,
>>   Jens
>>
>> --
>> *Jens Auer *| CGI | Software-Engineer
>> CGI (Germany) GmbH & Co. KG
>> Rheinstraße 95 | 64295 Darmstadt | Germany
>> T: +49 6151 36860 154
>> _jens.auer@cgi.com_ <mailto:jens.a...@cgi.com>
>> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
>> unter _de.cgi.com/pflichtangaben_ <http://de.cgi.com/pflichtangaben>.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Virtual ip resource restarted on node with down network device

[Ken Gaillot]

On 09/16/2016 10:08 AM, Auer, Jens wrote:
> Hi,
> 
> I have configured an Active/Passive cluster to host a virtual ip
> address. To test failovers, I shutdown the device the virtual ip is
> attached to and expected that it moves to the other node. However, the
> virtual ip is detected as FAILED, but is then restarted on the same
> node. I was able to solve this by using a ping resource which we want to
> do anyway, but I am wondering why the resource is restarted on the node
> and no failure is detected anymore.

If a *node* fails, pacemaker will recover all its resources elsewhere,
if possible.

If a *resource* fails but the node is OK, the response is configurable,
via the "on-fail" operation option and "migration-threshold" resource
option.

By default, on-fail=restart for monitor operations, and
migration-threshold=INFINITY. This means that if a monitor fails,
pacemaker will attempt to restart the resource on the same node.

To get an immediate failover of the resource, set migration-threshold=1
on the resource.

I'm not sure what you mean by "the device the virtual ip is attached
to", but a separate question is why the resource agent reported that
restarting the IP was successful, even though that device was
unavailable. If the monitor failed when the device was made unavailable,
I would expect the restart to fail as well.

> 
> On my setup, this is very easy to reproduce:
> 1. Start cluster with virtual ip
> 2. On the node hosting the virtual ip, bring down the network device
> with ifdown
> => The resource is detected as failed
> => The resource is restarted
> => No failures are dected from now on
> 
> Best wishes,
>   Jens
> 
> --
> *Jens Auer *| CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> _jens.auer@cgi.com_ 
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter _de.cgi.com/pflichtangaben_ .\

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Preferred location is sometimes ignored

[Ken Gaillot]

On 09/16/2016 10:59 AM, Auer, Jens wrote:
> On 09/16/2016 09:45 AM, Auer, Jens wrote:
>>> Hi,
>>>
>>> MDA1PFP-S01 14:41:35 1805 0 ~ # pcs constraint --full
>>> Location Constraints:
>>>   Resource: mda-ip
>>> Enabled on: MDA1PFP-PCS01 (score:50)
>>> (id:location-mda-ip-MDA1PFP-PCS01-50)
>>> Constraint: location-mda-ip
>>>   Rule: score=-INFINITY boolean-op=or  (id:location-mda-ip-rule)
>>> Expression: pingd lt 1  (id:location-mda-ip-rule-expr)
>>> Expression: not_defined pingd  (id:location-mda-ip-rule-expr-1)
>> If I'm reading this right, you have two separate location constraints
>> for mda-ip, a positive preference for one particular node (score 50 for
>> MDA1PFP-PCS01), and a -INFINITY preference whenever the ping attribute
> i>s bad.
> 
> Yes, I have two location constraints. The first one is a preferred location 
> for startup and the second one should move the resource if ping fails. When I 
> start the nodes for the test the ping should not fail as everything is ok. So 
> the location preference should be the only score taken into account. So I am 
> wondering why the resource is sometimes not started on the preferred node. 
> 
>>> Which constraint is sometimes ignored?
> The location constraint of 50 for node MDA1PFP-PCS01.

One possibility I see is: pingd happens to set the attribute for the
other node first; the cluster sees pingd not defined on the preferred
node, and so the -INFINITY overrides the +50, and it starts the IP on
the other node; and (I'm guessing) you have stickiness > 50 that keeps
the IP there even after pingd sets the attribute for the preferred node.

To avoid guessing :) look in the pacemaker detail log on the DC. The DC
will be the node with the most "pengine:" messages around that time.
You'll see messages like "Assigning (node) to (resource)" and various
results of actions and so forth, that might give some insight.

>>> Is there a way to get more debugging output from pcs, e.g. what
>>> triggered actions, which scores are computed and from which values?
> 
>> Not from pcs, but there are some lower-level tools that can sometimes be
>> helpful. "crm_simulate -sL" will show all the scores that went into the
>> current placement.
> 
> Thanks for the tip. Is there any way to get more output when starting the 
> cluster up initially? This is only problem I am concerned here because 
> afterwards it works fine.

I don't think so, the logs are the best alternative.

> Cheers,
>   Jens
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD failover in Pacemaker

[Ken Gaillot]

On 09/06/2016 02:04 PM, Devin Ortner wrote:
> I have a 2-node cluster running CentOS 6.8 and Pacemaker with DRBD. I have 
> been using the "Clusters from Scratch" documentation to create my cluster and 
> I am running into a problem where DRBD is not failing over to the other node 
> when one goes down. Here is my "pcs status" prior to when it is supposed to 
> fail over:

The most up-to-date version of Clusters From Scratch targets CentOS 7.1,
which has corosync 2, pcs, and a recent pacemaker:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html


There is an older version targeting Fedora 13, which has CMAN, corosync
1, the crm shell, and an older pacemaker:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html-single/Clusters_from_Scratch/index.html


Your system is in between, with CMAN, corosync 1, pcs, and a newer
pacemaker, so you might want to compare the two guides as you go.

> --
> 
> [root@node1 ~]# pcs status
> Cluster name: webcluster
> Last updated: Tue Sep  6 14:50:21 2016Last change: Tue Sep  6 
> 14:50:17 2016 by root via crm_attribute on node1
> Stack: cman
> Current DC: node2 (version 1.1.14-8.el6_8.1-70404b0) - partition with quorum
> 2 nodes and 5 resources configured
> 
> Online: [ node1 node2 ]
> 
> Full list of resources:
> 
>  Cluster_VIP  (ocf::heartbeat:IPaddr2):   Started node1
>  Master/Slave Set: ClusterDBclone [ClusterDB]
>  Masters: [ node1 ]
>  Slaves: [ node2 ]
>  ClusterFS(ocf::heartbeat:Filesystem):Started node1
>  WebSite  (ocf::heartbeat:apache):Started node1
> 
> Failed Actions:
> * ClusterFS_start_0 on node2 'unknown error' (1): call=61, status=complete, 
> exitreason='none',
> last-rc-change='Tue Sep  6 13:15:00 2016', queued=0ms, exec=40ms

'unknown error' means the Filesystem resource agent returned an error
status. Check the system log for messages from the resource agent to see
what the error actually was.

> 
> PCSD Status:
>   node1: Online
>   node2: Online
> 
> [root@node1 ~]#
> 
> When I put node1 in standby everything fails over except DRBD:
> --
> 
> [root@node1 ~]# pcs cluster standby node1
> [root@node1 ~]# pcs status
> Cluster name: webcluster
> Last updated: Tue Sep  6 14:53:45 2016Last change: Tue Sep  6 
> 14:53:37 2016 by root via cibadmin on node2
> Stack: cman
> Current DC: node2 (version 1.1.14-8.el6_8.1-70404b0) - partition with quorum
> 2 nodes and 5 resources configured
> 
> Node node1: standby
> Online: [ node2 ]
> 
> Full list of resources:
> 
>  Cluster_VIP  (ocf::heartbeat:IPaddr2):   Started node2
>  Master/Slave Set: ClusterDBclone [ClusterDB]
>  Slaves: [ node2 ]
>  Stopped: [ node1 ]
>  ClusterFS(ocf::heartbeat:Filesystem):Stopped
>  WebSite  (ocf::heartbeat:apache):Started node2
> 
> Failed Actions:
> * ClusterFS_start_0 on node2 'unknown error' (1): call=61, status=complete, 
> exitreason='none',
> last-rc-change='Tue Sep  6 13:15:00 2016', queued=0ms, exec=40ms
> 
> 
> PCSD Status:
>   node1: Online
>   node2: Online
> 
> [root@node1 ~]#
> 
> I have pasted the contents of "/var/log/messages" here: 
> http://pastebin.com/0i0FMzGZ 
> Here is my Configuration: http://pastebin.com/HqqBV90p 

One thing lacking in Clusters From Scratch is that master/slave
resources such as ClusterDB should have two monitor operations, one for
the master role and one for the slave role. Something like:

op monitor interval=59s role=Master op monitor interval=60s role=Slave

Not sure if that will help your issue, but it's a good idea.

Another thing the guide should do differently is configure stonith
before drbd.

Once you have fencing working in pacemaker, take a look at LINBIT's DRBD
User Guide for whatever version you installed (
https://www.drbd.org/en/doc ) and look for the Pacemaker chapter. It
will describe how to connect the fencing between DRBD and Pacemaker's CIB.

Your constraints need a few tweaks: you have two "ClusterFS with
ClusterDBclone", one with "with-rsc-role:Master" and one without. You
want the one with Master. Your "Cluster_VIP with ClusterDBclone" should
also be with Master. When you colocate with a clone without specifying
the role, it means the resource can run anywhere any instance of the
clone is running (whether slave or master). In this case, you only want
the resources to run with the master instance, so you need to specify
that. That could be the main source of your issue.

> When I unstandby node1, it comes back as the master for the DRBD and 
> everything else stays running on node2 (Which is fine because I haven't setup 
> colocation constraints for that)
> Here is what I have after node1 is back: 
> -
> 
> [root@node1 

Re: [ClusterLabs] Mysql slave did not start replication after failure, and read-only IP also remained active on the much outdated slave

[Ken Gaillot]

On 08/22/2016 03:56 PM, Attila Megyeri wrote:
> Hi Ken,
> 
> Thanks a lot for your feedback, my answers are inline.
> 
> 
> 
>> -Original Message-----
>> From: Ken Gaillot [mailto:kgail...@redhat.com]
>> Sent: Monday, August 22, 2016 4:12 PM
>> To: users@clusterlabs.org
>> Subject: Re: [ClusterLabs] Mysql slave did not start replication after 
>> failure,
>> and read-only IP also remained active on the much outdated slave
>>
>> On 08/22/2016 07:24 AM, Attila Megyeri wrote:
>>> Hi Andrei,
>>>
>>> I waited several hours, and nothing happened.
>>
>> And actually, we can see from the configuration you provided that
>> cluster-recheck-interval is 2 minutes.
>>
>> I don't see anything about stonith; is it enabled and tested? This looks
>> like a situation where stonith would come into play. I know that power
>> fencing can be rough on a MySQL database, but perhaps intelligent
>> switches with network fencing would be appropriate.
> 
> Yes, there is no stonith in place because we found it too agressive for this 
> purpose. And to be honest I'm not sure if that would have worked here.

The problem is, in a situation like this where corosync communication is
broken, both nodes will think they are the only surviving node, and
bring up all resources. Network fencing would isolate one of the nodes
so at least it can't cause any serious trouble.

>> The "Corosync main process was not scheduled" message is the start of
>> the trouble. It means the system was overloaded and corosync didn't get
>> any CPU time, so it couldn't maintain cluster communication.
>>
>  
> True, this was the cause of the issue, but we still couldn't find a solution 
> to get rid of the original problem.
> Nevertheless I think that the issue is that the RA did not properly detect 
> the state of mysql.
> 
> 
>> Probably the most useful thing would be to upgrade to a recent version
>> of corosync+pacemaker+resource-agents. Recent corosync versions run with
>> realtime priority, which makes this much less likely.
>>
>> Other than that, figure out what the load issue was, and try to prevent
>> it from recurring.
>>
> 
> Whereas the original problem might have been caused by corosync CPU issue, I 
> am sure that once the load was gone the proper mysql state should have been 
> detected.
> The RA, responsible for this is almost the latest version, and I did not see 
> any changes related to this functionality.
> 
>> I'm not familiar enough with the RA to comment on its behavior. If you
>> think it's suspect, check the logs during the incident for messages from
>> the RA.
>>
> 
> So did I, but there are very few details logged while this was happening, so 
> I am pretty much stuck :(
> 
> I thought that someone might have a clue what is wrong in the RA  - that 
> causes this fake state detection.
> (Un)fortunately I cannot reproduce this situation for now.

Perhaps with the split-brain, the slave tried to come up as master?

>  
> Who could help me in troubleshooting this?
> 
> Thanks,
> Attila
> 
> 
> 
> 
>>> I assume that the RA does not treat this case properly. Mysql was running,
>> but the "show slave status" command returned something that the RA was
>> not prepared to parse, and instead of reporting a non-readable attribute, it
>> returned some generic error, that did not stop the server.
>>>
>>> Rgds,
>>> Attila
>>>
>>>
>>> -Original Message-
>>> From: Andrei Borzenkov [mailto:arvidj...@gmail.com]
>>> Sent: Monday, August 22, 2016 11:42 AM
>>> To: Cluster Labs - All topics related to open-source clustering welcomed
>> <users@clusterlabs.org>
>>> Subject: Re: [ClusterLabs] Mysql slave did not start replication after 
>>> failure,
>> and read-only IP also remained active on the much outdated slave
>>>
>>> On Mon, Aug 22, 2016 at 12:18 PM, Attila Megyeri
>>> <amegy...@minerva-soft.com> wrote:
>>>> Dear community,
>>>>
>>>>
>>>>
>>>> A few days ago we had an issue in our Mysql M/S replication cluster.
>>>>
>>>> We have a one R/W Master, and a one RO Slave setup. RO VIP is
>> supposed to be
>>>> running on the slave if it is not too much behind the master, and if any
>>>> error occurs, RO VIP is moved to the master.
>>>>
>>>>
>>>>
>>>> Something happened with the slave Mysql (some disk issue, still
>>>> investigating), but the proble

Re: [ClusterLabs] ocf::heartbeat:IPaddr

[Ken Gaillot]

On 08/25/2016 10:51 AM, Gabriele Bulfon wrote:
> Hi,
> 
> I'm advancing with this monster cluster on XStreamOS/illumos ;)
> 
> In the previous older tests I used heartbeat, and I had these lines to
> take care of the swapping public IP addresses:
> 
> primitive xstorage1_wan1_IP ocf:heartbeat:IPaddr params ip="1.2.3.4"
> cidr_netmask="255.255.255.0" nic="e1000g1"
> primitive xstorage2_wan2_IP ocf:heartbeat:IPaddr params ip="1.2.3.5"
> cidr_netmask="255.255.255.0" nic="e1000g1"
> 
> location xstorage1_wan1_IP_pref xstorage1_wan1_IP 100: xstorage1
> location xstorage2_wan2_IP_pref xstorage2_wan2_IP 100: xstorage2
> 
> They get configured, but then I get this in crm status:
> 
> xstorage1_wan1_IP (ocf::heartbeat:IPaddr): Stopped
> xstorage2_wan2_IP (ocf::heartbeat:IPaddr): Stopped
> 
> Failed Actions:
> * xstorage1_wan1_IP_start_0 on xstorage1 'not installed' (5): call=20,
> status=complete, exitreason='Setup problem: couldn't find command:
> /usr/bin/gawk',
> last-rc-change='Thu Aug 25 17:50:32 2016', queued=1ms, exec=158ms
> * xstorage2_wan2_IP_start_0 on xstorage1 'not installed' (5): call=22,
> status=complete, exitreason='Setup problem: couldn't find command:
> /usr/bin/gawk',
> last-rc-change='Thu Aug 25 17:50:33 2016', queued=1ms, exec=29ms
> * xstorage1_wan1_IP_start_0 on xstorage2 'not installed' (5): call=22,
> status=complete, exitreason='Setup problem: couldn't find command:
> /usr/bin/gawk',
> last-rc-change='Thu Aug 25 17:50:30 2016', queued=1ms, exec=36ms
> * xstorage2_wan2_IP_start_0 on xstorage2 'not installed' (5): call=20,
> status=complete, exitreason='Setup problem: couldn't find command:
> /usr/bin/gawk',
> last-rc-change='Thu Aug 25 17:50:29 2016', queued=0ms, exec=150ms
> 
> 
> The crm configure process already checked of the presence of the
> required IPaddr shell, and it was ok.
> Now looks like it's looking for "/usr/bin/gawk", and that is actually there!
> Is there any known incompatibility with the mixed heartbeat ocf ? Should
> I use corosync specific ocf files or something else?

"heartbeat" in this case is just an OCF provider name, and has nothing
to do with the heartbeat messaging layer, other than having its origin
in the same project. There actually has been a recent proposal to rename
the provider to "clusterlabs" to better reflect the current reality.

The "couldn't find command" message comes from the ocf-binaries shell
functions. If you look at have_binary() there, it uses sed and which,
and I'm guessing that fails on your OS somehow. You may need to patch it.

> Thanks again!
> 
> Gabriele
> 
> 
> *Sonicle S.r.l. *: http://www.sonicle.com 
> *Music: *http://www.gabrielebulfon.com 
> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] FYI: pacemaker will keep compatibility with python 2.6

[Ken Gaillot]

Hello everybody, just an FYI:

Pacemaker's test suites are mostly written in python. A few months ago,
I raised the possibility of dropping support for python 2.6, to make it
easier to support python 3.

After some experience with updating our code, I found that we can
support python 2.6+ and 3.2+ with the same code base.

This means that the latest pacemaker will remain fully buildable on RHEL
6, SLES11, and other platforms that ship with python 2.6.

I've already updated the fencing and lrmd regression tests, so CTS is
all that's left. I don't expect CTS to be python 3 compatible by the
next release, but we're making progress.

As part of this work, I've developed our first official python coding
guidelines. They are included in the "Pacemaker Development" source in
the master branch. The online version hasn't been updated yet, but will
be before the next release.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] pcs cluster auth returns authentication error

[Ken Gaillot]

On 08/25/2016 03:04 PM, Jason A Ramsey wrote:
> Please help. Just getting this thing stood up on a new set of servers
> and getting stymied right out the gate:
> 
>  
> 
> # pcs cluster auth node1 node2
> 
> Username: hacluster
> 
> Password:
> 
>  
> 
> I am **certain** that the password I’m providing is correct. Even still
> I get:
> 
>  
> 
> Error: node1: Username and/or password is incorrect
> 
> Error: node2: Username and/or password is incorrect
> 
>  
> 
> I also see this is /var/log/audit/audit.log:
> 
>  
> 
> type=USER_AUTH msg=audit(1472154922.415:69): user pid=1138 uid=0
> auid=4294967295 ses=4294967295 subj=system_u:system_r:initrc_t:s0
> msg='op=PAM:authentication acct="hacluster" exe="/usr/bin/ruby"
> hostname=? addr=? terminal=? res=failed'

That's an SELinux error. To confirm, try again with SELinux disabled.

I think distributions that package pcs also provide any SELinux policies
it needs. I'm not sure what those are, or the best way to specify them
if you're building pcs yourself, but it shouldn't be difficult to figure
out.

> I’ve gone so far as to change the password to ensure that it didn’t have
> any “weird” characters in it, but the error persists. Appreciate the help!
> 
>  
> 
> --
> 
>  
> 
> *[ jR ]*
> 
>   @: ja...@eramsey.org 
> 
>  
> 
>   /there is no path to greatness; greatness is the path/

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Howto restart resource

[Ken Gaillot]

On 08/29/2016 01:38 AM, Stefano Ruberti wrote:
> Dear all,
> 
> I have following situation and I need an advice from you:
> 
> in my Active/Passive Cluster  (Ubuntu_16.04  corosync + pacemaker , no pcs)
> 
> Node_ANode_B
> Resource1Resource1
> Resource2Resource2
> Resource3Resource3
> rsyslogd   rsyslogd   
> 
> 1. is possible to configure rsyslogd to restarting when node (Node_A or 
> Node_B) go on-line and take resource? 
> 
> 
> Best Regards
> Stefano

Is rsyslogd a cluster resource? That might not be ideal, since pacemaker
itself needs to use syslog (unless you've configured it not to), and you
probably want syslog running even if the cluster services are stopped
(for an upgrade, for example).

If your goal is simply to reload rsyslogd when a resource state changes,
I would think one of these approaches would be suitable:

Use the new alerts interface (pacemaker 1.1.15+) with an alert script
that reloads rsyslogd at the desired events;

Or, write a custom resource agent that reloads rsyslogd, and set up
colocation/ordering constraints between it and the desired resources.
Pacemaker will then "start" it after starting the resources, and "stop"
it before stopping the resources. Note that the agent must know whether
it is "running", and there is a ha_pseudo_resource() function in the OCF
libraries to handle that.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] data loss of network would cause Pacemaker exit abnormally

[Ken Gaillot]

On 08/27/2016 09:15 PM, chenhj wrote:
> Hi all,
> 
> When i use the following command to simulate data lost of network at one
> member of my 3 nodes Pacemaker+Corosync cluster,
> sometimes it cause Pacemaker on another node exit.
> 
>   tc qdisc add dev eth2 root netem loss 90%
> 
> Is there any method to avoid this proleam?
> 
> [root@node3 ~]# ps -ef|grep pacemaker
> root  32540  1  0 00:57 ?00:00:00
> /usr/libexec/pacemaker/lrmd
> 189   32542  1  0 00:57 ?00:00:00
> /usr/libexec/pacemaker/pengine
> root  33491  11491  0 00:58 pts/100:00:00 grep pacemaker
> 
> /var/log/cluster/corosync.log 
> 
> Aug 27 12:33:59 [46855] node3cib: info: cib_process_request:
>Completed cib_modify operation for section status: OK (rc=0,
> origin=local/attrd/230, version=10.657.19)
> Aug 27 12:33:59 corosync [CPG   ] chosen downlist: sender r(0)
> ip(192.168.125.129) ; members(old:2 left:1)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: pcmk_cpg_membership:
>Node 2172496064 joined group pacemakerd (counter=12.0)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: pcmk_cpg_membership:
>Node 2172496064 still member of group pacemakerd (peer=node2,
> counter=12.0)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info:
> crm_update_peer_proc:   pcmk_cpg_membership: Node node2[2172496064]
> - corosync-cpg is now online
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: pcmk_cpg_membership:
>Node 2273159360 still member of group pacemakerd (peer=node3,
> counter=12.1)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: crm_cs_flush:  
> Sent 0 CPG messages  (1 remaining, last=19): Try again (6)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: pcmk_cpg_membership:
>Node 2273159360 left group pacemakerd (peer=node3, counter=13.0)
> Aug 27 12:33:59 [46849] node3 pacemakerd: info:
> crm_update_peer_proc:   pcmk_cpg_membership: Node node3[2273159360]
> - corosync-cpg is now offline
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: pcmk_cpg_membership:
>Node 2172496064 still member of group pacemakerd (peer=node2,
> counter=13.0)
> Aug 27 12:33:59 [46849] node3 pacemakerd:error: pcmk_cpg_membership:
>We're not part of CPG group 'pacemakerd' anymore!
> Aug 27 12:33:59 [46849] node3 pacemakerd:error: pcmk_cpg_dispatch:
>  Evicted from CPG membership

>From the above, I suspect that the node with the network loss was the
DC, and from its point of view, it was the other node that went away.

Proper quorum and fencing configuration should prevent this from being
an issue. Once the one node sees heavy network loss, the other node(s)
should fence it before it causes too many problems.

> Aug 27 12:33:59 [46849] node3 pacemakerd:error: mcp_cpg_destroy:  
>  Connection destroyed
> Aug 27 12:33:59 [46849] node3 pacemakerd: info: crm_xml_cleanup:  
>  Cleaning up memory from libxml2
> Aug 27 12:33:59 [46858] node3  attrd:error: crm_ipc_read:  
> Connection to pacemakerd failed
> Aug 27 12:33:59 [46858] node3  attrd:error:
> mainloop_gio_callback:  Connection to pacemakerd[0x1255eb0] closed
> (I/O condition=17)
> Aug 27 12:33:59 [46858] node3  attrd: crit: attrd_cs_destroy:  
> Lost connection to Corosync service!
> Aug 27 12:33:59 [46858] node3  attrd:   notice: main:   Exiting...
> Aug 27 12:33:59 [46858] node3  attrd:   notice: main:  
> Disconnecting client 0x12579a0, pid=46860...
> Aug 27 12:33:59 [46858] node3  attrd:error:
> attrd_cib_connection_destroy:   Connection to the CIB terminated...
> Aug 27 12:33:59 corosync [pcmk  ] info: pcmk_ipc_exit: Client attrd
> (conn=0x1955f80, async-conn=0x1955f80) left
> Aug 27 12:33:59 [46856] node3 stonith-ng:error: crm_ipc_read:  
> Connection to pacemakerd failed
> Aug 27 12:33:59 [46856] node3 stonith-ng:error:
> mainloop_gio_callback:  Connection to pacemakerd[0x2314af0] closed
> (I/O condition=17)
> Aug 27 12:33:59 [46856] node3 stonith-ng:error:
> stonith_peer_cs_destroy:Corosync connection terminated
> Aug 27 12:33:59 [46856] node3 stonith-ng: info: stonith_shutdown:  
> Terminating with  1 clients
> Aug 27 12:33:59 [46856] node3 stonith-ng: info:
> cib_connection_destroy: Connection to the CIB closed.
> ...
> 
> please see corosynclog.txt for detail of log 
> 
> 
> [root@node3 ~]# cat /etc/corosync/corosync.conf
> totem {
>version: 2
>secauth: off
>interface {
>member {
>memberaddr: 192.168.125.134
>}
>member {
>memberaddr: 192.168.125.129
>}
>member {
>memberaddr: 192.168.125.135
>}
> 
>ringnumber: 0
>bindnetaddr: 192.168.125.135
>mcastport: 5405
>

Re: [ClusterLabs] ocf scripts shell and local variables

[Ken Gaillot]

On 08/29/2016 04:17 AM, Gabriele Bulfon wrote:
> Hi Ken,
> 
> I have been talking with the illumos guys about the shell problem.
> They all agreed that ksh (and specially the ksh93 used in illumos) is
> absolutely Bourne-compatible, and that the "local" variables used in the
> ocf shells is not a Bourne syntax, but probably a bash specific.
> This means that pointing the scripts to "#!/bin/sh" is portable as long
> as the scripts are really Bourne-shell only syntax, as any Unix variant
> may link whatever Bourne-shell they like.
> In this case, it should point to "#!/bin/bash" or whatever shell the
> script was written for.
> Also, in this case, the starting point is not the ocf-* script, but the
> original RA (IPaddr, but almost all of them).
> 
> What about making the code base of RA and ocf-* portable?
> It may be just by changing them to point to bash, or with some kind of
> configure modifier to be able to specify the shell to use.
> 
> Meanwhile, changing the scripts by hands into #!/bin/bash worked like a
> charm, and I will start patching.
> 
> Gabriele

Interesting, I thought local was posix, but it's not. It seems everyone
but solaris implemented it:

http://stackoverflow.com/questions/18597697/posix-compliant-way-to-scope-variables-to-a-function-in-a-shell-script

Please open an issue at:

https://github.com/ClusterLabs/resource-agents/issues

The simplest solution would be to require #!/bin/bash for all RAs that
use local, but I'm not sure that's fair to the distros that support
local in a non-bash default shell. Another possibility would be to
modify all RAs to avoid local entirely, by using unique variable
prefixes per function. Or, it may be possible to guard every instance of
local with a check for ksh, which would use typeset instead. Raising the
issue will allow some discussion of the possibilities.

> 
> 
> *Sonicle S.r.l. *: http://www.sonicle.com <http://www.sonicle.com/>
> *Music: *http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
> 
> 
> 
> --
> 
> Da: Ken Gaillot <kgail...@redhat.com>
> A: gbul...@sonicle.com Cluster Labs - All topics related to open-source
> clustering welcomed <users@clusterlabs.org>
> Data: 26 agosto 2016 15.56.02 CEST
> Oggetto: Re: ocf scripts shell and local variables
> 
> On 08/26/2016 08:11 AM, Gabriele Bulfon wrote:
> > I tried adding some debug in ocf-shellfuncs, showing env and ps
> -ef into
> > the corosync.log
> > I suspect it's always using ksh, because in the env output I
> produced I
> > find this: KSH_VERSION=.sh.version
> > This is normally not present in the environment, unless ksh is running
> > the shell.
> 
> The RAs typically start with #!/bin/sh, so whatever that points to on
> your system is what will be used.
> 
> > I also tried modifiying all ocf shells with "#!/usr/bin/bash" at the
> > beginning, no way, same output.
> 
> You'd have to change the RA that includes them.
> 
> > Any idea how can I change the used shell to support "local" variables?
> 
> You can either edit the #!/bin/sh line at the top of each RA, or figure
> out how to point /bin/sh to a Bourne-compatible shell. ksh isn't
> Bourne-compatible, so I'd expect lots of #!/bin/sh scripts to fail with
> it as the default shell.
> 
> > Gabriele
> >
> >
> 
> 
> > *Sonicle S.r.l. *: http://www.sonicle.com <http://www.sonicle.com/>
> > *Music: *http://www.gabrielebulfon.com
> <http://www.gabrielebulfon.com/>
> > *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon
> >
> >
> 
> >
> >
> > *Da:* Gabriele Bulfon <gbul...@sonicle.com>
> > *A:* kgail...@redhat.com Cluster Labs - All topics related to
> > open-source clustering welcomed <users@clusterlabs.org>
> > *Data:* 26 agosto 2016 10.12.13 CEST
> > *Oggetto:* Re: [ClusterLabs] ocf::heartbeat:IPaddr
> >
> >
> > I looked around what you suggested, inside ocf-binaris and
> > ocf-shellfuncs etc.
> > So I found also these logs in corosync.log :
> >
> > Aug 25 17:50:33 [2250] crmd: notice: process_lrm_eve

Re: [ClusterLabs] Failed to retrieve meta-data for custom ocf resource

[Ken Gaillot]

On 09/28/2016 04:04 PM, Christopher Harvey wrote:
> My corosync/pacemaker logs are seeing a bunch of messages like the
> following:
> 
> Sep 22 14:50:36 [1346] node-132-60   crmd: info:
> action_synced_wait: Managed MsgBB-Active_meta-data_0 process 15613
> exited with rc=4

This is the (unmodified) exit status of the process, so the resource
agent must be returning "4" for some reason. Normally, that is used to
indicate "insufficient privileges".

> Sep 22 14:50:36 [1346] node-132-60   crmd:error:
> generic_get_metadata:   Failed to retrieve meta-data for
> ocf:acme:MsgBB-Active
> Sep 22 14:50:36 [1346] node-132-60   crmd:  warning:
> get_rsc_metadata:   No metadata found for MsgBB-Active::ocf:acme:
> Input/output error (-5)
> Sep 22 14:50:36 [1346] node-132-60   crmd:error:
> build_operation_update: No metadata for acme::ocf:MsgBB-Active
> Sep 22 14:50:36 [1346] node-132-60   crmd:   notice:
> process_lrm_event:  Operation MsgBB-Active_start_0: ok
> (node=node-132-60, call=25, rc=0, cib-update=27, confirmed=true)
> 
> I am able to run the meta-data command on the command line:

I would suspect that your user account has some privileges that the lrmd
user (typically hacluster:haclient) doesn't have. Try "su - hacluster"
first and see if it's any different. Maybe directory or file
permissions, or SELinux?

> node-132-43 # /lib/ocf/resource.d/acme/MsgBB-Active meta-data
> 
> 
> 
> 1.0
> 
> 
> MsgBB-Active resource (long desc)
> 
> MsgBB-Active resource
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> node-132-43 # echo $?
> 0
> 
> Resource code here:
> #! /bin/bash
> 
> ###
> # Initialization:
> 
> : ${OCF_FUNCTIONS=${OCF_ROOT}/resource.d/heartbeat/.ocf-shellfuncs}
> . ${OCF_FUNCTIONS}
> : ${__OCF_ACTION=$1}
> 
> ###
> 
> meta_data()
> {
> cat < 
> 
> 
> 1.0
> 
> 
> MsgBB-Active resource (long desc)
> 
> MsgBB-Active resource
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> END
> }
> 
> # don't exit on TERM, to test that lrmd makes sure that we do exit
> trap sigterm_handler TERM
> sigterm_handler() {
> ocf_log info "They use TERM to bring us down. No such luck."
> return
> }
> 
> msgbb_usage() {
> cat < usage: $0 {start|stop|monitor|validate-all|meta-data}
> 
> Expects to have a fully populated OCF RA-compliant environment set.
> END
> }
> 
> msgbb_monitor() {
> # trimmed.
> }
> 
> msgbb_stop() {
> # trimmed.
> }
> 
> msgbb_start() {
> # trimmed.
> }
> 
> msgbb_validate() {
> # trimmed.
> }
> 
> case $__OCF_ACTION in
> meta-data)  meta_data
> exit $OCF_SUCCESS
> ;;
> start)  msgbb_start;;
> stop)   msgbb_stop;;
> monitor)msgbb_monitor;;
> reload) ocf_log err "Reloading..."
> msgbb_start
> ;;
> validate-all)   msgbb_validate;;
> usage|help) msgbb_usage
> exit $OCF_SUCCESS
> ;;
> *)  msgbb_usage
> exit $OCF_ERR_UNIMPLEMENTED
> ;;
> esac
> rc=$?
> ocf_log debug "${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc"
> exit $rc
> 
> 
> Thanks,
> Chris

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] hi list

[Ken Gaillot]

On 09/30/2016 10:10 AM, Антон Сацкий wrote:
> so U mean that in fact IPaddr is  IPaddr2 version
>  Is there  a way to see that ip address was added  except to see logs?
>  Before i can see ip using ifconfig
> Regards and thanks for reply.

"ip address show" (which can be abbreviated "ip a")

See https://tty1.net/blog/2010/ifconfig-ip-comparison_en.html

>  
> 
> 2016-09-30 16:14 GMT+03:00 Klaus Wenninger  >:
> 
> On 09/30/2016 02:39 PM, Антон Сацкий wrote:
> > Greeting for everyone
> > Got a question is it normal?
> >
> > below i can see
> >
> > Running /etc/ha.d/resource.d/IPaddr 10.0.10.113/24/eth1/10.0.10.255 
> 
> >  > start
> > IPaddr(IPaddr_10.0.10.113)[2085]:2016/09/30_15:13:39 INFO: Adding
> > inet address 10.0.10.113/24 
>  with broadcast
> > address 10.0.10.255 to device eth1
> > IPaddr(IPaddr_10.0.10.113)[2085]:2016/09/30_15:13:39 INFO:
> > Bringing device eth1 up
> >
> > but when i run ifconfig
> > i cant see 10.0.10.113 as eth1:0
> 
> That is fine IPaddr uses subinterfaces.
> As this is not the state of the art way to setup
> multiple IP addresses on one physical interface
> anymore IPaddr2 doesn't use subinterfaces
> anymore.
> 
> Regards,
> Klaus
> 
> 
> >
> >
> > BUT IT EXIST
> > i can ping 10.0.10.113  from pouter  and it is this  server
> >
> > PS
> > heartbeat.x86_64
> > 3.0.4-2.el6
> > @epe
> >
> > Centos  6.8
> >
> >
> >
> > Sep 30 15:02:24 voipserver.drbd heartbeat: [1568]: WARN:
> > string2msg_ll: node [asterisk.retail.ua
>  ]
> > failed authentication
> > Sep 30 15:02:28 voipserver.drbd heartbeat: [1568]: info: Heartbeat
> > shutdown in progress. (1568)
> > Sep 30 15:02:28 voipserver.drbd heartbeat: [2768]: info: Giving up all
> > HA resources.
> > ResourceManager(default)[2781]:2016/09/30_15:02:28 info: Releasing
> > resource group: voipserver.drbd drbddisk::r0
> > Filesystem::/dev/drbd0::/replica::ext4
> > IPaddr::10.0.10.113/24/eth1/10.0.10.255 
> 
> >  > mysqld asterisk httpd
> > ResourceManager(default)[2781]:2016/09/30_15:02:28 info: Running
> > /etc/init.d/httpd  stop
> > ResourceManager(default)[2781]:2016/09/30_15:02:28 info: Running
> > /etc/init.d/asterisk  stop
> > ResourceManager(default)[2781]:2016/09/30_15:02:28 info: Running
> > /etc/init.d/mysqld  stop
> > Sep 30 15:02:29 voipserver.drbd heartbeat: [1568]: WARN:
> > string2msg_ll: node [asterisk.retail.ua
>  ]
> > failed authentication
> > ResourceManager(default)[2781]:2016/09/30_15:02:30 info: Running
> > /etc/ha.d/resource.d/IPaddr 10.0.10.113/24/eth1/10.0.10.255 
> 
> >  > stop
> > IPaddr(IPaddr_10.0.10.113)[2960]:2016/09/30_15:02:30 INFO: IP
> > status = ok, IP_CIP=
> > /usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.10.113)[2934]:
> > 2016/09/30_15:02:30 INFO:  Success
> > ResourceManager(default)[2781]:2016/09/30_15:02:30 info: Running
> > /etc/ha.d/resource.d/Filesystem /dev/drbd0 /replica ext4 stop
> > Filesystem(Filesystem_/dev/drbd0)[3050]:2016/09/30_15:02:31 INFO:
> > Running stop for /dev/drbd0 on /replica
> > Filesystem(Filesystem_/dev/drbd0)[3050]:2016/09/30_15:02:31 INFO:
> > Trying to unmount /replica
> > Filesystem(Filesystem_/dev/drbd0)[3050]:2016/09/30_15:02:31 INFO:
> > unmounted /replica successfully
> >
> 
> /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[3042]:
> > 2016/09/30_15:02:31 INFO:  Success
> > ResourceManager(default)[2781]:2016/09/30_15:02:31 info: Running
> > /etc/ha.d/resource.d/drbddisk r0 stop
> > Sep 30 15:02:31 voipserver.drbd heartbeat: [2768]: info: All HA
> > resources relinquished.
> > Sep 30 15:02:31 voipserver.drbd heartbeat: [1568]: WARN: 1 lost
> > packet(s) for [voipbackup.drbd] [358:360]
> > Sep 30 15:02:31 voipserver.drbd heartbeat: [1568]: info: No pkts
> > missing from voipbackup.drbd!
> > Sep 30 15:02:33 voipserver.drbd heartbeat: [1568]: info: killing
> > HBFIFO process 1578 with signal 15
> > Sep 30 15:02:33 voipserver.drbd heartbeat: [1568]: info: killing
> > HBWRITE process 1580 with signal 15
> > Sep 30 15:02:33 voipserver.drbd heartbeat: [1568]: 

Re: [ClusterLabs] RFC: allowing soft recovery attempts before ignore/block/etc.

[Ken Gaillot]

On 09/28/2016 10:54 PM, Andrew Beekhof wrote:
> On Sat, Sep 24, 2016 at 9:12 AM, Ken Gaillot <kgail...@redhat.com> wrote:
>>> "Ignore" is theoretically possible to escalate, e.g. "ignore 3 failures
>>> then migrate", but I can't think of a real-world situation where that
>>> makes sense,
>>>
>>>
>>> really?
>>>
>>> it is not uncommon to hear "i know its failed, but i dont want the
>>> cluster to do anything until its _really_ failed"
>>
>> Hmm, I guess that would be similar to how monitoring systems such as
>> nagios can be configured to send an alert only if N checks in a row
>> fail. That's useful where transient outages (e.g. a webserver hitting
>> its request limit) are acceptable for a short time.
>>
>> I'm not sure that's translatable to Pacemaker. Pacemaker's error count
>> is not "in a row" but "since the count was last cleared".
> 
> It would be a major change, but perhaps it should be "in-a-row" and
> successfully performing the action clears the count.
> Its entirely possible that the current behaviour is like that because
> I wasn't smart enough to implement anything else at the time :-)

Or you were smart enough to realize what a can of worms it is. :) Take a
look at all of nagios' options for deciding when a failure becomes "real".

If you clear failures after a success, you can't detect/recover a
resource that is flapping.

>> "Ignore up to three monitor failures if they occur in a row [or, within
>> 10 minutes?], then try soft recovery for the next two monitor failures,
>> then ban this node for the next monitor failure." Not sure being able to
>> say that is worth the complexity.
> 
> Not disagreeing

It only makes sense to escalate from ignore -> restart -> hard, so maybe
something like:

  op monitor ignore-fail=3 soft-fail=2 on-hard-fail=ban


To express current default behavior:

  op start ignore-fail=0 soft-fail=0on-hard-fail=ban
  op stop  ignore-fail=0 soft-fail=0on-hard-fail=fence
  op * ignore-fail=0 soft-fail=INFINITY on-hard-fail=ban


on-fail, migration-threshold, and start-failure-is-fatal would be
deprecated (and would be easy to map to the new parameters).

I'd avoid the hassles of counting failures "in a row", and stick with
counting failures since the last cleanup.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker dependency fails when upgrading OS (Amazon Linux)

[Ken Gaillot]

On 09/29/2016 05:30 PM, neeraj ch wrote:
> Hello, 
> 
> I have pacemaker cluster running on Amazon Linux 2013.03 , details as
> follows.
> 
> OS : Amazon Linux 2013.03  64 bit (based off on el6)
> Pacemaker version : 1.1.12 downloaded
> form 
> http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/
> I downloaded all of pacemakers dependencies from the same repository as
> well.
> corosync version : 1.4.1
> 
> I worked well until I tried to update using yum update. It looks like
> yum wants to update perl. As soon as I run the update, a dependency
> error is popping showing the pacemaker dependency perl(
> module_compat_5.10.0) will change. 

I believe the only perl used by pacemaker is in the
fence_legacy/fence_pcmk fence agents, and the crm_report tool. If you
don't use any of those, you should be fine with ignoring the dependency.

Since Amazon Linux is derived from RHEL, you'd probably have more luck
with the corosync/pacemaker/etc. packages from the comparable CentOS
version instead. There are some library version differences between
RHEL/CentOS and AL though, so no guarantees.

> did any one else have a similar problem ?  Any help would be much
> appreciated.  
> 
> Thank you 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Colocation and ordering with live migration

[Ken Gaillot]

On 10/10/2016 10:21 AM, Klaus Wenninger wrote:
> On 10/10/2016 04:54 PM, Ken Gaillot wrote:
>> On 10/10/2016 07:36 AM, Pavel Levshin wrote:
>>> 10.10.2016 15:11, Klaus Wenninger:
>>>> On 10/10/2016 02:00 PM, Pavel Levshin wrote:
>>>>> 10.10.2016 14:32, Klaus Wenninger:
>>>>>> Why are the order-constraints between libvirt & vms optional?
>>>>> If they were mandatory, then all the virtual machines would be
>>>>> restarted when libvirtd restarts. This is not desired nor needed. When
>>>>> this happens, the node is fenced because it is unable to restart VM in
>>>>> absence of working libvirtd.
>>>> Was guessing something like that ...
>>>> So let me reformulate my question:
>>>>Why does libvirtd have to be restarted?
>>>> If it is because of config-changes making it reloadable might be a
>>>> solution ...
>>>>
>>> Right, config changes come to my mind first of all. But sometimes a
>>> service, including libvirtd, may fail unexpectedly. In this case I would
>>> prefer to restart it without disturbing VirtualDomains, which will fail
>>> eternally.
>> I think the mandatory colocation of VMs with libvirtd negates your goal.
>> If libvirtd stops, the VMs will have to stop anyway because they can't
>> be colocated with libvirtd. Making the colocation optional should fix that.
>>
>>> The question is, why the cluster does not obey optional constraint, when
>>> both libvirtd and VM stop in a single transition?
>> If it truly is in the same transition, then it should be honored.
>>
>> You have *mandatory* constraints for DLM -> CLVMd -> cluster-config ->
>> libvirtd, but only an *optional* constraint for libvirtd -> VMs.
>> Therefore, libvirtd will generally have to wait longer than the VMs to
>> be started.
>>
>> It might help to add mandatory constraints for cluster-config -> VMs.
>> That way, they have the same requirements as libvirtd, and are more
>> likely to start in the same transition.
>>
>> However I'm sure there are still problematic situations. What you want
>> is a simple idea, but a rather complex specification: "If rsc1 fails,
>> block any instances of this other RA on the same node."
>>
>> It might be possible to come up with some node attribute magic to
>> enforce this. You'd need some custom RAs. I imagine something like one
>> RA that sets a node attribute, and another RA that checks it.
>>
>> The setter would be grouped with libvirtd. Anytime that libvirtd starts,
>> the setter would set a node attribute on the local node. Anytime that
>> libvirtd stopped or failed, the setter would unset the attribute value.
>>
>> The checker would simply monitor the attribute, and fail if the
>> attribute is unset. The group would have on-fail=block. So anytime the
>> the attribute was unset, the VM would not be started or stopped. (There
>> would be no constraints between the two groups -- the checker RA would
>> take the place of constraints.)
> 
> In how far would that behave differently to just putting libvirtd
> into this on-fail=block group? (apart from of course the
> possibility to group the vms into more than one group ...)

You could stop or restart libvirtd without stopping the VMs. It would
cause a "failure" of the checker that would need to be cleaned later,
but the VMs wouldn't stop.

>>
>> I haven't thought through all possible scenarios, but it seems feasible
>> to me.
>>
>>> In my eyes, these services are bound by a HARD obvious colocation
>>> constraint: VirtualDomain should never ever be touched in absence of
>>> working libvirtd. Unfortunately, I cannot figure out a way to reflect
>>> this constraint in the cluster.
>>>
>>>
>>> -- 
>>> Pavel Levshin

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/07/2016 12:03 PM, Jehan-Guillaume de Rorthais wrote:
> On Mon, 7 Nov 2016 09:31:20 -0600
> Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> On 11/07/2016 03:47 AM, Klaus Wenninger wrote:
>>> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote:  
>>>> On Mon, 7 Nov 2016 10:12:04 +0100
>>>> Klaus Wenninger <kwenn...@redhat.com> wrote:
>>>>  
>>>>> On 11/07/2016 08:41 AM, Ulrich Windl wrote:  
>>>>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 04.11.2016 um 22:37 in
>>>>>>>>> Nachricht
>>>>>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:
>>>>>>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:
>>>>>>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 03.11.2016 um 17:08
>>>>>>>>>>> in
>>>>>>>> Nachricht
>>>>>>>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:
>>>> ...  
>>>>>>> Another possible use would be for a cron that needs to know whether a
>>>>>>> particular resource is running, and an attribute query is quicker and
>>>>>>> easier than something like parsing crm_mon output or probing the
>>>>>>> service.
>>>>>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so
>>>>>> besides of lacking options and inefficient implementation, why should one
>>>>>> be faster than the other?
>>>>> attrd_updater doesn't go for the CIB  
>>>> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private"
>>>> since 1.1.13:
>>>> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177  
>>> That prevents values being stored in the CIB. attrd_updater should
>>> always talk to attrd as I got it ...  
>>
>> It's a bit confusing: Both crm_attribute and attrd_updater will
>> ultimately affect both attrd and the CIB in most cases, but *how* they
>> do so is different. crm_attribute modifies the CIB, and lets attrd pick
>> up the change from there; attrd_updater notifies attrd, and lets attrd
>> modify the CIB.
>>
>> The difference is subtle.
>>
>> With corosync 2, attrd only modifies "transient" node attributes (which
>> stay in effect till the next reboot), not "permanent" attributes.
> 
> So why "--private" is not compatible with corosync 1.x as attrd_updater only 
> set
> "transient" attributes anyway?

Corosync 1 does not support certain reliability guarantees required by
the current attrd, so when building against the corosync 1 libraries,
pacemaker will install "legacy" attrd instead. The difference is mainly
that the current attrd can guarantee atomic updates to attribute values.
attrd_updater actually can set permanent attributes when used with
legacy attrd.

> How and where private attributes are stored?

They are kept in memory only, in attrd. Of course, attrd is clustered,
so they are kept in sync across all nodes.

>> So crm_attribute must be used if you want to set a permanent attribute.
>> crm_attribute also has the ability to modify cluster properties and
>> resource defaults, as well as node attributes.
>>
>> On the other hand, by contacting attrd directly, attrd_updater can
>> change an attribute's "dampening" (how often it is flushed to the CIB),
>> and it can (as mentioned above) set "private" attributes that are never
>> written to the CIB (and thus never cause the cluster to re-calculate
>> resource placement).
> 
> Interesting, thank you for the clarification.
> 
> As I understand it, it resumes to:
> 
>   crm_attribute -> CIB <-(poll/notify?) attrd
>   attrd_updater -> attrd -> CIB

Correct. On startup, attrd registers with CIB to be notified of all changes.

> Just a quick question about this, is it possible to set a "dampening" high
> enough so attrd never flush it to the CIB (kind of private attribute too)?

I'd expect that to work, if the dampening interval was higher than the
lifetime of the cluster being up.

It's also possible to abuse attrd to create a kind of private attribute
by using a node name that doesn't exist and never will. :) This ability
is intentionally allowed, so you can set attributes for nodes that the
current partition isn't aware of, or nodes that are planned to be added
later, but only attributes for known nodes will be written to the CIB.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Preventing switchover in case of failing ping node

[Ken Gaillot]

On 11/03/2016 08:49 AM, Detlef Gossrau wrote:
> Hi all,
> 
> is it possible to prevent a switchover in a active/passive cluster if a
> ping node completely fails ?
> 
> Situation:
> 
> A ping node is put into maintenance and not reachable for a certain
> time. The cluster nodes getting the information about the failing ping
> node at different times. For example, node 1 - which is the passive node
> - is getting the information earlier than node 2. As a result the
> resources are moved to node 2. But in the case the ping node is not
> available at all, no switchover should be executed. Is it possible to
> prevent this ?

There's no way for one node to predict that all other nodes will also be
unable to see the target. The only workarounds I see are:

* Use a second ping target on a different path, if available

* If the maintenance is planned, you could disable the relevant
constraints during the window

> 
> Thanks for any help !
> 
> Best regards,
> Detlef


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Live migration not working on shutdown

[Ken Gaillot]

On 11/04/2016 05:51 AM, IT Nerb GmbH wrote:
> Zitat von Klaus Wenninger <kwenn...@redhat.com>:
> 
>> On 11/02/2016 06:32 PM, Ken Gaillot wrote:
>>> On 10/26/2016 06:12 AM, Rainer Nerb wrote:
>>>> Hello all,
>>>>
>>>> we're currently testing a 2-node-cluster with 2 vms and live migration
>>>> on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
>>>> migration via ssh-method.
>>>>
>>>> Live migration works, if we issue "pcs resource move ...", "pcs cluster
>>>> standby", "pcs cluster stop" and even "systemctl rescue".
>>>> The latter only worked, after adding the following additional
>>>> dependencies to pacemaker.service and leaving the management of those
>>>> services to systemd:
>>>>
>>>>   * After/Requires=systemd-machined.service
>>>>   * After/Requires=systemd-machine-id-commit.service
>>>>   * After/Requires=remote-fs.target
>>>>   * After/Requires=libvirtd.service
>>>>   * After/Requires=iscsi.service
>>>>   * After/Requires=iscsid.service
>>>>   * After/Requires=sshd.service
>>> This makes sense when clustered resources depend on services that aren't
>>> themselves managed by the cluster. It's dependent on your situation, so
>>> it's not something that pacemaker can solve generically.
> First approach was to use systemd-resources as there are no ocf:
> resource-agents for iSCSI-Initiators or libvirtd in our distribution.
> But then migration failed even on "systemctl rescue".
>>>
>>> You may already be aware, but the easiest way to add such requirements
>>> is to put them in a systemd unit override, e.g.
>>> /etc/systemd/system/pacemaker.service.d/dependencies.conf.
> Yes, that's how we implemented the additional dependencies.
>>>
>>>> When shutting down or rebooting migration fails and not even the
>>>> regular
>>>> shutdown of the vms succeeds. Systemd seems to tear down the vms by
>>>> terminating something they depend on.
>>>>
>>>> Is this a known issue? Did we miss any further dependencies?
>>> There was a shutdown issue when using systemd-class cluster resources
>>> (systemd: instead of ocf:), but I believe that was fixed in the package
>>> you're using, and it's probably not relevant here anyway.
>> Speaking of
>> https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?
>>
>>
>> It shouldn't be in Centos 7.2 but I agree unless there are no
>> systemd-resources involved it wouldn't matter.
>>
>>>
>>> It does sound like there's another dependency, but I don't know what.
>>>
>>> What log messages do you see on the failure?
> See attached log files.

The line that stands out to me is:

Nov  4 11:11:39 kvm02 systemd: Stopping Virtual Machine qemu-2-samba2.

Systemd is stopping the VM before pacemaker is able to migrate it. I'm
guessing that is due to this line in the libvirt unit:

Before=libvirt-guests.service

It appears systemd feels free to do that part in parallel, even though
libvirt itself has to wait until pacemaker finishes stopping. Try adding
libvirt-guests to your pacemaker override.

>>>
>>>> Tia
>>>> Rainer
>>>>
>>>>
>>>> 
>>>>
>>>> IT Nerb GmbH
>>>> Lessingstraße 8
>>>> 85098 Großmehring
>>>>
>>>> Telefon : +49 700 ITNERBGMBH
>>>> Telefax : +49 8407 939 284
>>>> email : i...@it-nerb.de
>>>> Internet : www.it-nerb.de <http://www.it-nerb.de>
>>>> Geschäftsführer:Rainer Nerb
>>>> Handelsregister:HRB 2592
>>>> HR-Gericht:Ingolstadt
>>>>
>>>> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?

[Ken Gaillot]

On 11/04/2016 01:57 PM, CART Andreas wrote:
> Hi
>  
> I have a basic 2 node active/passive cluster with Pacemaker (1.1.14 ,
> pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.
> This cluster runs NFS on top of DRBD (8.4.4).
>  
> Basically the system is working on both nodes and I can switch the
> resources from one node to the other.
> But switching resources to the other node does not work, if I try to
> move just one resource and have the others follow due to the location
> constraints.
>  
> From the logged messages I see that in this “failure case” there is NO
> attempt to demote/promote the DRBD clone resource.
>  
> Here is my setup:
> ==
> Cluster Name: clst1
> Corosync Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
> Pacemaker Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
>  
> Resources:
> Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)
>   Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24
>   Operations: start interval=0s timeout=20s (IPaddrNFS-start-interval-0s)
>   stop interval=0s timeout=20s (IPaddrNFS-stop-interval-0s)
>   monitor interval=5s (IPaddrNFS-monitor-interval-5s)
> Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)
>   Attributes: nfs_shared_infodir=/var/lib/nfsserversettings/
> nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"
>   Operations: start interval=0s timeout=40 (NFSServer-start-interval-0s)
>   stop interval=0s timeout=20s (NFSServer-stop-interval-0s)
>   monitor interval=10s timeout=20s
> (NFSServer-monitor-interval-10s)
> Master: DRBDClone
>   Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
>   Resource: DRBD (class=ocf provider=linbit type=drbd)
>Attributes: drbd_resource=nfsdata
>Operations: start interval=0s timeout=240 (DRBD-start-interval-0s)
>promote interval=0s timeout=90 (DRBD-promote-interval-0s)
>demote interval=0s timeout=90 (DRBD-demote-interval-0s)
>stop interval=0s timeout=100 (DRBD-stop-interval-0s)
>monitor interval=1s timeout=5 (DRBD-monitor-interval-1s)
> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)
>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4
>   Operations: start interval=0s timeout=60
> (DRBD_global_clst-start-interval-0s)
>   stop interval=0s timeout=60
> (DRBD_global_clst-stop-interval-0s)
>   monitor interval=20 timeout=40
> (DRBD_global_clst-monitor-interval-20)
>  
> Stonith Devices:
> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst1-monitor-interval-60s)
> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst2-monitor-interval-60s)
> Fencing Levels:
>  
> Location Constraints:
>   Resource: ipmi-fence-clst1
> Disabled on: ventsi-clst1-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)
>   Resource: ipmi-fence-clst2
> Disabled on: ventsi-clst2-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)
> Ordering Constraints:
>   start IPaddrNFS then start NFSServer (kind:Mandatory)
> (id:order-IPaddrNFS-NFSServer-mandatory)
>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)
> (id:order-DRBDClone-DRBD_global_clst-mandatory)
>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)
> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)
> Colocation Constraints:
>   NFSServer with IPaddrNFS (score:INFINITY)
> (id:colocation-NFSServer-IPaddrNFS-INFINITY)
>   DRBD_global_clst with DRBDClone (score:INFINITY)
> (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)

It took me a while to notice it, it's easily overlooked, but the above
constraint is the problem. It says DRBD_global_clst must be located
where DRBDClone is running ... not necessarily where DRBDClone is
master. This constraint should be created like this:

pcs constraint colocation add DRBD_global_clst with master DBRDClone

>   IPaddrNFS with DRBD_global_clst (score:INFINITY)
> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)
>  
> Resources Defaults:
> resource-stickiness: INFINITY
> Operations Defaults:
> timeout: 10s
>  
> Cluster Properties:
> cluster-infrastructure: cman
> dc-version: 1.1.14-8.el6-70404b0
> have-watchdog: false
> last-lrm-refresh: 1478277432
> no-quorum-policy: ignore
> stonith-enabled: true
> symmetric-cluster: true
> 

Re: [ClusterLabs] pacemaker after upgrade from wheezy to jessie

[Ken Gaillot]

gt; We found some strange behavior when setting "validate-with" via
>> cibadmin, corosync.log shows the successful transaction, issuing
>> cibadmin --query gives the correct value but it is NOT written into
>> cib.xml.
>>
>> We restarted pacemaker and value is reset to pacemaker-1.1
>> If signatures for the cib.xml are generated from pacemaker/cib, which
>> algorithm is used? looks like md5 to me.
>>
>> Would it be possible to manual edit the cib.xml and generate a valid
>> cib.xml.sig to get one step further in debugging process?
>>
>> Regards, Toni
>>
>> -- 
>> Mit freundlichen Grüßen
>>
>> Toni Tschampke | t...@halle.it
>> bcs kommunikationslösungen
>> Inh. Dipl. Ing. Carsten Burkhardt
>> Harz 51 | 06108 Halle (Saale) | Germany
>> tel +49 345 29849-0 | fax +49 345 29849-22
>> www.b-c-s.de | www.halle.it | www.wivewa.de
>>
>>
>> EINFACH ADRESSEN, TELEFONATE UND DOKUMENTE VERWALTEN - MIT WIVEWA -
>> IHREM WISSENSVERWALTER FUER IHREN BETRIEB!
>>
>> Weitere Informationen erhalten Sie unter www.wivewa.de
>>
>> Am 03.11.2016 um 16:39 schrieb Toni Tschampke:
>>>  > I'm going to guess you were using the experimental 1.1 schema as the
>>>  > "validate-with" at the top of /var/lib/pacemaker/cib/cib.xml. Try
>>>  > changing the validate-with to pacemaker-next or pacemaker-1.2 and
>>> see if
>>>  > you get better results. Don't edit the file directly though; use the
>>>  > cibadmin command so it signs the end result properly.
>>>  >
>>>  > After changing the validate-with, run:
>>>  >
>>>  >crm_verify -x /var/lib/pacemaker/cib/cib.xml
>>>  >
>>>  > and fix any errors that show up.
>>>
>>> strange, the location of our cib.xml differs from your path, our cib is
>>> located in /var/lib/heartbeat/crm/
>>>
>>> running cibadmin --modify --xml-text '>> validate-with="pacemaker-1.2"/>'
>>>
>>> gave no output but was logged to corosync:
>>>
>>> cib: info: cib_perform_op:-- >> validate-with="pacemaker-1.1"/>
>>> cib: info: cib_perform_op:++ >> num_updates="1" validate-with="pacemaker-1.2" crm_feature_set="3.0.6"
>>>   have-quorum="1" cib-last-written="Thu Nov  3 10:05:52 2016"
>>> update-origin="nebel1" update-client="cibadmin" update-user="root"/>
>>>
>>> I'm guessing this change should be instantly written into the xml file?
>>> If this is the case something is wrong, greping for validate gives the
>>> old string back.
>>>
>>> >> validate-with="pacemaker-1.1" crm_feature_set="3.0.6" have-quorum="1"
>>> cib-last-written="Thu Nov  3 16:19:51 2016" update-origin="nebel1"
>>> update-client="cibadmin" update-user="root">
>>>
>>> pacemakerd --features
>>> Pacemaker 1.1.15 (Build: e174ec8)
>>> Supporting v3.0.10:
>>>
>>> Should the crm_feature_set be updated this way too? I'm guessing this is
>>> done when "cibadmin --upgrade" succeeds?
>>>
>>> We just get an timeout error when trying to upgrade it with cibadmin:
>>> Call cib_upgrade failed (-62): Timer expired
>>>
>>> Do have permissions changed from 1.1.7 to 1.1.15? when looking at our
>>> quite big /var/lib/heartbeat/crm/ folder some permissions changed:
>>>
>>> -rw--- 1 hacluster root  80K Nov  1 16:56 cib-31.raw
>>> -rw-r--r-- 1 hacluster root   32 Nov  1 16:56 cib-31.raw.sig
>>> -rw--- 1 hacluster haclient  80K Nov  1 18:53 cib-32.raw
>>> -rw--- 1 hacluster haclient   32 Nov  1 18:53 cib-32.raw.sig
>>>
>>> cib-31 was before upgrading, cib-32 after starting upgraded pacemaker
>>>
>>>
>>> -- 
>>> Mit freundlichen Grüßen
>>>
>>> Toni Tschampke | t...@halle.it
>>> bcs kommunikationslösungen
>>> Inh. Dipl. Ing. Carsten Burkhardt
>>> Harz 51 | 06108 Halle (Saale) | Germany
>>> tel +49 345 29849-0 | fax +49 345 29849-22
>>> www.b-c-s.de | www.halle.it | www.wivewa.de
>>>
>>>
>>> EINFACH ADRESSEN, TELEFONATE UND DOKUMENTE VERWALTEN - MIT WIVEWA -
>>> IHREM WISSENSVERWALTER FUER IHREN BETRIEB!
>>>
>>> Weitere Informationen erhalten Sie unter www.wivewa.de
>>>
>>> Am 03.11.2016 u

Re: [ClusterLabs] Resources wont start on new node unless it is the only active node

[Ken Gaillot]

On 11/08/2016 12:54 PM, Ryan Anstey wrote:
> I've been running a ceph cluster with pacemaker for a few months now.
> Everything has been working normally, but when I added a fourth node it
> won't work like the others, even though their OS is the same and the
> configs are all synced via salt. I also don't understand pacemaker that
> well since I followed a guide for it. If anyone could steer me in the
> right direction I would greatly appreciate it. Thank you!
> 
> - My resources only start if the new node is the only active node.
> - Once started on new node, if they are moved back to one of the
> original nodes, it won't go back to the new one.
> - My resources work 100% if I start them manually (without pacemaker).
> - (In the logs/configs below, my resources are named "unifi",
> "rbd_unifi" being the main one that's not working.)

The key is all the location constraints starting with "cli-" in your
configuration. Such constraints were added automatically by command-line
tools, rather than added by you explicitly.

For example, Pacemaker has no concept of "moving" a resource. It places
all resources where they can best run, as specified by the
configuration. So, to move a resource, command-line tools add a location
constraint making the resource prefer a different node.

The downside is that the preference doesn't automatically go away. The
resource will continue to prefer the other node until you explicitly
remove the constraint.

Command-line tools that add such constraints generally provide some way
to clear them. If you clear all those constraints, resources will again
be placed on any node equally.

Separately, you also have a default resource stickiness of 100. That
means that even after you remove the constraints, resources that are
already running will tend to stay where they are. But if you stop and
start a resource, or add a new resource, it could start on a different node.

> 
> Log when running cleaning up the resource on the NEW node:
> 
> Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
> device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
> Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr
> [ unifi doesn't exist ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0:
> not running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [
> unifi doesn't exist\n ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
> running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0:
> not running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
> 
> Log when running cleaning up the resource on the OLD node:  
> 
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
> message of desired type
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   rbd_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   fs_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   lxc_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
> (ref=pe_calc-dc-1478625686-648) derived from
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
> rbd_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
> 

Re: [ClusterLabs] Antw: Resources wont start on new node unless it is the only active node

[Ken Gaillot]

On 11/09/2016 02:33 AM, Ulrich Windl wrote:
 Ryan Anstey  schrieb am 08.11.2016 um 19:54 in 
 Nachricht
>> Log when running cleaning up the resource on the OLD node:
>>
>> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
>> 167838209
> 
> This indicates a node communication problem!

Not necessarily; it's fairly routine. In 1.1.15, we changed the wording
to "No reason to expect node to be down", and in 1.1.16, we only log it
if there's a problem.

It indicates that Pacemaker checked whether a node was expected to be
down. It does that routinely for certain normal events to react
appropriately, so most of the time it's unimportant. It's only a problem
if the node went down, so since 1.1.16, it's only logged in that situation.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/07/2016 03:47 AM, Klaus Wenninger wrote:
> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote:
>> On Mon, 7 Nov 2016 10:12:04 +0100
>> Klaus Wenninger <kwenn...@redhat.com> wrote:
>>
>>> On 11/07/2016 08:41 AM, Ulrich Windl wrote:
>>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 04.11.2016 um 22:37 in
>>>>>>> Nachricht  
>>>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:  
>>>>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:  
>>>>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 03.11.2016 um 17:08 in  
>>>>>> Nachricht
>>>>>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:  
>> ...
>>>>> Another possible use would be for a cron that needs to know whether a
>>>>> particular resource is running, and an attribute query is quicker and
>>>>> easier than something like parsing crm_mon output or probing the service. 
>>>>>  
>>>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so
>>>> besides of lacking options and inefficient implementation, why should one
>>>> be faster than the other?  
>>> attrd_updater doesn't go for the CIB
>> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private"
>> since 1.1.13:
>> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177
> That prevents values being stored in the CIB. attrd_updater should
> always talk to attrd as I got it ...

It's a bit confusing: Both crm_attribute and attrd_updater will
ultimately affect both attrd and the CIB in most cases, but *how* they
do so is different. crm_attribute modifies the CIB, and lets attrd pick
up the change from there; attrd_updater notifies attrd, and lets attrd
modify the CIB.

The difference is subtle.

With corosync 2, attrd only modifies "transient" node attributes (which
stay in effect till the next reboot), not "permanent" attributes. So
crm_attribute must be used if you want to set a permanent attribute.
crm_attribute also has the ability to modify cluster properties and
resource defaults, as well as node attributes.

On the other hand, by contacting attrd directly, attrd_updater can
change an attribute's "dampening" (how often it is flushed to the CIB),
and it can (as mentioned above) set "private" attributes that are never
written to the CIB (and thus never cause the cluster to re-calculate
resource placement).

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?

[Ken Gaillot]

erval-10)
> 
> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4
> 
>   Operations: start interval=0 timeout=60
> (DRBD_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (DRBD_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (DRBD_global_clst-monitor-interval-20)
> 
> Resource: NFS_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=xxx.xxx.xxx.xxx:/drbdmnts/global_clst/nfs
> directory=/global/nfs fstype=nfs
> 
>   Operations: start interval=0 timeout=60 (NFS_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (NFS_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (NFS_global_clst-monitor-interval-20)
> 
> Resource: BIND_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=/drbdmnts/global_clst/nfs directory=/global/nfs
> fstype=none options=bind
> 
>   Operations: start interval=0 timeout=60
> (BIND_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (BIND_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (BIND_global_clst-monitor-interval-20)
> 
>  
> 
> Stonith Devices:
> 
> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)
> 
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1
> 
>   Operations: monitor interval=60 (ipmi-fence-clst1-monitor-interval-60)
> 
> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)
> 
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1
> 
>   Operations: monitor interval=60 (ipmi-fence-clst2-monitor-interval-60)
> 
> Fencing Levels:
> 
>  
> 
> Location Constraints:
> 
>   Resource: DRBD_global_clst
> 
> Disabled on: ventsi-clst1-sync (score:-INFINITY) (role: Started)
> (id:cli-ban-DRBD_global_clst-on-ventsi-clst1-sync)
> 
>   Resource: ipmi-fence-clst1
> 
> Disabled on: ventsi-clst1-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)
> 
>   Resource: ipmi-fence-clst2
> 
> Disabled on: ventsi-clst2-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)
> 
> Ordering Constraints:
> 
>   start IPaddrNFS then start NFSServer (kind:Mandatory)
> (id:order-IPaddrNFS-NFSServer-mandatory)
> 
>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)
> (id:order-DRBDClone-DRBD_global_clst-mandatory)
> 
>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)
> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)
> 
>   start NFSServer then start NFS_global_clst (kind:Mandatory)
> (id:order-NFSServer-NFS_global_clst-mandatory)
> 
>   start NFSServer then start BIND_global_clst (kind:Mandatory)
> (id:order-NFSServer-BIND_global_clst-mandatory)
> 
> Colocation Constraints:
> 
>   NFSServer with IPaddrNFS (score:INFINITY)
> (id:colocation-NFSServer-IPaddrNFS-INFINITY)
> 
>   IPaddrNFS with DRBD_global_clst (score:INFINITY)
> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)
> 
>   NFS_global_clst with NFSServer (score:-INFINITY)
> (id:colocation-NFS_global_clst-NFSServer--INFINITY)
> 
>   BIND_global_clst with NFSServer (score:INFINITY)
> (id:colocation-BIND_global_clst-NFSServer-INFINITY)
> 
>   DRBD_global_clst with DRBDClone (score:INFINITY) (rsc-role:Started)
> (with-rsc-role:Master) (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)
> 
>  
> 
> Resources Defaults:
> 
> resource-stickiness: INFINITY
> 
> Operations Defaults:
> 
> timeout: 10s
> 
>  
> 
> Cluster Properties:
> 
> cluster-infrastructure: cman
> 
> dc-version: 1.1.14-8.el6-70404b0
> 
> have-watchdog: false
> 
> last-lrm-refresh: 1478703150
> 
> no-quorum-policy: ignore
> 
> stonith-enabled: true
> 
> symmetric-cluster: true
> 
> Node Attributes:
> 
> ventsi-clst1-sync: PostgresSon-data-status=DISCONNECT
> 
> ventsi-clst2-sync: PostgresSon-data-status=DISCONNECT
> 
>  
> 
>  
> 
> Kind regards
> 
> Andi
> 
>  
> 
> -Original Message-
> From: Ken Gaillot [mailto:kgail...@redhat.com]
> Sent: Dienstag, 8. November 2016 22:29
> To: users@clusterlabs.org
> Subject: Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?
> 
&g

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/04/2016 02:53 AM, Jan Pokorný wrote:
> On 04/11/16 08:29 +0100, Ulrich Windl wrote:
>> Ken Gaillot <kgail...@redhat.com> schrieb am 03.11.2016 um 17:08 in
>> Nachricht <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:
>>> ClusterLabs is happy to announce the first release candidate for
>>> Pacemaker version 1.1.16. Source code is available at:
>>>
>>> https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 
>>>
>>> The most significant enhancements in this release are:
>>>
>>> [...]
>>>
>>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>>> properly passed to multistate resources with notification enabled. This
>>> has been fixed. To help resource agents detect when the fix is
>>> available, the CRM feature set has been incremented. (Whenever the
>>> feature set changes, mixed-version clusters are supported only during
>>> rolling upgrades -- nodes with an older version will not be allowed to
>>> rejoin once they shut down.)
>>
>> Where can I find a description of current "CRM feature sets"?
> 
> It's originally internal-only versioning for cluster to know which
> node has the oldest software and hence is predestined to be DC
> (in rolling update scenario).
> 
> Ken recently mapped these versions (together with LRMD protocol
> versions relevant in the context of pacemaker remote communication)
> to proper release versions: http://clusterlabs.org/wiki/ReleaseCalendar

There is also a description of how the cluster uses the CRM feature set
in the new upgrade documentation:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_rolling_node_by_node



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Migration-threshold, timeout and interval

[Ken Gaillot]

On 10/19/2016 06:06 AM, Marcos Renato da Silva Junior wrote:
> Hi,
> 
> 
> It would be correct to say that both options have the same function :
> 
> 
> meta migration-threshold="3" op monitor interval=10s
> 
> op monitor interval=10 timeout=30
> 
> 
> If not what is the difference?

You have two separate configuration items here, the migration-threshold
and the op monitor.

The migration-threshold is the number of times the cluster will try
restarting a failed resource on any one node before marking that node
ineligible to serve the resource. The default is INFINITY (which is
actually interpreted as 1,000,000).

The monitor operation is the recurring check of the resource's status.
interval= 10 or 10s is equivalent, and means the check will be done
every 10 seconds. timeout=30 means the check will be killed (and failed)
if it takes longer than 30 seconds. IIRC the default timeout is 20 seconds.

> 
> 
> My resource config :
> 
> crm configure primitive slapd_mirrormode ocf:heartbeat:slapd params\
>  slapd="/usr/sbin/slapd" \
>  config="/etc/ldap/slapd.d/" \
>  user="openldap" group="openldap" \
>  services="ldap:///; \
>  watch_suffix="dc=mrsj,dc=lab" \
>  meta migration-threshold="3" \
>  op monitor interval=10s
> 
> 
> 
> Thanks,
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Can't do anything right; how do I start over?

[Ken Gaillot]

On 10/14/2016 02:48 PM, Jay Scott wrote:
> I've been trying a lot of things from the introductory manual.
> I have updated the instructions (on my hardcopy) to the versions
> of corosync etc. that I'm using.  I can't get hardly anything to
> work reliably beyond the ClusterIP.
> 
> So I start over -- I had been reinstalling the machines but I've
> grown tired of that.  So, before I start in on my other tales of woe,
> I figured I should find out how to start over "according to Hoyle".
> 
> When I "start over" I stop all the services, delete the packages,
> empty the configs and logs as best I know how.  But this doesn't
> completely clear everything:  the drbd metadata is evidently still
> on the partitions I've set aside for it.
> 
> 
> 
> Oh, before I forget, in particular:
> in corosync.conf:
> totem {
> interface {
> # This is normally the *network* address of the
> # interface to bind to. This ensures that you can use
> # identical instances of this configuration file
> # across all your cluster nodes, without having to
> # modify this option.
> bindnetaddr: 10.1.1.22
> [snip]
> }
> }
> bindnetaddr:  I've tried using an address on ONE of the machines
> (everywhere),
> and I've tried using an address that's on each participating machine,
> thus a diff corosync.conf file for each machine (but otherwise identical).
> What's the right thing?  From the comment it seems that there should
> be one address used among all machines.  But I kept getting messages
> about addresses already in use, so I thought I'd try to "fix" it.

The comment may be unclear ... bindnetaddr isn't an address *on* the
network, it's the address *of* the network.

For example, if you're using a /24 subnet (255.255.255.0 netmask), the
above bindnetaddr should be 10.1.1.0, which would cover any hosts with
addresses in the range 10.1.1.1 - 10.1.1.254.

> 
> This is my burn script.
> Am I missing something?  Doing it wrong?
> 
> #!/bin/bash
> pkill -9 -f pacemaker
> systemctl stop pacemaker.service
> systemctl stop corosync.service
> systemctl stop pcsd.service
> drbdadm down alpha
> drbdadm down bravo
> drbdadm down delta
> systemctl stop drbd.service
> 
> rpm -e drbd84-utils kmod-drbd84
> rpm -e pcs
> rpm -e pacemaker
> rpm -e pacemaker-cluster-libs
> rpm -e pacemaker-cli
> rpm -e pacemaker-libs
> rpm -e pacemaker-doc
> rpm -e lvm2-cluster
> rpm -e dlm
> rpm -e corosynclib corosync
> cd /var/lib/pacemaker
> rm cib/*
> rm pengine/*
> cd
> nullfile /var/log/cluster/corosync.conf

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Unexpected Resource movement after failover

[Ken Gaillot]

On 10/17/2016 11:29 PM, Nikhil Utane wrote:
> Thanks Ken.
> I will give it a shot.
> 
> http://oss.clusterlabs.org/pipermail/pacemaker/2011-August/011271.html
> On this thread, if I interpret it correctly, his problem was solved when
> he swapped the anti-location constraint 
> 
> From (mapping to my example)
> cu_2 with cu_4 (score:-INFINITY)
> cu_3 with cu_4 (score:-INFINITY)
> cu_2 with cu_3 (score:-INFINITY)
> 
> To
> cu_2 with cu_4 (score:-INFINITY)
> cu_4 with cu_3 (score:-INFINITY)
> cu_3 with cu_2 (score:-INFINITY)
> 
> Do you think that would make any difference? The way you explained it,
> sounds to me it might.

It would create a dependency loop:

cu_2 must be placed before cu_3
cu_3 must be placed before cu_4
cu_4 must be placed before cu_2
(loop)

The cluster tries to detect and break such loops, but I wouldn't rely on
that resulting in a particular behavior.

> -Regards
> Nikhil
> 
> On Mon, Oct 17, 2016 at 11:36 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 10/17/2016 09:55 AM, Nikhil Utane wrote:
> > I see these prints.
> >
> > pengine: info: rsc_merge_weights:cu_4: Rolling back scores from cu_3
> > pengine:debug: native_assign_node:Assigning Redun_CU4_Wb30 to cu_4
> > pengine: info: rsc_merge_weights:cu_3: Rolling back scores from cu_2
> > pengine:debug: native_assign_node:Assigning Redund_CU5_WB30 to cu_3
> >
> > Looks like rolling back the scores is causing the new decision to
> > relocate the resources.
> > Am I using the scores incorrectly?
> 
> No, I think this is expected.
> 
> Your anti-colocation constraints place cu_2 and cu_3 relative to cu_4,
> so that means the cluster will place cu_4 first if possible, before
> deciding where the others should go. Similarly, cu_2 has a constraint
> relative to cu_3, so cu_3 gets placed next, and cu_2 is the one left
> out.
> 
> The anti-colocation scores of -INFINITY outweigh the stickiness of 100.
> I'm not sure whether setting stickiness to INFINITY would change
> anything; hopefully, it would stop cu_3 from moving, but cu_2 would
> still be stopped.
> 
> I don't see a good way around this. The cluster has to place some
> resource first, in order to know not to place some other resource on the
> same node. I don't think there's a way to make them "equal", because
> then none of them could be placed to begin with -- unless you went with
> utilization attributes, as someone else suggested, with
> placement-strategy=balanced:
> 
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140521708557280
> 
> <http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140521708557280>
> 
> >
> > [root@Redund_CU5_WB30 root]# pcs constraint
> > Location Constraints:
> >   Resource: cu_2
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> >   Resource: cu_3
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> >   Resource: cu_4
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> > Ordering Constraints:
> > Colocation Constraints:
> >   cu_2 with cu_4 (score:-INFINITY)
> >   cu_3 with cu_4 (score:-INFINITY)
> >   cu_2 with cu_3 (score:-INFINITY)
> >
> >
> > On Mon, Oct 17, 2016 at 8:16 PM, Nikhil Utane
> > <nikhil.subscri...@gmail.com <mailto:nikhil.subscri...@gmail.com>
> <mailto:nikhil.subscri...@gmail.com
> <mailto:nikhil.subscri...@gmail.com>>> wrote:
> >
> > This is driving me insane.
> >
> > This is how the resources were started. Redund_CU1_WB30  was the DC
> > which I rebooted.
> >  cu_4(ocf::redundancy:RedundancyRA):Started Redund_CU1_WB30
> >  cu_2(ocf::redundancy:RedundancyRA):Started Redund_CU5_WB30
> >  cu_3(ocf::redundancy:RedundancyRA):Started Redun_CU4_Wb30
> >
> > Since the standby node was not UP. I was expecting resource 

Re: [ClusterLabs] Antw: Re: Antw: Unexpected Resource movement after failover

[Ken Gaillot]

On 10/17/2016 09:55 AM, Nikhil Utane wrote:
> I see these prints. 
> 
> pengine: info: rsc_merge_weights:cu_4: Rolling back scores from cu_3 
> pengine:debug: native_assign_node:Assigning Redun_CU4_Wb30 to cu_4
> pengine: info: rsc_merge_weights:cu_3: Rolling back scores from cu_2 
> pengine:debug: native_assign_node:Assigning Redund_CU5_WB30 to cu_3   
> 
> Looks like rolling back the scores is causing the new decision to
> relocate the resources.
> Am I using the scores incorrectly?

No, I think this is expected.

Your anti-colocation constraints place cu_2 and cu_3 relative to cu_4,
so that means the cluster will place cu_4 first if possible, before
deciding where the others should go. Similarly, cu_2 has a constraint
relative to cu_3, so cu_3 gets placed next, and cu_2 is the one left out.

The anti-colocation scores of -INFINITY outweigh the stickiness of 100.
I'm not sure whether setting stickiness to INFINITY would change
anything; hopefully, it would stop cu_3 from moving, but cu_2 would
still be stopped.

I don't see a good way around this. The cluster has to place some
resource first, in order to know not to place some other resource on the
same node. I don't think there's a way to make them "equal", because
then none of them could be placed to begin with -- unless you went with
utilization attributes, as someone else suggested, with
placement-strategy=balanced:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140521708557280

> 
> [root@Redund_CU5_WB30 root]# pcs constraint
> Location Constraints:
>   Resource: cu_2
> Enabled on: Redun_CU4_Wb30 (score:0)
> Enabled on: Redund_CU5_WB30 (score:0)
> Enabled on: Redund_CU3_WB30 (score:0)
> Enabled on: Redund_CU1_WB30 (score:0)
>   Resource: cu_3
> Enabled on: Redun_CU4_Wb30 (score:0)
> Enabled on: Redund_CU5_WB30 (score:0)
> Enabled on: Redund_CU3_WB30 (score:0)
> Enabled on: Redund_CU1_WB30 (score:0)
>   Resource: cu_4
> Enabled on: Redun_CU4_Wb30 (score:0)
> Enabled on: Redund_CU5_WB30 (score:0)
> Enabled on: Redund_CU3_WB30 (score:0)
> Enabled on: Redund_CU1_WB30 (score:0)
> Ordering Constraints:
> Colocation Constraints:
>   cu_2 with cu_4 (score:-INFINITY)
>   cu_3 with cu_4 (score:-INFINITY)
>   cu_2 with cu_3 (score:-INFINITY)
> 
> 
> On Mon, Oct 17, 2016 at 8:16 PM, Nikhil Utane
> <nikhil.subscri...@gmail.com <mailto:nikhil.subscri...@gmail.com>> wrote:
> 
> This is driving me insane. 
> 
> This is how the resources were started. Redund_CU1_WB30  was the DC
> which I rebooted.
>  cu_4(ocf::redundancy:RedundancyRA):Started Redund_CU1_WB30 
>  cu_2(ocf::redundancy:RedundancyRA):Started Redund_CU5_WB30 
>  cu_3(ocf::redundancy:RedundancyRA):Started Redun_CU4_Wb30 
> 
> Since the standby node was not UP. I was expecting resource cu_4 to
> be waiting to be scheduled.
> But then it re-arranged everything as below.
>  cu_4(ocf::redundancy:RedundancyRA):Started Redun_CU4_Wb30 
>  cu_2(ocf::redundancy:RedundancyRA):Stopped 
>  cu_3(ocf::redundancy:RedundancyRA):Started Redund_CU5_WB30 
> 
> There is not much information available in the logs on new DC. It
> just shows what it has decided to do but nothing to suggest why it
> did it that way.
> 
> notice: Start   cu_4(Redun_CU4_Wb30)   
> notice: Stopcu_2(Redund_CU5_WB30)  
> notice: Movecu_3(Started Redun_CU4_Wb30 -> Redund_CU5_WB30)
> 
> I have default stickiness set to 100 which is higher than any score
> that I have configured.
> I have migration_threshold set to 1. Should I bump that up instead?
> 
> -Thanks
> Nikhil
> 
> On Sat, Oct 15, 2016 at 12:36 AM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 10/14/2016 06:56 AM, Nikhil Utane wrote:
> > Hi,
> >
> > Thank you for the responses so far.
> > I added reverse colocation as well. However seeing some other issue 
> in
> > resource movement that I am analyzing.
> >
> > Thinking further on this, why doesn't "/a not with b" does not
> imply "b
> > not with a"?/
> > Coz wouldn't putting "b with a" violate "a not with b"?
> >
> > Can someone confirm that colocation is required to be configured 
> both ways?
> 
> The anti-colocation should only be defined one-way. Otherwise,
> you get a
> dependency loop (as seen in logs you showed elsewhere).
&

Re: [ClusterLabs] set start-failure-is-fatal per resource?

[Ken Gaillot]

On 10/17/2016 12:42 PM, Israel Brewster wrote:
> I have one resource agent (redis, to be exact) that sometimes apparently
> fails to start on the first attempt. In every case, simply running a
> 'pcs resource cleanup' such that pacemaker tries to start it again
> successfully starts the process. Now, obviously, the proper thing to do
> is to figure out why redis sometimes fails to start, and fix that (it
> sort of feels like there may be a race condition going on somewhere to
> me), which I fully intend to spend some time investigating. However, in
> the meantime the quick "fix" is to simply have Pacemaker try starting
> the resource again if it fails the first time. This is easily
> accomplished by setting the property start-failure-is-fatal to false,
> which works beautifully for the redis resource. However, this is a
> global setting, and for most resources it doesn't make sense - if they
> don't start on the first try, I'm probably going to need to fix
> something before they will start.
> 
> So the question is, is there a way to set the start-failure-is-fatal
> property to false only for one resource, or in some other way configure
> the resources such that the effect is the same - i.e. the one resource I
> want will retry start, but the others will give up after a single
> failure? Thanks.

Currently, start-failure-is-fatal can only be set globally. There was a
recent thread about a major overhaul of Pacemaker's failure handling
configuration that would make the equivalent functionality per-resource,
so this may change in a future version.

In the meantime, you can get the behavior you want by setting
start-failure-is-fatal=false, and setting migration-threshold=1 on the
resources you want to fail after a single failure. However, that will
also affect monitor failures (i.e. the resource will move to another
node after a single monitor failure).

> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD Insufficient Privileges Error

[Ken Gaillot]

On 11/20/2016 01:58 PM, Jasim Alam wrote:
> Hi,
> 
>  
> 
> I am trying to setup  two node H/A cluster with DRBD. Following is my
> configuration
> 
>  
> 
> /[root@node-1 ~]# pcs config/
> 
> /Cluster Name: Cluster-1/
> 
> /Corosync Nodes:/
> 
> /node-1 node-2 /
> 
> /Pacemaker Nodes:/
> 
> /node-1 node-2 /
> 
> / /
> 
> /Resources: /
> 
> / Resource: vip (class=ocf provider=heartbeat type=IPaddr2)/
> 
> /  Attributes: ip=103.9.185.211 cidr_netmask=32 /
> 
> /  Operations: start interval=0s timeout=20s (vip-start-interval-0s)/
> 
> /  stop interval=0s timeout=20s (vip-stop-interval-0s)/
> 
> /  monitor interval=30s (vip-monitor-interval-30s)/
> 
> /Resource: apache (class=ocf provider=heartbeat type=apache)/
> 
> /  Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://localhost/server-status /
> 
> /  Operations: start interval=0s timeout=40s (apache-start-interval-0s)/
> 
> /  stop interval=0s timeout=60s (apache-stop-interval-0s)/
> 
> /  monitor interval=1min (apache-monitor-interval-1min)/
> 
> /Master: StorageClone/
> 
> /  Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true /
> 
> /  Resource: storage (class=ocf provider=linbit type=drbd)/
> 
> /   Attributes: drbd_resource=drbd0 /
> 
> /   Operations: start interval=0s timeout=240 (storage-start-interval-0s)/
> 
> /   promote interval=0s timeout=90
> (storage-promote-interval-0s)/
> 
> /   demote interval=0s timeout=90 (storage-demote-interval-0s)/
> 
> /   stop interval=0s timeout=100 (storage-stop-interval-0s)/
> 
> /   monitor interval=60s (storage-monitor-interval-60s)/
> 
> / /
> 
> /Stonith Devices: /
> 
> /Fencing Levels: /
> 
> / /
> 
> /Location Constraints:/
> 
> /  Resource: apache/
> 
> /Enabled on: node-1 (score:50) (id:location-apache-node-1-50)/
> 
> /Ordering Constraints:/
> 
> /  start vip then start apache (kind:Mandatory)
> (id:order-vip-apache-mandatory)/
> 
> /Colocation Constraints:/
> 
> /  vip with apache (score:INFINITY) (id:colocation-vip-apache-INFINITY)/
> 
> / /
> 
> /Resources Defaults:/
> 
> /No defaults set/
> 
> /Operations Defaults:/
> 
> /No defaults set/
> 
> / /
> 
> /Cluster Properties:/
> 
> /cluster-infrastructure: corosync/
> 
> /cluster-name: Cluster-1/
> 
> /dc-version: 1.1.13-10.el7_2.4-44eb2dd/
> 
> /have-watchdog: false/
> 
> /no-quorum-policy: ignore/
> 
> /stonith-enabled: false/
> 
>  
> 
> The problem is I am getting insufficient privilege error on second node
> 
>  
> 
> /[root@node-1 ~]# pcs status/
> 
> /Cluster name: PSD-1/
> 
> /Last updated: Mon Nov 21 01:44:52 2016  Last change: Mon Nov 21
> 01:19:17 2016 by root via cibadmin on node-1/
> 
> /Stack: corosync/
> 
> /Current DC: node-1 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with
> quorum/
> 
> /2 nodes and 4 resources configured/
> 
> / /
> 
> /Online: [ node-1 node-2 ]/
> 
> / /
> 
> /Full list of resources:/
> 
> / /
> 
> /vip(ocf::heartbeat:IPaddr2):   Started node-1/
> 
> /apache (ocf::heartbeat:apache):Started node-1/
> 
> /Master/Slave Set: StorageClone [storage]/
> 
> / storage(ocf::linbit:drbd): FAILED node-2 (unmanaged)/
> 
> / Masters: [ node-1 ]/
> 
> / /
> 
> /Failed Actions:/
> 
> /* storage_stop_0 on node-2 'insufficient privileges' (4): call=16,
> status=complete, exitreason='none',/
> 
> /last-rc-change='Mon Nov 21 01:19:17 2016', queued=0ms, exec=2ms/
> 
> / /
> 
> / /
> 
> /PCSD Status:/
> 
> /  node-1: Online/
> 
> /  node-2: Online/
> 
> / /
> 
> /Daemon Status:/
> 
> /  corosync: active/disabled/
> 
> /  pacemaker: active/disabled/
> 
> /  pcsd: active/enabled/
> 
>  
> 
> but DRBD seems okay for both nodes
> 
>  
> 
> /  [root@node-1 ~]# drbd-overview /
> 
> / 0:drbd0/0  Connected Primary/Secondary UpToDate/UpToDate/
> 
> //
> 
> / [root@node-2 ~]# drbd-overview /
> 
> / 0:drbd0/0  Connected Secondary/Primary UpToDate/UpToDate/
> 
>  
> 
> Log of node2
> 
>  
> 
> /[root@node-2 ~]# tail -n 10 /var/log/messages/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: State transition S_NOT_DC
> -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> origin=do_election_count_vote ]/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: State transition S_PENDING
> -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
> origin=do_cl_join_finalize_respond ]/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:   error: Failed to retrieve
> meta-data for ocf:linbit:drbd/

Pacemaker is unable to get the metadata for the resource agent. Try
getting it manually:

 OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/linbit/drbd meta-data

If that works, check whether selinux is blocking it.

> 
> /Nov 21 01:19:17 node-2 crmd[4060]: warning: No metadata found for
> drbd::ocf:linbit: Input/output error (-5)/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:   error: No metadata for
> linbit::ocf:drbd/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: Operation
> 

Re: [ClusterLabs] Reliable check for "is starting" state of a resource

[Ken Gaillot]

On 11/22/2016 10:53 AM, Kostiantyn Ponomarenko wrote:
> Hi folks,
> 
> I am looking for a good way of checking if a resource is in "starting"
> state.
> The thing is - I need to issue a command and I don't want to issue that
> command when this particular resource is starting. This resource start
> can take up to a few min. 
> As a note, I am OK with issuing that command if a resource is stopped
> (before it starts).
> 
> The best I can think of, is to check for an ongoing state transition of
> a resource in a loop with "crm_simulate -Ls" command.
> But with this approach I need to put the command into a loop, cut
> "Transition Summary" part, and then grep for the needed resource.
> And even that I have a question if this way is reliable?
> 
> Maybe be there is a better way of achieving the same result.  
> 
> Thank you,
> Kostia

Probably the cleanest way is to set record-pending=true (either as an
operation default, or just on the start operation you're interested in).
Then your command (or a wrapper) could check the CIB for a pending start.

A simpler approach would be "crm_resource --wait", which blocks until
the cluster is idle. The downside is you might wait for other actions
that you don't care about.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> Using "shadow cib" in crmsh looks like a good idea, but it doesn't work
> with node attributes set into "status" section of Pacemaker config.
> I wonder it it is possible to make it work that way.

Forgot to mention -- the shadow CIB is probably the best way to do this.
I don't know if there's a way to do it in crmsh, but you can use it with
the low-level commands crm_shadow and crm_attribute --lifetime=reboot.

> Ken,
>>> start dampening timer
> Could you please elaborate more on this. I don't get how I can set this
> timer.
> Do I need to set this timer for each node?
> 
> 
> Thank you,
> Kostia
> 
> On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
> <ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>> wrote:
> 
> >>> Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>>
> schrieb am 18.11.2016 um 16:17 in Nachricht
> <d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>:
> > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> >> Hi folks,
> >>
> >> Is there a way to set a node attribute to the "status" section for few
> >> nodes at the same time?
> >>
> >> In my case there is a node attribute which allows some resources to
> >> start in the cluster if it is set.
> >> If I set this node attribute for say two nodes in a way - one and then
> >> another, than these resources are not distributed equally between these
> >> two nodes. That because Pacemaker picks the first node to with this
> >> attribute is set and immediately starts all allowed resources on it. 
> And
> >> this is not the behavior i would like to get.
> >>
> >> Thank you,
> >> Kostia
> >
> > Not that I know of, but it would be a good feature to add to
> > attrd_updater and/or crm_attribute.
> 
> With crm (shell) you don't have transactions for node attributes,
> but for the configuration. So if you add a location restriction
> preventing any resources on your nodes, then enable the nodes, and
> then delete the location restrictions in one transaction, you might
> get what you want. It's not elegant, but itt ill do.
> 
> To the crm shell maintainer: Is is difficult to build transactions
> to node status changes? The problem I see is this: For configuration
> you always have transactions (requiring "commit), but for nodes you
> traditionally have non (effects are immediate). So you'd need a
> thing like "start transaction" which requires a "commit" or some
> kind of abort later.
> 
> I also don't know whether a "shadow CIB" would help for the original
> problem.
> 
> Ulrich
> 
> >
> > You can probably hack it with a dampening value of a few seconds. If
> > your rule checks for a particular value of the attribute, set all the
> > nodes to a different value first, which will write that value and
> start
> > the dampening timer. Then set all the attributes to the desired value,
> > and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> Using "shadow cib" in crmsh looks like a good idea, but it doesn't work
> with node attributes set into "status" section of Pacemaker config.
> I wonder it it is possible to make it work that way.
> 
> Ken,
>>> start dampening timer
> Could you please elaborate more on this. I don't get how I can set this
> timer.
> Do I need to set this timer for each node?

Dampening is per attribute, so it applies to all nodes. You can set it
when you first create the attribute:

  attrd_updater -n $NAME --update $VALUE --delay $SECONDS

With dampening (delay), the attribute daemon will wait that long between
writes to the CIB. The goal is to reduce I/O activity for frequently
changing attributes, but it could also be handy here.


The --delay will be ignored if the above command is run after the
attribute already exists. You can change it for an already existing
attribute with

  attrd_updater -n $NAME --update-delay --delay $SECONDS

or

  attrd_updater -n $NAME --update-both $VALUE --delay $SECONDS


It's intentionally more trouble to set it on an already-created
attribute, because repeatedly changing the delay will make it useless
(each delay change requires an immediate write). Having a separate
command makes it less likely to be accidental.

> 
> 
> Thank you,
> Kostia
> 
> On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
> <ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>> wrote:
> 
> >>> Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>>
> schrieb am 18.11.2016 um 16:17 in Nachricht
> <d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>:
> > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> >> Hi folks,
> >>
> >> Is there a way to set a node attribute to the "status" section for few
> >> nodes at the same time?
> >>
> >> In my case there is a node attribute which allows some resources to
> >> start in the cluster if it is set.
> >> If I set this node attribute for say two nodes in a way - one and then
> >> another, than these resources are not distributed equally between these
> >> two nodes. That because Pacemaker picks the first node to with this
> >> attribute is set and immediately starts all allowed resources on it. 
> And
> >> this is not the behavior i would like to get.
> >>
> >> Thank you,
> >> Kostia
> >
> > Not that I know of, but it would be a good feature to add to
> > attrd_updater and/or crm_attribute.
> 
> With crm (shell) you don't have transactions for node attributes,
> but for the configuration. So if you add a location restriction
> preventing any resources on your nodes, then enable the nodes, and
> then delete the location restrictions in one transaction, you might
> get what you want. It's not elegant, but itt ill do.
> 
> To the crm shell maintainer: Is is difficult to build transactions
> to node status changes? The problem I see is this: For configuration
> you always have transactions (requiring "commit), but for nodes you
> traditionally have non (effects are immediate). So you'd need a
> thing like "start transaction" which requires a "commit" or some
> kind of abort later.
> 
> I also don't know whether a "shadow CIB" would help for the original
> problem.
> 
> Ulrich
> 
> >
> > You can probably hack it with a dampening value of a few seconds. If
> > your rule checks for a particular value of the attribute, set all the
> > nodes to a different value first, which will write that value and
> start
> > the dampening timer. Then set all the attributes to the desired value,
> > and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Locate resource with functioning member of clone set?

[Ken Gaillot]

On 11/22/2016 02:28 PM, Israel Brewster wrote:
> On Nov 17, 2016, at 4:04 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
>>
>> On 11/17/2016 11:37 AM, Israel Brewster wrote:
>>> I have a resource that is set up as a clone set across my cluster,
>>> partly for pseudo-load balancing (If someone wants to perform an action
>>> that will take a lot of resources, I can have them do it on a different
>>> node than the primary one), but also simply because the resource can
>>> take several seconds to start, and by having it already running as a
>>> clone set, I can failover in the time it takes to move an IP resource -
>>> essentially zero down time.
>>>
>>> This is all well and good, but I ran into a problem the other day where
>>> the process on one of the nodes stopped working properly. Pacemaker
>>> caught the issue, and tried to fix it by restarting the resource, but
>>> was unable to because the old instance hadn't actually exited completely
>>> and was still tying up the TCP port, thereby preventing the new instance
>>> that pacemaker launched from being able to start.
>>>
>>> So this leaves me with two questions: 
>>>
>>> 1) is there a way to set up a "kill script", such that before trying to
>>> launch a new copy of a process, pacemaker will run this script, which
>>> would be responsible for making sure that there are no other instances
>>> of the process running?
>>
>> Sure, it's called a resource agent :)
>>
>> When recovering a failed resource, Pacemaker will call the resource
>> agent's stop action first, then start. The stop should make sure the
>> service has exited completely. If it doesn't, the agent should be fixed
>> to do so.
> 
> Ah, gotcha. I wasn't thinking along those lines in this case because the
> resource in question doesn't have a dedicated resource agent - it's a
> basic system service type resource. So then the proper approach would be
> to modify the init.d script such that when "stop" is called, it makes
> sure to completely clean up any associated processes - even if the PID
> file disappears or gets changed.

Correct. You could also use the init script as a starting point for a
proper OCF agent if you wanted the flexibility to add more capabilities
in the future. If you're interested, see:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf

>>
>>> 2) Even in the above situation, where pacemaker couldn't launch a good
>>> copy of the resource on the one node, the situation could have been
>>> easily "resolved" by pacemaker moving the virtual IP resource to another
>>> node where the cloned resource was running correctly, and notifying me
>>> of the problem. I know how to make colocation constraints in general,
>>> but how do I do a colocation constraint with a cloned resource where I
>>> just need the virtual IP running on *any* node where there clone is
>>> working properly? Or is it the same as any other colocation resource,
>>> and pacemaker is simply smart enough to both try to restart the failed
>>> resource and move the virtual IP resource at the same time?
>>
>> Correct, a simple colocation constraint of "resource R with clone C"
>> will make sure R runs with a working instance of C.
>>
>> There is a catch: if *any* instance of C restarts, R will also restart
>> (even if it stays in the same place), because it depends on the clone as
>> a whole. Also, in the case you described, pacemaker would first try to
>> restart both C and R on the same node, rather than move R to another
>> node (although you could set on-fail=stop on C to force R to move).
> 
> It *looked* like Pacemaker was continually trying to restart the cloned
> resource in this case - I think the issue being that from Pacemakers
> perspective the service *did* start successfully, it just failed again
> moments later (when it tried to bind to the port, and, being unable to,
> bailed out). So under the "default" configuration, Pacemaker would try
> restarting the service for quite a while before marking it as failed on
> that node. As such, it sounds like under the current configuration, the
> IP resource would never move (at least not in a reasonable time frame),
> as Pacemaker would simply continue to try restarting on the same node.
> 
> So to get around this, I'm thinking I could set the migration-threshold
> property on the clustered resource to something low, like two or three,
> perhaps combined with a failure-

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/23/2016 11:27 AM, Kostiantyn Ponomarenko wrote:
> Maybe I am doing something wrong, but I cannot set "status" section node
> attributes to a shadow cib, cluster applies them immediately.
> To try it out I do in a console:
> 
> crm_shadow --create test
> crm_attribute --type nodes --node node-0 --name my-attribute
> --update 1 --lifetime=reboot
> 
> And this attribute is set to the live cluster configuration immediately.
> What am I doing wrong?

Strange ... if you use --lifetime=forever, it's recorded in the shadow
CIB, but if you use --lifetime=reboot, it's recorded in the live CIB.
That looks like a bug to me; I'll make a note to investigate.

> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 11:33 PM, Kostiantyn Ponomarenko
> <konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>> wrote:
> 
> Ken,
> Thank you for the explanation.
> I will try this low-level way of shadow cib creation tomorrow.
> PS: I will sleep much better with this excellent news/idea. =)
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 10:53 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> > Using "shadow cib" in crmsh looks like a good idea, but it doesn't 
> work
> > with node attributes set into "status" section of Pacemaker config.
> > I wonder it it is possible to make it work that way.
> 
> Forgot to mention -- the shadow CIB is probably the best way to
> do this.
> I don't know if there's a way to do it in crmsh, but you can use
> it with
> the low-level commands crm_shadow and crm_attribute
> --lifetime=reboot.
> 
> > Ken,
> >>> start dampening timer
> > Could you please elaborate more on this. I don't get how I can set 
> this
> > timer.
> > Do I need to set this timer for each node?
> >
> >
> > Thank you,
> > Kostia
> >
> > On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
> > <ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>
> > <mailto:ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>>> wrote:
> >
> > >>> Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com> <mailto:kgail...@redhat.com
> <mailto:kgail...@redhat.com>>>
> > schrieb am 18.11.2016 um 16:17 in Nachricht
> > <d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>
> > <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>>:
> > > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> > >> Hi folks,
> > >>
> > >> Is there a way to set a node attribute to the "status"
> section for few
> > >> nodes at the same time?
> > >>
> > >> In my case there is a node attribute which allows some
> resources to
> > >> start in the cluster if it is set.
> > >> If I set this node attribute for say two nodes in a way
> - one and then
> > >> another, than these resources are not distributed
> equally between these
> > >> two nodes. That because Pacemaker picks the first node
> to with this
> > >> attribute is set and immediately starts all allowed
> resources on it. And
> > >> this is not the behavior i would like to get.
> > >>
> > >> Thank you,
> > >> Kostia
> > >
> > > Not that I know of, but it would be a good feature to add to
> > > attrd_updater and/or crm_attribute.
> >
> > With crm (shell) you don't have transactions for node
> attributes,
> > but for the configuration. So if you add a location
> restriction
> > preventing any resources on your nodes, then enable the
> nodes, and
> > then delete the location restrictions in one tra

[ClusterLabs] Pacemaker 1.1.16 - Release Candidate 2

[Ken Gaillot]

The second release candidate for Pacemaker version 1.1.16 is now
available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc2

It contains only minor fixes compared to 1.1.16-rc1. If there are no
significant issues found in the next couple of weeks, I will release
this as the final 1.1.16. Any feedback is appreciated.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: late stop of dependency?

[Ken Gaillot]

On 11/17/2016 02:46 AM, Ulrich Windl wrote:
> Hi!
> 
> I have a question:
> When having dependencies like "A has to start before B" and "A has to start 
> before C". Now when shutting down, B and C are shut down before A, as 
> requested.
> Now when B takes a long time to stop, C is stopped early. I wonder whether I 
> can tell pacemaker to stop C late, but I don't want to add a dependency like 
> "C has to stop before B" (I don't want a functional dependency between B and 
> C, because there is none).
> If that's too abstract: Think A is a resource B needs, and C is a performance 
> monitor for A.
> 
> Regards,
> Ulrich

I'd make "stop B then stop C" an asymmetrical, optional constraint.

asymmetrical = it only applies when stopping, not starting

optional = only apply the order if they happen to be stopping in the
same transition

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> Hi folks,
> 
> Is there a way to set a node attribute to the "status" section for few
> nodes at the same time?
> 
> In my case there is a node attribute which allows some resources to
> start in the cluster if it is set.
> If I set this node attribute for say two nodes in a way - one and then
> another, than these resources are not distributed equally between these
> two nodes. That because Pacemaker picks the first node to with this
> attribute is set and immediately starts all allowed resources on it. And
> this is not the behavior i would like to get.
> 
> Thank you,
> Kostia

Not that I know of, but it would be a good feature to add to
attrd_updater and/or crm_attribute.

You can probably hack it with a dampening value of a few seconds. If
your rule checks for a particular value of the attribute, set all the
nodes to a different value first, which will write that value and start
the dampening timer. Then set all the attributes to the desired value,
and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Query about resource stickiness

[Ken Gaillot]

On 11/17/2016 06:41 PM, phanidhar prattipati wrote:
> Good Morning All,
> 
> I have configured HA on 3 nodes and in order to disable automatic fail
> over i need to set resource stickiness value and not sure how to
> calculate it. Currently i set it o INFINITY which i believe is not the
> right way of doing it. Any pointers how to calculate based on the
> environment set up will be really great help?
> 
> 
> -- 
> Thanks,
> Phanidhar

INFINITY is fine -- many people use that for stickiness.

It's simply a matter of preference. How it matters is when weighing
against other scores in your configuration.

For example, let's say you have a resource R with a location constraint
preferring node N, and you have resource stickiness.

If N crashes or is shut down, R will move to another node. When N comes
back, R will stay on the other node, if the resource stickiness is
higher than the location constraint's score; it will move back to N, if
the location constraint's score is higher.

A score of INFINITY means never move back, as long as the new node stays up.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Locate resource with functioning member of clone set?

[Ken Gaillot]

On 11/17/2016 11:37 AM, Israel Brewster wrote:
> I have a resource that is set up as a clone set across my cluster,
> partly for pseudo-load balancing (If someone wants to perform an action
> that will take a lot of resources, I can have them do it on a different
> node than the primary one), but also simply because the resource can
> take several seconds to start, and by having it already running as a
> clone set, I can failover in the time it takes to move an IP resource -
> essentially zero down time.
> 
> This is all well and good, but I ran into a problem the other day where
> the process on one of the nodes stopped working properly. Pacemaker
> caught the issue, and tried to fix it by restarting the resource, but
> was unable to because the old instance hadn't actually exited completely
> and was still tying up the TCP port, thereby preventing the new instance
> that pacemaker launched from being able to start.
> 
> So this leaves me with two questions: 
> 
> 1) is there a way to set up a "kill script", such that before trying to
> launch a new copy of a process, pacemaker will run this script, which
> would be responsible for making sure that there are no other instances
> of the process running?

Sure, it's called a resource agent :)

When recovering a failed resource, Pacemaker will call the resource
agent's stop action first, then start. The stop should make sure the
service has exited completely. If it doesn't, the agent should be fixed
to do so.

> 2) Even in the above situation, where pacemaker couldn't launch a good
> copy of the resource on the one node, the situation could have been
> easily "resolved" by pacemaker moving the virtual IP resource to another
> node where the cloned resource was running correctly, and notifying me
> of the problem. I know how to make colocation constraints in general,
> but how do I do a colocation constraint with a cloned resource where I
> just need the virtual IP running on *any* node where there clone is
> working properly? Or is it the same as any other colocation resource,
> and pacemaker is simply smart enough to both try to restart the failed
> resource and move the virtual IP resource at the same time?

Correct, a simple colocation constraint of "resource R with clone C"
will make sure R runs with a working instance of C.

There is a catch: if *any* instance of C restarts, R will also restart
(even if it stays in the same place), because it depends on the clone as
a whole. Also, in the case you described, pacemaker would first try to
restart both C and R on the same node, rather than move R to another
node (although you could set on-fail=stop on C to force R to move).

If that's not sufficient, you could try some magic with node attributes
and rules. The new ocf:pacemaker:attribute resource in 1.1.16 could help
there.

> As an addendum to question 2, I'd be interested in any methods there may
> be to be notified of changes in the cluster state, specifically things
> like when a resource fails on a node - my current nagios/icinga setup
> doesn't catch that when pacemaker properly moves the resource to a
> different node, because the resource remains up (which, of course, is
> the whole point), but it would still be good to know something happened
> so I could look into it and see if something needs fixed on the failed
> node to allow the resource to run there properly.

Since 1.1.15, Pacemaker has alerts:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm47975782138832

Before 1.1.15, you can use the ocf:pacemaker:ClusterMon resource to do
something similar.

> 
> Thanks!
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Bug in ocf-shellfuncs, ocf_local_nodename function?

[Ken Gaillot]

On 11/17/2016 11:59 AM, Israel Brewster wrote:
> This refers specifically to build version
> 5434e9646462d2c3c8f7aad2609d0ef1875839c7 of the ocf-shellfuncs file, on
> CentOS 6.8, so it might not be an issue on later builds (if any) or
> different operating systems, but it would appear that the
> ocf_local_nodename function can have issues with certain configurations.
> Specially, I was debugging an issue I was having with a resource agent
> that I traced down to that function returning the FQDN of the machine
> rather than the actual node name, which in my case was a short name.
> 
> In looking at the code, I see that the function is looking for a
> pacemaker version greater than 1.1.8, in which case it uses crm_node
> (which works), otherwise it just uses "uname -n", which returns the FQDN
> (at least in my configuration). To get the current version, it runs the
> command:
> 
> local version=$(pacemakerd -$ | grep "Pacemaker .*" | awk '{ print $2 }')
> 
> Which on CentOS 6.8 returns (as of today, at least):
> 
> 1.1.14-8.el6_8.1
> 
> Unfortunately, when that string is passed to the ocf_version_cmp
> function to compare against 1.1.8, it returns 3, for "bad format", and
> so falls back to using "uname -n", even though the version *is* greater
> than 1.1.8, and crm_node would return the proper value.
> 
> Of course, if you always set up your cluster to use the FQDN of the
> servers as the node name, or more specifically always set them up such
> that the output of uname -n is the node name, then there isn't an issue
> other than perhaps a undetectably slight loss of efficiency. However, as
> I accidentally proved by doing otherwise, there is no actual requirement
> when setting up a cluster that the node names match uname -n (although
> perhaps it is considered "best practice"?), as long as they resolve to
> an IP.

Yes, it is considered a "best practice" (or at least a "safer
practice"), because issues like this tend to pop up periodically. :(

I'd recommend filing a bug against the resource-agents package, so the
version comparison can be made more intelligent.

> 
> I've worked around this in my installation by simply modifying the
> resource agent to call crm_node directly (since I know I am running on a
> version greater than 1.1.8), but I figured I might mention it, since I
> don't get any results when trying to google the issue.
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker

[Ken Gaillot]

On 11/01/2016 02:31 AM, Siwakoti, Ganesh wrote:
> Hi,
> 
> 
> i'm using CentOS release 6.8 (Final) as a KVM and i configured 3
> nodes(PM1.local,PM2.local and PM3.local), and using
> CMAN clustering. Resources running at two nodes as Active node then
> another one node is for Fail-over resource as a Standby node. Configured
> resources are groupA and groupB, 5 resources are configured in each
> resource group. groupA is Basically run at PM1.local and groupB at
> PM2.local. If fail any resource group it'll fail-over at PM3.local. On
> PM3.local can run only one Resource group at a time. groupA should not
> run at PM2.local and groupB should not run at PM1.local. groupA or
> groupB resources not move back on own nodes from PM3.local
> automatically, if need to move back on own nodes, can move by
> manually.someone pease help me.

Your configuration looks good to me. I don't think you need the -inf
rules for fail-counts; migration-threshold=1 handles that. Are you
seeing any problems?

> 
> 
> version information:
> 
> CentOS release 6.8 (Final) (KVM)
> 
> Pacemaker 1.1.14-8.el6_8.1
> 
> Cman Version: 6.2.0
> 
> 
> I want to be sure that my configuration will work properly or not.
> 
> 
> my Cluster configuration and crm_configuration are on below:
> 
> 
> cluster.conf:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  port="pm1.local"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  port="pm2.local"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  port="pm3.local"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> crm_configuration:
> 
> 
> node pm1.local \
> 
> attributes \
> 
> utilization capacity=1
> 
> node pm2.local \
> 
> attributes \
> 
> utilization capacity=1
> 
> node pm3.local \
> 
> attributes \
> 
> utilization capacity=1
> 
> primitive asteriskA asterisk \
> 
> params binary="/usr/sbin/asterisk" canary_binary=astcanary
> additional_parameters="-vvvg -I" config="/etc/asteri
> 
> sk_pm1/asterisk.conf" user=root group=root additional_parameters="-vvvg
> -I" realtime=true maxfiles=32768 \
> 
> meta migration-threshold=2 \
> 
> op monitor interval=20s start-delay=5s timeout=30s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive asteriskB asterisk \
> 
> params binary="/usr/sbin/asterisk" canary_binary=astcanary
> additional_parameters="-vvvg -I" config="/etc/asteri
> 
> sk_pm2/asterisk.conf" user=root group=root additional_parameters="-vvvg
> -I" realtime=true maxfiles=32768 \
> 
> meta migration-threshold=2 \
> 
> op monitor interval=20s start-delay=5s timeout=30s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive changeSrcIpA ocf:pacemaker:changeSrcIp \
> 
> params vip=192.168.12.215 mask=23 device=eth0 \
> 
> op start interval=0s timeout=0 \
> 
> op monitor interval=10s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive changeSrcIpB ocf:pacemaker:changeSrcIp \
> 
> params vip=192.168.12.216 mask=23 device=eth0 \
> 
> op start interval=0s timeout=0 \
> 
> op monitor interval=10s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive cronA lsb:crond \
> 
> meta migration-threshold=2 \
> 
> op monitor interval=20s start-delay=5s timeout=15s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive cronB lsb:crond \
> 
> meta migration-threshold=2 \
> 
> op monitor interval=20s start-delay=5s timeout=15s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive vip-local-checkA VIPcheck \
> 
> params target_ip=192.168.12.215 count=1 wait=5 \
> 
> op start interval=0s on-fail=restart timeout=60s \
> 
> op monitor interval=10s timeout=60s \
> 
> op stop interval=0s on-fail=ignore timeout=60s \
> 
> utilization capacity=1
> 
> primitive vip-local-checkB VIPcheck \
> 
> params target_ip=192.168.12.216 count=1 wait=5 \
> 
> op start interval=0s on-fail=restart timeout=60s \
> 
> op monitor interval=10s timeout=60s \
> 
> op stop interval=0s on-fail=ignore timeout=60s \
> 
> utilization capacity=1
> 
> primitive vip-localA IPaddr2 \
> 
> params ip=192.168.12.215 cidr_netmask=23 nic=eth0 iflabel=0
> broadcast=192.168.13.255 \
> 
> op start interval=0s timeout=20s \
> 
> op monitor interval=5s timeout=20s \
> 
> op stop interval=0s on-fail=ignore
> 
> primitive vip-localB IPaddr2 \
> 
> params ip=192.168.12.216 cidr_netmask=23 nic=eth0 

Re: [ClusterLabs] Live migration not working on shutdown

[Ken Gaillot]

On 10/26/2016 06:12 AM, Rainer Nerb wrote:
> Hello all,
> 
> we're currently testing a 2-node-cluster with 2 vms and live migration
> on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
> migration via ssh-method.
> 
> Live migration works, if we issue "pcs resource move ...", "pcs cluster
> standby", "pcs cluster stop" and even "systemctl rescue".
> The latter only worked, after adding the following additional
> dependencies to pacemaker.service and leaving the management of those
> services to systemd:
> 
>   * After/Requires=systemd-machined.service
>   * After/Requires=systemd-machine-id-commit.service
>   * After/Requires=remote-fs.target
>   * After/Requires=libvirtd.service
>   * After/Requires=iscsi.service
>   * After/Requires=iscsid.service
>   * After/Requires=sshd.service

This makes sense when clustered resources depend on services that aren't
themselves managed by the cluster. It's dependent on your situation, so
it's not something that pacemaker can solve generically.

You may already be aware, but the easiest way to add such requirements
is to put them in a systemd unit override, e.g.
/etc/systemd/system/pacemaker.service.d/dependencies.conf.

> When shutting down or rebooting migration fails and not even the regular
> shutdown of the vms succeeds. Systemd seems to tear down the vms by
> terminating something they depend on.
> 
> Is this a known issue? Did we miss any further dependencies?

There was a shutdown issue when using systemd-class cluster resources
(systemd: instead of ocf:), but I believe that was fixed in the package
you're using, and it's probably not relevant here anyway.

It does sound like there's another dependency, but I don't know what.

What log messages do you see on the failure?

> 
> Tia
> Rainer
> 
> 
> 
> IT Nerb GmbH
> Lessingstraße 8
> 85098 Großmehring
> 
> Telefon   :   +49 700 ITNERBGMBH
> Telefax   :   +49 8407 939 284
> email :   i...@it-nerb.de
> Internet  :   www.it-nerb.de 
> Geschäftsführer   :   Rainer Nerb
> Handelsregister   :   HRB 2592
> HR-Gericht:   Ingolstadt
> 
> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Which Pacemaker version is Best.

[Ken Gaillot]

On 11/01/2016 03:17 AM, Ganesh Siwakoti wrote:
> Hello,
> 
> I'm using CentOS release 6.8 (Final) (KVM) and I want to make failover
> cluster.i've 2 same nodes for clustering(both are KVM), so which version
> of Pacemaker and Cman is appropriate for me? 

Personally, I'd stick with the packages provided with CentOS. They are
identical to the RHEL packages, which are actively patched with
important bug fixes and are tested with the other software versions used
in the release.

If you prefer building code yourself, I'd generally recommend using the
latest official upstream release (1.1.15 at this point). But if you're
comfortable with that, I'd recommend upgrading to CentOS 7, which uses
corosync 2 instead of CMAN.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] packmaker: After migrate a resource, changing resource non-unique param lead to resoruce restart

[Ken Gaillot]

On 11/03/2016 02:01 AM, 李清硕 wrote:
> Hi everyone,
> I'm testing pacemaker resoruce live migration,  in a simple test
> environment with two virtual machines.
> The resource is something encapsulated kvm, when i perfrom migrate, for
> example, form node1 to node2, 
> i notice, the pacemaker invoke migrate_to, a one-off monitor on node1,
> like this:
>   (migrate_to:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>(migrate_to:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
>   (monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN  
>   (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 7
> invoke two one-off monitor on node2 like this:
>   (monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>  (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
>(monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>  (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
> When i perfrom crm resource operations resoruce_name(actually i don't
> what this is, maybe lead to the resource restart) , the output is: 
>rgavhcnP-iOgMD4-Bdh (ocf::heartbeat:fronvm) Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h122, call=41, rc=0,
> last-rc-change=Thu Nov  3 14:33:25 2016, exec=232ms): complete
>rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h122, call=41, rc=0,
> last-rc-change=Thu Nov  3 14:33:25 2016, exec=232ms): complete
>  
> rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): 
> Started:rgavhcnP-iOgMD4-Bdht_monitor_12
> (node=h122, call=42, rc=0, last-rc-change=Thu Nov  3 14:33:34
> 2016,exec=197ms):complete
>   rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h123, call=35, rc=7,
> last-rc-change=Thu Nov  3 14:34:25 2016, exec=213ms): complete
> The last i change resource params "config" (config is a param name),
> would lead to resource restart, which i expect reload, not restart.
> Is that a normal or a bug or  i do something wrong. if you need more
> detail, please tell me, Any hint will help

Pacemaker should reload a resource after a parameter change if:

* unique=0 in the parameter's metadata in the resource agent ()

* the resource agent advertises support for the reload operation in its
metadata ()

* no unique parameter was also changed at the same time

Are all those true in this case?

> Thank you！
> 
> The pacemaker config:
> node 1084759930: h122
> node 1084759931: h123
> primitive fence stonith:fence_agent \
> params pcmk_host_check=none action=reboot \
> op monitor interval=3600s \
> meta target-role=Started
> primitive rgavhcnP-iOgMD4-Bdht fronvm \
> params config="/mnt/m6dd63be9650f737da8c65d6caa00bc00"
> vm_name=rgavhcnP-iOgMD4-Bdht \
> op start timeout=600s interval=0 \
> op stop timeout=180s interval=0 \
> op migrate_to timeout=1800s interval=0 \
> op migrate_from timeout=60s interval=0 \
> op monitor interval=120s timeout=60s on-fail=ignore \
> meta target-role=Started allow-migrate=true
> clone fence-clone fence \
> meta target-role=Started
> location loc-fence-h122 fence-clone 100: h122
> location loc-fence-h123 fence-clone 100: h123
> location loc-paPupXaC-Ns5mnb-Xdfe-rgavhcnP-iOgMD4-Bdht
> rgavhcnP-iOgMD4-Bdht 1: h122
> location loc-tEJVJGWl-ifcTSp-Ta1v-rgavhcnP-iOgMD4-Bdht
> rgavhcnP-iOgMD4-Bdht 2: h123
> property cib-bootstrap-options: \
> stonith-enabled=true \
> stonith-action=reboot \
> stonith-timeout=120s \
> no-quorum-policy=stop \
> symmetric-cluster=false \
> crmd-transition-delay=5s \
> start-failure-is-fatal=FALSE \
> have-watchdog=false \
> dc-version=1.1.13-10.el7.centos-44eb2dd \
> cluster-infrastructure=corosync \
> cluster-name=OQSQoxu5XrsUcds8 \
> last-lrm-refresh=1478154860
> rsc_defaults rsc-options: \
> resource-stickiness=1024 \
> migration-threshold=3
> 
> The log:
>  Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:293   )info:
> cib_file_backup:Archived previous version as
> /var/lib/pacemaker/cib/cib-83.raw
> Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:423   )info:
> cib_file_write_with_digest:Wrote version 0.494.0 of the CIB to disk
> (digest: 97a2f450d62400802a2e04cf20ae010c)
> Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:442   )info:
> cib_file_write_with_digest:Reading cluster configuration file
> /var/lib/pacemaker/cib/cib.lTaDlA (digest:
> /var/lib/pacemaker/cib/cib.IlPpJD)
> Nov 03 14:34:34 [87769] h123cib: (  messages.c:239   )info:
> cib_process_ping:Reporting our current digest to h122:
> 01ebcac1c89e4493808c256c6c4b86a4 for 0.494.0 (0x2265720 0)
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1660  )info:
> cib_perform_op:Diff: --- 0.494.0 2
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1662  )info:
> cib_perform_op:Diff: +++ 0.494.1 (null)
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1728  )info:
> cib_perform_op:+  /cib:  @num_updates=1
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1684  )info:
> cib_perform_op:++
> 

[ClusterLabs] Fix in Pacemaker 1.1.15 retroactively assigned CVE-2016-7797

[Ken Gaillot]

Hello all,

Pacemaker 1.1.15, released earlier this year, contained a fix for a
potential denial-of-service vulnerability in pacemaker_remote. This
vulnerability has been retroactively assigned the Common Vulnerabilities
and Exposures identifier CVE-2016-7797.

This was mentioned in the 1.1.15 release notes, but is being raised
again for anyone interested in the CVE ID, such as distribution packagers.

Before Pacemaker 1.1.15, an unprivileged user able to attempt connection
to the IP address and port used for an active Pacemaker Remote
connection could trivially force the connection to drop. The
vulnerability only affects clusters with Pacemaker Remote nodes.

For details, see:

  http://bugs.clusterlabs.org/show_bug.cgi?id=5269

-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [SECURITY] CVE-2016-7035 - pacemaker - improper IPC guarding

[Ken Gaillot]

On 11/03/2016 06:03 AM, Jan Pokorný wrote:
> Following issue is being publicly disclosed today; more information
> regarding the release process will arrive later today and also this
> is an opportunity to announce http://clusterlabs.org/wiki/Security
> page that was intoduced to help keeping track of security issues
> (any fellow project is welcome to use that as well, Andrew or Ken
> can make and account on the wiki on your behalf).
> 
> It was discovered that at some not so uncommon circumstances, some
> pacemaker daemons could be talked to, via libqb-facilitated IPC, by
> unprivileged clients due to flawed authorization decision.  Depending
> on the capabilities of affected daemons, this might equip unauthorized
> user with local privilege escalation or up to cluster-wide remote
> execution of possibly arbitrary commands when such user happens to
> reside at standard or remote/guest cluster node, respectively.
> 
> The original vulnerability was introduced in an attempt to allow
> unprivileged IPC clients to clean up the file system materialized
> leftovers in case the server (otherwise responsible for the lifecycle
> of these files) crashes.  While the intended part of such behavior is
> now effectively voided (along with the unintended one), a best-effort
> fix to address this corner case systemically at libqb is coming along
> (https://github.com/ClusterLabs/libqb/pull/231).
> 
> Affected versions:  1.1.10-rc1 (2013-04-17) - 1.1.15 (2016-06-21)
> Impact: Important
> CVSSv3 ranking: 8.8 : AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H
> 
> Credits for independent findings, in chronological order:
>   Jan "poki" Pokorný, of Red Hat
>   Alain Moulle, of ATOS/BULL
> 
> 
> Patch for the issue, which is applicable on all affected versions:
> https://github.com/ClusterLabs/pacemaker/pull/1166/commits/5a20855d6054ebaae590c09262b328d957cc1fc2

More details:

A fix has been applied to Pacemaker's upstream master branch. Also, we
are starting the release process for 1.1.16 today, so the fix is
included in the 1.1 branch as well. It is additionally attached in patch
format to this message.

Anyone who has built one of the affected Pacemaker versions from source
is strongly encouraged to apply the patch, or rebuild from a current
branch. Popular distributions are expected to have patched packages
available soon (some already released today).

>From edba45825a11513661840fbc8c4b81f607fed18b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jan=20Pokorn=C3=BD?= 
Date: Tue, 23 Aug 2016 18:09:49 +0200
Subject: [PATCH] High: libcrmcommon: fix CVE-2016-7035 (improper IPC guarding)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

It was discovered that at some not so uncommon circumstances, some
pacemaker daemons could be talked to, via libqb-facilitated IPC, by
unprivileged clients due to flawed authorization decision.  Depending
on the capabilities of affected daemons, this might equip unauthorized
user with local privilege escalation or up to cluster-wide remote
execution of possibly arbitrary commands when such user happens to
reside at standard or remote/guest cluster node, respectively.

The original vulnerability was introduced in an attempt to allow
unprivileged IPC clients to clean up the file system materialized
leftovers in case the server (otherwise responsible for the lifecycle
of these files) crashes.  While the intended part of such behavior is
now effectively voided (along with the unintended one), a best-effort
fix to address this corner case systemically at libqb is coming along.

Affected versions:  1.1.10-rc1 (2013-04-17) - 1.1.15 (2016-06-21)
Impact: Important
CVSSv3 ranking: 8.8 : AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H

Credits for independent findings, in chronological order:
  Jan "poki" Pokorný, of Red Hat
  Alain Moulle, of ATOS/BULL
---
 lib/common/ipc.c | 14 +++---
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/lib/common/ipc.c b/lib/common/ipc.c
index 6d6d3cd..9f63dfe 100644
--- a/lib/common/ipc.c
+++ b/lib/common/ipc.c
@@ -293,7 +293,6 @@ crm_client_disconnect_all(qb_ipcs_service_t *service)
 crm_client_t *
 crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 {
-static uid_t uid_server = 0;
 static gid_t gid_cluster = 0;
 
 crm_client_t *client = NULL;
@@ -304,7 +303,6 @@ crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 }
 
 if (gid_cluster == 0) {
-uid_server = getuid();
 if(crm_user_lookup(CRM_DAEMON_USER, NULL, _cluster) < 0) {
 static bool have_error = FALSE;
 if(have_error == FALSE) {
@@ -314,16 +312,10 @@ crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 }
 }
 
-if(gid_cluster != 0 && gid_client != 0) {
-uid_t best_uid = -1; /* Passing -1 to chown(2) means don't change */
-
-if(uid_client == 0 || uid_server == 0) { 

Re: [ClusterLabs] Coming in 1.1.16: versioned resource parameters

[Ken Gaillot]

> With a new feature in the current master branch, which will be part of
> the next Pacemaker release, you will be able to specify different
> resource parameters to be used with different versions of a resource agent.

FYI, this feature is being held back to 1.1.17, due to a planned
reimplementation that adds functionality and handles rolling upgrades
better. It will still be available in the master branch, but not the 1.1
branch.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Special care needed when upgrading Pacemaker Remote nodes

[Ken Gaillot]

On 10/29/2016 07:55 AM, Ferenc Wágner wrote:
> Ken Gaillot <kgail...@redhat.com> writes:
> 
>> This spurred me to complete a long-planned overhaul of Pacemaker
>> Explained's "Upgrading" appendix:
>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/_upgrading.html
>>
>> Feedback is welcome.
> 
> Since you asked for it..:)
> 
> 1. Table D.1.: why does a rolling upgrade imply any service outage? Always?

Putting a node into standby will move all resources off it, so there's
at least a brief outage. Although I suppose if all of your resources are
live-migratable there wouldn't be. :) Probably worth a footnote to clarify.

> 2. Detach method: why use rsc_defaults instead of maintenance mode?

Good question. I think that would be equivalent (and easier). I'll
verify there are no special concerns and swap it.

> 
> 3. When do you think 1.1.16 will be released?  With approximately how
>much ABI incompatibility in the libraries?

I plan on starting the release cycle for 1.1.16 very soon. I expect all
the ABI changes will be backward-compatible. I think we had a few public
function additions but that's it.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Special care needed when upgrading Pacemaker Remote nodes

[Ken Gaillot]

On 10/31/2016 11:17 AM, Andrei Borzenkov wrote:
> 31.10.2016 17:15, Ken Gaillot пишет:
>> On 10/29/2016 07:55 AM, Ferenc Wágner wrote:
>>> Ken Gaillot <kgail...@redhat.com> writes:
>>>
>>>> This spurred me to complete a long-planned overhaul of Pacemaker
>>>> Explained's "Upgrading" appendix:
>>>>
>>>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/_upgrading.html
>>>>
>>>> Feedback is welcome.
>>>
>>> Since you asked for it..:)
>>>
>>> 1. Table D.1.: why does a rolling upgrade imply any service outage? Always?
>>
>> Putting a node into standby will move all resources off it, so there's
>> at least a brief outage. Although I suppose if all of your resources are
>> live-migratable there wouldn't be. :) Probably worth a footnote to clarify.
>>
> 
> 
> What about stopping pacemaker leaving resources running, upgrading
> pacemaker on the node, starting pacemaker? Is it technically possible?

Yes, that's the "detach-and-reattach" method described at the link above.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Special care needed when upgrading Pacemaker Remote nodes

[Ken Gaillot]

Hi all,

See the following wiki page if you are planning a rolling upgrade to
Pacemaker 1.1.15 from an earlier version, and your cluster includes
Pacemaker Remote nodes:

http://clusterlabs.org/wiki/Upgrading_to_Pacemaker_1.1.15_or_later_from_an_earlier_version


Details for anyone interested:

Pacemaker 1.1.15 included an increase in the LRMD protocol version,
which applies to communication between Pacemaker Remote nodes and the
full cluster nodes.

Because that was the first time that had ever happened, the impact on
rolling upgrades was not immediately noticed. I'm surprised we haven't
received any reports of problems since 1.1.15 was released.

Pacemaker 1.1.14 and earlier requires that the LRMD protocol version be
identical between the cluster and remote nodes. Any change means the
remote connection will fail. The workaround (detailed in the wiki page)
is to use constraints to tie older remote nodes to older cluster nodes,
and newer remote nodes to newer cluster nodes, until the rolling upgrade
is complete.

As of Pacemaker 1.1.15, the interpretation was modified so that older
remote nodes may connect to newer cluster nodes. So going forward, it
will be sufficient to upgrade all cluster nodes, then the remote nodes,
in a rolling upgrade. It is even possible to leave the remote nodes on
the older version, which might be useful, for example, if you are using
a remote node to run a legacy application in an older OS environment.

This spurred me to complete a long-planned overhaul of Pacemaker
Explained's "Upgrading" appendix:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/_upgrading.html

Feedback is welcome.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: OCFS2 on cLVM with node waiting for fencing timeout

[Ken Gaillot]

On 10/13/2016 03:36 AM, Ulrich Windl wrote:
> That's what I'm talking about: If 1 of 3 nodes is rebooting (or the cluster 
> is split-brain 1:2), the single node CANNOT continue due to lack of quorum, 
> while the remaining two nodes can. Is it still necessary to wait for 
> completion of stonith?

If the 2 nodes have working communication with the 1 node, then the 1
node will leave the cluster in an orderly way, and fencing will not be
involved. In that case, yes, quorum is used to prevent the 1 node from
starting services until it rejoins the cluster.

However, if the 2 nodes lose communication with the 1 node, they cannot
be sure it is functioning well enough to respect quorum. In this case,
they have to fence it. DLM has to wait for the fencing to succeed to be
sure the 1 node is not messing with shared resources.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Replicated PGSQL woes

[Ken Gaillot]

On 10/13/2016 12:04 PM, Israel Brewster wrote:
> Summary: Two-node cluster setup with latest pgsql resource agent.
> Postgresql starts initially, but failover never happens.
> 
> Details:
> 
> I'm trying to get a cluster set up with Postgresql 9.6 in a streaming
> replication using named slots scenario. I'm using the latest pgsql
> Resource Agent, which does appear to support the named replication slot
> feature, and I've pulled in the various utility functions the RA uses
> that weren't available in my base install, so the RA itself no longer
> gives me errors.
> 
> Setup: Two machines, centtest1 and centtest2. Both are running CentOS
> 6.8. Centtest1 has an IP of 10.211.55.100, and centtest2 has an IP of
> 10.211.55.101. The cluster is set up and functioning, with a shared
> virtual IP resource at 10.211.55.200. Postgresql has been set up and
> tested functioning properly on both nodes with centtest1 as the master
> and centtest2 as the streaming replica slave. 
> 
> I then set up the postgresql master/slave resource using the following
> commands:
> 
> pcs resource create pgsql_96 pgsql \
> pgctl="/usr/pgsql-9.6/bin/pg_ctl" \
> logfile="/var/log/pgsql/test2.log" \
> psql="/usr/pgsql-9.6/bin/psql" \
> pgdata="/pgsql96/data" \
> rep_mode="async" \
> repuser="postgres" \
> node_list="tcentest1.ravnalaska.net 
> centtest2.ravnalaska.net " \
> master_ip="10.211.55.200" \
> archive_cleanup_command="" \
> restart_on_promote="true" \
> replication_slot_name="centtest_2_slot" \
> monitor_user="postgres" \
> monitor_password="SuperSecret" \
> op start timeout="60s" interval="0s" on-fail="restart" \
> op monitor timeout="60s" interval="4s" on-fail="restart" \
> op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \
> op promote timeout="60s" interval="0s" on-fail="restart" \
> op demote timeout="60s" interval="0s" on-fail=stop \
> op stop timeout="60s" interval="0s" on-fail="block" \
> op notify timeout="60s" interval="0s";
> 
> pcs resource master msPostgresql pgsql_96 master-max=1 master-node-max=1
> clone-max=2 clone-node-max=1 notify=true
> 
> pcs constraint colocation add virtual_ip with Master msPostgresql INFINITY
> pcs constraint order promote msPostgresql then start virtual_ip
> symmetrical=false score=INFINITY
> pcs constraint order demote  msPostgresql then stop  virtual_ip
> symmetrical=false score=0
> 
> My preference would be that the master runs on centtest1, so I add the
> following constraint as well:
> 
> pcs constraint location --master msPostgresql prefers
> centtest1.ravnalaska.net=50
> 
> When I then start the cluster, I first see *both* machines come up as
> "slave", which I feel is somewhat odd, however the cluster software
> quickly figures things out and promotes centtest2 to master. I've tried

This is inherent to pacemaker's model of multistate resources. Instances
are always started in slave mode, and then promotion to master is a
separate step.

> this a dozen different times, and it *always* promotes centtest2 to
> master - even if I put INFINITY in for the location constraint.

Surprisingly, location constraints do not directly support limiting to
one role (your "--master" option is ignored, and I'm surprised it
doesn't give an error). To do what you want, you need a rule, like:

pcs constraint location msPostgresql rule \
   role=master score=50 \
   #uname eq centtest1.ravnalaska.net


> But whatever- this is a cluster, it doesn't really matter which node
> things are running on, as long as they are running. So the cluster is
> working - postgresql starts, the master process is on the same node as
> the IP, you can connect, etc, everything looks good. Obviously the next
> thing to try is failover - should the master node fail, the slave node
> should be promoted to master. So I try testing this by shutting down the
> cluster on the primary server: "pcs cluster stop"
> ...and nothing happens. The master shuts down (uncleanly, I might add -
> it leaves behind a lock file that prevents it from starting again until
> I manually remove said lock file), but the slave is never promoted to

This definitely needs to be corrected. What creates the lock file, and
how is that entity managed?

> master. Neither pcs status or crm_mon show any errors, but centtest1
> never becomes master.

I remember a situation where a resource agent improperly set master
scores, that led to no master being promoted. I don't remember the
details, though.

> 
> If instead of stoping the cluster on centtest2, I try to simply move the
> master using the command "pcs resource move --master msPostgresql", I
> first run into the aforementioned unclean shutdown issue (lock file left
> behind that has to be manually removed), and after removing the lock
> file, I wind up with *both* nodes being slaves, and no master node. "pcs
> resource clear --master msPostgresql" re-promotes centtest2 to master.
> 
> What it looks like is that for 

Re: [ClusterLabs] Colocation and ordering with live migration

[Ken Gaillot]

On 10/10/2016 07:36 AM, Pavel Levshin wrote:
> 10.10.2016 15:11, Klaus Wenninger:
>> On 10/10/2016 02:00 PM, Pavel Levshin wrote:
>>> 10.10.2016 14:32, Klaus Wenninger:
 Why are the order-constraints between libvirt & vms optional?
>>> If they were mandatory, then all the virtual machines would be
>>> restarted when libvirtd restarts. This is not desired nor needed. When
>>> this happens, the node is fenced because it is unable to restart VM in
>>> absence of working libvirtd.
>> Was guessing something like that ...
>> So let me reformulate my question:
>>Why does libvirtd have to be restarted?
>> If it is because of config-changes making it reloadable might be a
>> solution ...
>>
> 
> Right, config changes come to my mind first of all. But sometimes a
> service, including libvirtd, may fail unexpectedly. In this case I would
> prefer to restart it without disturbing VirtualDomains, which will fail
> eternally.

I think the mandatory colocation of VMs with libvirtd negates your goal.
If libvirtd stops, the VMs will have to stop anyway because they can't
be colocated with libvirtd. Making the colocation optional should fix that.

> The question is, why the cluster does not obey optional constraint, when
> both libvirtd and VM stop in a single transition?

If it truly is in the same transition, then it should be honored.

You have *mandatory* constraints for DLM -> CLVMd -> cluster-config ->
libvirtd, but only an *optional* constraint for libvirtd -> VMs.
Therefore, libvirtd will generally have to wait longer than the VMs to
be started.

It might help to add mandatory constraints for cluster-config -> VMs.
That way, they have the same requirements as libvirtd, and are more
likely to start in the same transition.

However I'm sure there are still problematic situations. What you want
is a simple idea, but a rather complex specification: "If rsc1 fails,
block any instances of this other RA on the same node."

It might be possible to come up with some node attribute magic to
enforce this. You'd need some custom RAs. I imagine something like one
RA that sets a node attribute, and another RA that checks it.

The setter would be grouped with libvirtd. Anytime that libvirtd starts,
the setter would set a node attribute on the local node. Anytime that
libvirtd stopped or failed, the setter would unset the attribute value.

The checker would simply monitor the attribute, and fail if the
attribute is unset. The group would have on-fail=block. So anytime the
the attribute was unset, the VM would not be started or stopped. (There
would be no constraints between the two groups -- the checker RA would
take the place of constraints.)

I haven't thought through all possible scenarios, but it seems feasible
to me.

> In my eyes, these services are bound by a HARD obvious colocation
> constraint: VirtualDomain should never ever be touched in absence of
> working libvirtd. Unfortunately, I cannot figure out a way to reflect
> this constraint in the cluster.
> 
> 
> -- 
> Pavel Levshin

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Unexpected Resource movement after failover

[Ken Gaillot]

On 10/14/2016 06:56 AM, Nikhil Utane wrote:
> Hi,
> 
> Thank you for the responses so far.
> I added reverse colocation as well. However seeing some other issue in
> resource movement that I am analyzing.
> 
> Thinking further on this, why doesn't "/a not with b" does not imply "b
> not with a"?/
> Coz wouldn't putting "b with a" violate "a not with b"?
> 
> Can someone confirm that colocation is required to be configured both ways?

The anti-colocation should only be defined one-way. Otherwise, you get a
dependency loop (as seen in logs you showed elsewhere).

The one-way constraint is enough to keep the resources apart. However,
the question is whether the cluster might move resources around
unnecessarily.

For example, "A not with B" means that the cluster will place B first,
then place A somewhere else. So, if B's node fails, can the cluster
decide that A's node is now the best place for B, and move A to a free
node, rather than simply start B on the free node?

The cluster does take dependencies into account when placing a resource,
so I would hope that wouldn't happen. But I'm not sure. Having some
stickiness might help, so that A has some preference against moving.

> -Thanks
> Nikhil
> 
> /
> /
> 
> On Fri, Oct 14, 2016 at 1:09 PM, Vladislav Bogdanov
> > wrote:
> 
> On October 14, 2016 10:13:17 AM GMT+03:00, Ulrich Windl
>  > wrote:
>  Nikhil Utane  > schrieb am 13.10.2016 um
> >16:43 in
> >Nachricht
> > 
> >:
> >> Ulrich,
> >>
> >> I have 4 resources only (not 5, nodes are 5). So then I only need 6
> >> constraints, right?
> >>
> >>  [,1]   [,2]   [,3]   [,4]   [,5]  [,6]
> >> [1,] "A"  "A"  "A""B"   "B""C"
> >> [2,] "B"  "C"  "D"   "C"  "D""D"
> >
> >Sorry for my confusion. As Andrei Borzenkovsaid in
> > 
> >
> >you probably have to add (A, B) _and_ (B, A)! Thinking about it, I
> >wonder whether an easier solution would be using "utilization": If
> >every node has one token to give, and every resource needs on token, no
> >two resources will run on one node. Sounds like an easier solution to
> >me.
> >
> >Regards,
> >Ulrich
> >
> >
> >>
> >> I understand that if I configure constraint of R1 with R2 score as
> >> -infinity, then the same applies for R2 with R1 score as -infinity
> >(don't
> >> have to configure it explicitly).
> >> I am not having a problem of multiple resources getting schedule on
> >the
> >> same node. Rather, one working resource is unnecessarily getting
> >relocated.
> >>
> >> -Thanks
> >> Nikhil
> >>
> >>
> >> On Thu, Oct 13, 2016 at 7:45 PM, Ulrich Windl <
> >> ulrich.wi...@rz.uni-regensburg.de
> > wrote:
> >>
> >>> Hi!
> >>>
> >>> Don't you need 10 constraints, excluding every possible pair of your
> >5
> >>> resources (named A-E here), like in this table (produced with R):
> >>>
> >>>  [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
> >>> [1,] "A"  "A"  "A"  "A"  "B"  "B"  "B"  "C"  "C"  "D"
> >>> [2,] "B"  "C"  "D"  "E"  "C"  "D"  "E"  "D"  "E"  "E"
> >>>
> >>> Ulrich
> >>>
> >>> >>> Nikhil Utane  > schrieb am 13.10.2016
> >um
> >>> 15:59 in
> >>> Nachricht
> >>>
> > 
> >:
> >>> > Hi,
> >>> >
> >>> > I have 5 nodes and 4 resources configured.
> >>> > I have configured constraint such that no two resources can be
> >>> co-located.
> >>> > I brought down a node (which happened to be DC). I was expecting
> >the
> >>> > resource on the failed node would be migrated to the 5th waiting
> >node
> >>> (that
> >>> > is not running any resource).
> >>> > However what happened was the failed node resource was started on
> >another
> >>> > active node (after stopping it's existing resource) and that
> >node's
> >>> > resource was moved to the waiting node.
> >>> >
> >>> > What could I be doing wrong?
> >>> >
> >>> >  >>> > name="have-watchdog"/>
> >>> >  >value="1.1.14-5a6cdd1"
> >>> > name="dc-version"/>
> >>> >  >>> value="corosync"
> >>> > 

Re: [ClusterLabs] Antw: Trying this question again re: arp_interval

[Ken Gaillot]

On 10/14/2016 03:36 AM, Ulrich Windl wrote:
 Eric Robinson  schrieb am 14.10.2016 um 09:15 in
> Nachricht
> 
> 
>> Does anyone know how many arp_intervals must pass without a reply before the 
>> bonding driver downs the primary NIC? Just one?
> 
> Maybe Bob Dylan (receiving the Nobel price for literature soon) would have 
> added it to "Blowing in the Wind":
> 
> "How many times must arp_interval pass, before it downs the primary NIC?" 

Otherwise, as in "Rainy Day Cluster # 12 & 35":

 Well, they'll stonith ya when you're startin' via virtd
 They'll stonith ya when you're usin' an APC
 They'll stonith ya when you're talkin' iSCSI
 They'll stonith ya when you're mountin' DRBD
 But I would not feel so all alone
 Every node must get stonithed.


> (Sorry I couldn't resist; I don't know the answer ("The answer my friend is 
> blowing in the wind"))"

Ditto, sorry ;-)

> Regards,
> Ulrich
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Error performing operation: Argument list too long

[Ken Gaillot]

On 12/05/2016 02:29 PM, Shane Lawrence wrote:
> I'm experiencing a strange issue with pacemaker. It is unable to check
> the status of a systemd resource.
> 
> systemctl shows that the service crashed:
> [root@xx ~]# systemctl status rsyslog
> ● rsyslog.service - System Logging Service
>Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
> vendor preset: enabled)
>Active: inactive (dead) since Mon 2016-12-05 07:41:11 UTC; 12h ago
>  Docs: man:rsyslogd(8)
>http://www.rsyslog.com/doc/
>  Main PID: 22703 (code=exited, status=0/SUCCESS)
> 
> Dec 02 21:41:41 xx...xx systemd[1]: Starting Cluster
> Controlled rsyslog...
> Dec 02 21:41:41 xx...xx systemd[1]: Started Cluster
> Controlled rsyslog.
> Dec 05 07:41:08 xx...xx systemd[1]: Stopping System
> Logging Service...
> Dec 05 07:41:11 xx...xx systemd[1]: Stopped System
> Logging Service.
> Dec 05 07:41:40 xx...xx systemd[1]: Stopped System
> Logging Service.
> 
> Attempting to view the status through Pacemaker shows:
> [root@xx ~]# crm_resource --force-check -V -r rsyslog
> Error performing operation: Argument list too long
> [root@xx ~]# pcs resource debug-monitor rsyslog --full
> Error performing operation: Argument list too long
> 
> The problem seems to be resolved (temporarily) by restarting corosync
> and then starting the cluster again.
> 
> Has anyone else experienced this?

That is odd behavior. You may want to open a bug report at
bugs.clusterlabs.org and attach your configuration and logs.

On Linux, the system error number for "Argument list too long" is the
same as the OCF monitor status "Not running", so I suspect that it's a
display issue rather than an actual error, but I'm not sure.

Then the question would just be why is rsyslog stopping.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: hawk - pacemaker remote

[Ken Gaillot]

On 12/12/2016 07:05 AM, philipp.achmuel...@arz.at wrote:
> 
>> Von: Ken Gaillot <kgail...@redhat.com>
>> An: users@clusterlabs.org
>> Datum: 02.12.2016 19:32
>> Betreff: Re: [ClusterLabs] hawk - pacemaker remote
>>
>> On 12/02/2016 07:38 AM, philipp.achmuel...@arz.at wrote:
>> > Hi,
>> >
>> > pacemaker remote nodes do not show up in hawk gui.
>> > regarding documentation, this should work - any hints to activate this?
>> >
>> > thank you!
>> >
>> > env: (SLES12.2)
>> > pacemaker-1.1.15-19.15.x86_64
>> > hawk2-2.0.0+git.1468428505.0135e38-24.32.x86_64
>>
>> I'm not familiar with hawk but based on earlier conversations with
>> someone who uses it, it may only see nodes in the  CIB section,
>> so try setting any permanent node attribute on the remote nodes to make
>> them show up
>>
> 
> tried several things, didn't get this working.
> Any examples how to configure this?
> Also how to configure for VirtualDomain with remote_node enabled

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html#_integrate_guest_into_cluster

You can set a node attribute on a remote node the same as any other
node, using crm_attribute (or your favorite higher-level CLI or GUI tool)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [cluster-lab] reboot standby node

[Ken Gaillot]

On 12/11/2016 04:19 PM, Omar Jaber wrote:
> Hi all ,
> 
> I have cluster contains three  nodes  with different sore  for location 
> constrain and  I have  group resource (it’s a service  exsists  in
> /etc/init.d/  folder)
> 
> Running  on the  node  the  have  the highest score  for   location
> constrain when I  try to  reboot one  of  the standby nodeI  see
> when the standby node become  up  the resource  stopped  in master node
> and restart againafter  I check the  pacemaker  status  I see the
> following error  :
> 
> "error: resource  'resource_name' is active on 2 nodes attempting
> recovery "  
> 
> Then I disables the  pcs  cluster  service in boot t time in standby
> node by run the command  "/_pcs_//cluster disable / " then I reboot the
> node  and I  see the resource  is started in standby node ( because  the
> resource  stored in /etc/init.d folder)
> 
> After that I  run the  pcs cluster  service  in standby node  and  I see
> the same  error is  generated  
> 
> "error: resource  'resource_name' is active on 2 nodes attempting
> recovery "
> 
>  
> 
> The problem  is  without reboot standby node this  problem not  happen
> for  example  
> 
> If  I stop pcs  cluster service  in standby , run the  resource  in
> standby node , then I start  pcs cluster
> 
> The error   "error: resource  'resource_name' is active on 2 nodes
> attempting recovery "   not  generated in this case.

Make sure your resource agent returns exit codes expected by Pacemaker:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-ocf-return-codes

In particular, if a monitor command returns 0 (OCF_SUCCESS), it means
the service is running.

When any node reboots, Pacemaker will "probe" the existing state of all
resources on it, by running a one-time monitor command. If the service
is not running, the command should return 7 (OCF_NOT_RUNNING).

So, I'm guessing that either the resource agent is wrongly returning 0
for monitor when the service is not actually running, or the node is
wrongly starting the service at boot, outside cluster control.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Random failure with clone of IPaddr2

[Ken Gaillot]

On 12/15/2016 12:37 PM, al...@amisw.com wrote:
> Hi,
> 
> I got some trouble since one week and can't find solution by myself. Any
> help will be really appreciated !
> I use corosync / pacemaker for 3 or 4 years and all works well, for
> failover or load-balancing.
> 
> I have shared ip between 3 servers, and need to remove one for upgrade.
> But after I remove the server from the cluster i got random fail to access
> to my shared ip. I think first that some packet want go to the old server.
> So I put it again in the cluster, can reach it, but random failure is
> still here :-/
> 
> My test is just a curl http://my_ip (or ssh same stuff, random failed to
> connect).
> A ping didn't loose any packet.
> I can reach each of the three servers, but sometime, the request hang, and
> got a timeout.
> I see via tcpdump the packet coming, and resend, but no one respond. How I
> can diagnostic this ?
> I think one request on five fail. But I didn't see any messages in
> firewall or /var/log/message, nothing, just like the switch choose to
> remove random packet. I didn't see any counter on network interface, check
> the iptable setting, recheck the log, recheck all firewall ... Where go
> these packets ??
> 
> I try with another new ip, and same problem append. I try ip on two
> differents subnets (10.xxx and external ip) and same stuff.
> 
> I have no problem with virtual ip in failover mode.
> 
> If someone has any clue ...

Seeing your configuration might help. Did you set globally-unique=true
and clone-node-max=3 on the clone? If not, the other nodes can't pick up
the lost node's share of requests.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] question about dc-deadtime

[Ken Gaillot]

On 12/15/2016 02:00 PM, Chris Walker wrote:
> Hello,
> 
> I have a quick question about dc-deadtime.  I believe that Digimer and
> others on this list might have already addressed this, but I want to
> make sure I'm not missing something.
> 
> If my understanding is correct, dc-deadtime sets the amount of time that
> must elapse before a cluster is formed (DC is elected, etc), regardless
> of which nodes have joined the cluster.  In other words, even if all
> nodes that are explicitly enumerated in the nodelist section have
> started Pacemaker, they will still wait dc-deadtime before forming a
> cluster.  
> 
> In my case, I have a two-node cluster on which I'd like to allow a
> pretty long time (~5 minutes) for both nodes to join before giving up on
> them.  However, if they both join quickly, I'd like to proceed to form a
> cluster immediately; I don't want to wait for the full five minutes to
> elapse before forming a cluster.  Further, if a node doesn't respond
> within five minutes, I want to fence it and start resources on the node
> that is up.

Pacemaker+corosync behaves as you describe by default.

dc-deadtime is how long to wait for an election to finish, but if the
election finishes sooner than that (i.e. a DC is elected), it stops
waiting. It doesn't even wait for all nodes, just a quorum.

Also, with startup-fencing=true (the default), any unseen nodes will be
fenced, and the remaining nodes will proceed to host resources. Of
course, it needs quorum for this, too.

With two nodes, quorum is handled specially, but that's a different topic.

> With Pacemaker/Heartbeat, the initdead parameter did exactly what I
> want, but I don't see any way to do this with Pacemaker/Corosync.  From
> reading other posts, it looks like people use an external agent to start
> HA daemons once nodes are up ... is this a correct understanding?
> 
> Thanks very much,
> Chris

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Random failure with clone of IPaddr2

[Ken Gaillot]

On 12/15/2016 02:02 PM, al...@amisw.com wrote:
>>
>> Seeing your configuration might help. Did you set globally-unique=true
>> and clone-node-max=3 on the clone? If not, the other nodes can't pick up
>> the lost node's share of requests.
> 
> Yes for both, I have globally-unique=true, and I change clone-node-max=3
> to clone-node-max=2, and now, as I come back to old configuration, I come
> back to clone-node-max=3
> 
> So now I have three node in the cluster.
> Here my config:
> 
> primitive ip_apache_localnet ocf:heartbeat:IPaddr2 params ip="10.0.0.99" 
> cidr_netmask="32" op monitor interval="30s"
> clone cl_ip_apache_localnet ip_apache_localnet \
> meta globally-unique="true" clone-max="3" clone-node-max="1"


^^^ Here you have clone-node-max="1", which will prevent surviving nodes
from picking up any failed node's share of requests. clone-max and
clone-node-max should both stay at 3, regardless of whether you are
intentionally taking down any node.

> target-role="Started" is-managed="true"
> 
>  sudo  /usr/sbin/iptables -L
> CLUSTERIP  all  --  anywhere 10.0.0.99  CLUSTERIP
> hashmode=sourceip-sourceport clustermac=A1:99:D6:EA:43:77 total_nodes=3
> local_node=2 hash_init=0
> 
> and check I have different local_node on each node.
> And just a question. Is the mac adress "normal" ? Doesn't need to begin
> with 01-00-5E ?

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: Antwort: Re: hawk - pacemaker remote

[Ken Gaillot]

On 12/13/2016 05:26 AM, philipp.achmuel...@arz.at wrote:
> 
>> Von: Kristoffer Grönlund 
>> An: philipp.achmuel...@arz.at, kgail...@redhat.com, Cluster Labs -
>> All topics related to open-source clustering welcomed
> 
>> Datum: 12.12.2016 16:13
>> Betreff: Re: [ClusterLabs] Antwort: Re:  hawk - pacemaker remote
>>
>> philipp.achmuel...@arz.at writes:
>>
>> >
>> > tried several things, didn't get this working.
>> > Any examples how to configure this?
>> > Also how to configure for VirtualDomain with remote_node enabled
>> >
>> > thank you!
>> >
>>
>> Without any details, it is difficult to help - what things did you try,
>> what does "not working" mean? Hawk can show remote nodes, but it only
>> shows them if they have entries in the nodes section of the
>> configuration (as Ken said).
>>
> 
> hi,
> 
> this is my current testsystem:
> 
> /Online: [ lnx0223a lnx0223b ]/
> /GuestOnline: [ vm-lnx0106a@lnx0223b vm-lnx0107a@lnx0223a ]/
> 
> /Full list of resources:/
> 
> /stonith_sbd (stonith:external/sbd): Started lnx0223a/
> / Clone Set: base-clone [base-group]/
> / Started: [ lnx0223a lnx0223b ]/
> / Stopped: [ vm-lnx0106a vm-lnx0107a ]/
> /FAKE1   (ocf::pacemaker:Dummy): Started lnx0223b/
> / Clone Set: post-clone [postfix-service]/
> / Started: [ vm-lnx0106a vm-lnx0107a ]/
> / Stopped: [ lnx0223a lnx0223b ]/
> /fence-lnx0106a  (stonith:external/libvirt): Started lnx0223b/
> /fence-lnx0107a  (stonith:external/libvirt): Started lnx0223a/
> /lnx0106a(ocf::heartbeat:VirtualDomain): Started lnx0223b/
> /lnx0107a(ocf::heartbeat:VirtualDomain): Started lnx0223a/
> /remote-paceip   (ocf::heartbeat:IPaddr):Started vm-lnx0106a/
> 
> node section shows:
> /.../
> //
> /  /
> //
> /   value="off"/>/
> //
> /  /
> /  /
> //
> /   value="off"/>/
> //
> /  /
> /  /
> //
> /   value="true"/>/
> //
> /  /
> /  /
> //
> /   value="true"/>/
> //
> /  /
> //
> /.../
> 
> 
> Hawk/Status still show remote nodes as "Offline"

Ah, it's good then ;)

Getting them to show up at all is the main goal. I'm going to guess it
pulls the status from the node state section; that has been maintained
for remote nodes only since Pacemaker 1.1.15, and there are still some
corner cases being addressed. Cluster nodes' state is learned
automatically from corosync, but remote nodes have no such mechanism, so
tracking state is much trickier.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Warning: handle_startup_fencing: Blind faith: not fencing unseen nodes

[Ken Gaillot]

On 12/14/2016 11:14 AM, Denis Gribkov wrote:
> Hi Everyone,
> 
> Our company have 15-nodes asynchronous cluster without actually
> configured FENCING/STONITH (as I think) features.
> 
> The DC node log getting tons of messages like in subject:
> 
> pengine:  warning: handle_startup_fencing:  Blind faith: not fencing
> unseen nodes

This is logged because you have set "startup-fencing: false".

It's logged as a warning because that setting is potentially dangerous:
a node that hasn't been seen by the cluster is not necessarily down --
it could be up and accessing shared resources, but unable to communicate
with the cluster. The only safe action is for the cluster to fence the node.

As of Pacemaker 1.1.16, the message will be logged only once. Before
Pacemaker 1.1.16, it is logged once per node, every time Pacemaker
checks the cluster state.

Of course, having "stonith-enabled: false" is also dangerous, because
fencing is the only way to recover from certain error conditions.

> the message repeated 15 times before next 15 messages:
> 
> pengine: info: determine_online_status: Node Node1 is online
> 
> ...
> 
> pengine: info: determine_online_status: Node Node15 is online
> 
> 
> The issue looks like similar to:
> 
> http://oss.clusterlabs.org/pipermail/pacemaker/2014-June/021995.html
> 
> but with own features.
> 
> 
> Our variables:
> 
> Oracle Linux Server release 6.8
> 
> Pacemaker 1.1.14-8.el6
> 
> Corosync Cluster Engine, version '1.4.7'
> 
> CMAN 3.0.12.1
> 
> 
> Cluster Properties:
>  cluster-infrastructure: cman
>  cluster-recheck-interval: 1
>  dc-version: 1.1.14-8.el6-70404b0
>  expected-quorum-votes: 3
>  have-watchdog: false
>  last-lrm-refresh: 1481444797
>  maintenance-mode: false
>  no-quorum-policy: ignore
>  startup-fencing: false
>  stonith-enabled: false
>  symmetric-cluster: false
> 
> 
> Example of /etc/cluster/cluster.conf:
> 
> 
>   
> 
>   
> 
>   
> 
>   
> 
>   
>   
> 
> 
> <...>
> 
> 
>   
> 
>   
> 
>   
>   
> 
>   
>   
>   
> 
>   
>   
> 
> 
>   
>   
> 
> 
> Example of /etc/corosync/corosync.conf:
> 
> compatibility: whitetank
> 
> totem {
> version: 2
> secauth: on
> threads: 4
> rrp_mode: active
> 
> interface {
> 
> member {
> memberaddr: PRIVATE_IP_1
> }
> 
> ...
> 
> member {
> memberaddr: PRIVATE_IP_15
> }
> 
> ringnumber: 0
> bindnetaddr: PRIVATE_NET_ADDR
> mcastaddr: 226.0.0.1
> mcastport: 5407
> ttl: 1
> }
> 
>interface {
> 
> member {
> memberaddr: PUBLIC_IP_1
> }
> ...
> 
> member {
> memberaddr: PUBLIC_IP_15
> }
> 
> ringnumber: 1
> bindnetaddr: PUBLIC_NET_ADDR
> mcastaddr: 224.0.0.1
> mcastport: 5405
> ttl: 1
> }
> 
> transport: udpu
> logging {
> fileline: off
> to_stderr: no
> to_logfile: yes
> logfile: /var/log/cluster/corosync.log
> logfile_priority: warning
> to_syslog: no
> debug: off
> timestamp: on
> logger_subsys {
>subsys: AMF
>debug: off
> }
> }
> 
> 
> Please let me know if you will need any other information.
> 
> Thank you.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] changing default cib.xml directory

[Ken Gaillot]

On 12/13/2016 09:57 AM, Christopher Harvey wrote:
> I was wondering if it is possible to tell pacemaker to store the cib.xml
> file in a specific directory. I looked at the code and searched the web
> a bit and haven't found anything. I just wanted to double check here in
> case I missed anything.
> 
> Thanks,
> Chris

Only when building the source code, with ./configure --localstatedir=DIR
which defaults to /var (pacemaker will always add /lib/pacemaker/cib to it)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: Antwort: Re: clone resource - pacemaker remote

[Ken Gaillot]

On 12/07/2016 06:26 AM, philipp.achmuel...@arz.at wrote:
>> Von: Ken Gaillot <kgail...@redhat.com>
>> An: philipp.achmuel...@arz.at, Cluster Labs - All topics related to
>> open-source clustering welcomed <users@clusterlabs.org>
>> Datum: 05.12.2016 17:38
>> Betreff: Re: Antwort: Re: [ClusterLabs] clone resource - pacemaker remote
>>
>> On 12/05/2016 09:20 AM, philipp.achmuel...@arz.at wrote:
>> > Ken Gaillot <kgail...@redhat.com> schrieb am 02.12.2016 19:27:09:
>> >
>> >> Von: Ken Gaillot <kgail...@redhat.com>
>> >> An: users@clusterlabs.org
>> >> Datum: 02.12.2016 19:32
>> >> Betreff: Re: [ClusterLabs] clone resource - pacemaker remote
>> >>
>> >> On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
>> >> > hi,
>> >> >
>> >> > what is best way to prevent clone resource trying to run on
> remote/guest
>> >> > nodes?
>> >>
>> >> location constraints with a negative score:
>> >>
>> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> >>
> Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on
>> >>
>> >>
>> >> you can even use a single constraint with a rule based on #kind ne
>> >> cluster, so you don't need a separate constraint for each node:
>> >>
>> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> >> Pacemaker_Explained/index.html#_node_attribute_expressions
>> >>
>> >>
>> >> alternatively, you can set symmetric-cluster=false and use positive
>> >> constraints for cluster nodes only
>> >>
>> >
>> > set constraint to single primitive, group, or on clone resource?
>> > are there any advantages/disadvantages using one of these methods?
>>
>> When a resource is cloned, you want to refer to the clone name in any
>> constraints, rather than the primitive name.
>>
>> For a group, it doesn't really matter, but it's simplest to use the
>> group name in constraints -- mainly that keeps you from accidentally
>> setting conflicting constraints on different members of the group. And
>> of course group members are automatically ordered/colocated with each
>> other, so you don't need individual constraints for that.
>>
> 
> set location constraint to group didn't work:
> 
> ERROR: error: unpack_location_tags:Constraint
> 'location-base-group': Invalid reference to 'base-group'

Maybe a syntax error in your command, or a bug in the tool you're using?
The CIB XML is fine with something like this:



> for clone it works like expected.
> but crm_mon is showing "disallowed" set as "stopped". is this "works as
> designed" or how to prevent this?

You asked it to :)

-r == show inactive resources

> 
> crm configure show
> ...
> location location-base-clone base-clone resource-discovery=never \
>rule -inf: #kind ne cluster
> ...
> 
> crm_mon -r
>  Clone Set: base-clone [base-group]
>  Started: [ lnx0223a lnx0223b ]
>  Stopped: [ vm-lnx0106a vm-lnx0107a ]
> 
>> >
>> >> >
>> >> > ...
>> >> > node 167873318: lnx0223a \
>> >> > attributes maintenance=off
>> >> > node 167873319: lnx0223b \
>> >> > attributes maintenance=off
>> >> > ...
>> >> > /primitive vm-lnx0107a VirtualDomain \/
>> >> > /params hypervisor="qemu:///system"
>> >> > config="/etc/kvm/lnx0107a.xml" \/
>> >> > /meta remote-node=lnx0107a238 \/
>> >> > /utilization cpu=1 hv_memory=4096/
>> >> > /primitive remote-lnx0106a ocf:pacemaker:remote \/
>> >> > /params server=xx.xx.xx.xx \/
>> >> > /meta target-role=Started/
>> >> > /group base-group dlm clvm vg1/
>> >> > /clone base-clone base-group \/
>> >> > /meta interleave=true target-role=Started/
>> >> > /.../
>> >> >
>> >> > /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
>> >> > operation dlm_start_0 on lnx0107a238/
>> >> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice:
> executing -
>> >> > rsc:dlm action:start call_id:7/
>> >> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice:
> finished -
>> >> > rsc:dlm action:st

Re: [ClusterLabs] Nodes see each other as OFFLINE - fence agent (fence_pcmk) may not be working properly on RHEL 6.5

[Ken Gaillot]

On 12/16/2016 07:46 AM, avinash shankar wrote:
> 
> Hello team,
> 
> I am a newbie in pacemaker and corosync cluster.
> I am facing trouble with fence_agent on RHEL 6.5
> I have installed pcs, pacemaker, corosync, cman on RHEL 6.5 on two
> virtual nodes (libvirt) cluster.
> SELINUX and firewall is completely disabled.
> 
> # yum list installed | egrep 'pacemaker|corosync|cman|fence'
> cman.x86_64  3.0.12.1-78.el6
> @rhel-ha-for-rhel-6-server-rpms
> corosync.x86_64  1.4.7-5.el6
> @rhel-ha-for-rhel-6-server-rpms
> corosynclib.x86_64   1.4.7-5.el6
> @rhel-ha-for-rhel-6-server-rpms
> fence-agents.x86_64  4.0.15-12.el6  
> @rhel-6-server-rpms   
> fence-virt.x86_640.2.3-19.el6   
> @rhel-ha-for-rhel-6-server-eus-rpms
> pacemaker.x86_64 1.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-cli.x86_64 1.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-cluster-libs.x86_641.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-libs.x86_641.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
>  
> 
> I bring up cluster using pcs cluster start --all
> also done pcs property set stonith-enabled=false

fence_pcmk simply tells CMAN to use pacemaker's fencing ... it can't
work if pacemaker's fencing is disabled.

> Below is the status
> ---
> # pcs status
> Cluster name: roamclus
> Last updated: Fri Dec 16 18:54:40 2016Last change: Fri Dec 16
> 17:44:50 2016 by root via cibadmin on cnode1
> Stack: cman
> Current DC: NONE
> 2 nodes and 2 resources configured
> 
> Online: [ cnode1 ]
> OFFLINE: [ cnode2 ]
> 
> Full list of resources:
> 
> PCSD Status:
>   cnode1: Online
>   cnode2: Online
> ---
> Same kind of output is observed on other node = cnode2
> So nodes see each other as OFFLINE.
> Expected result is Online: [ cnode1 cnode2 ]
> I did same packages installation on RHEL 6.8 and when I am starting the
> cluster,
> it shows both nodes ONLINE from each other.
> 
> I need to resolve this such that on RHEL 6.5 nodes when we start cluster
> by default
> both nodes should display each others status as online.
> --
> Below is the  /etc/cluster/cluster.conf
> 
> 
>   
>   
> 
>   
> 
>   
> 
>   
> 
> 
>   
> 
>   
> 
>   
> 
>   
>   
>   
> 
>   
>   
> 
> 
>   
> 
> --
> # cat /var/lib/pacemaker/cib/cib.xml
>  num_updates="0" admin_epoch="0" cib-last-written="Fri Dec 16 18:57:10
> 2016" update-origin="cnode1" update-client="cibadmin" update-user="root"
> have-quorum="1" dc-uuid="cnode1">
>   
> 
>   
>  name="have-watchdog" value="false"/>
>  value="1.1.14-8.el6_8.2-70404b0"/>
>  name="cluster-infrastructure" value="cman"/>
>  name="stonith-enabled" value="false"/>
>   
> 
> 
>   
>   
> 
> 
> 
>   
> 
> 
> /var/log/messages have below contents :
> 
> Dec 15 20:29:43 cnode2 kernel: DLM (built Oct 26 2016 10:26:08) installed
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Corosync Cluster
> Engine ('1.4.7'): started and ready to provide service.
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Corosync built-in
> features: nss dbus rdma snmp
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Successfully read
> config from /etc/cluster/cluster.conf
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Successfully parsed
> cman config
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] Initializing transport
> (UDP/IP Multicast).
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] Initializing
> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] The network interface
> [10.10.18.138] is now up.
> Dec 15 20:29:46 cnode2 corosync[2464]:   [QUORUM] Using quorum provider
> quorum_cman
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync cluster quorum service v0.1
> Dec 15 20:29:46 cnode2 corosync[2464]:   [CMAN  ] CMAN 3.0.12.1 (built
> Feb  1 2016 07:06:19) started
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync CMAN membership service 2.90
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> openais checkpoint service B.01.01
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync extended virtual synchrony service
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync configuration service
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync cluster closed process group service v1.01

Re: [ClusterLabs] New ClusterLabs logo unveiled :-)

[Ken Gaillot]

On 01/11/2017 03:54 AM, Kostiantyn Ponomarenko wrote:
> Nice logo!
> 
> http://wiki.clusterlabgs.org/ doesn't load for me.

Whoops, typo! There should be no "g" in there:

   http://wiki.clusterlabs.org/

> I also have a question which bothers me for a long time. Not a
> significant one, but anyways ... 
> I have seen a lot "Linux-HA" name around. But it seems that the name it
> not used anymore for this particular stack of HA software.
> So I wonder, what it the correct name for this stack? 
> 
> Thank you,
> Kostia

I don't know if there's an "official" name, but I've been calling it the
"ClusterLabs stack".

> On Mon, Jan 2, 2017 at 11:35 AM, Kristoffer Grönlund <kgronl...@suse.com
> <mailto:kgronl...@suse.com>> wrote:
> 
> Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>> writes:
> 
> > Hi all,
> >
> > ClusterLabs is happy to unveil its new logo! Many thanks to the
> > designer, Kristoffer Grönlund <kgronl...@suse.com 
> <mailto:kgronl...@suse.com>>, who graciously
> > donated the clever approach.
> >
> > You can see it on our GitHub page:
> >
> >   https://github.com/ClusterLabs
> >
> > It is also now the site icon for www.clusterlabs.org 
> <http://www.clusterlabs.org> and
> > wiki.clusterlabgs.org <http://wiki.clusterlabgs.org>. Your browser
> might have cached the old version,
> > so you might not see the new one immediately, but you can see it by
> > going straight to the links and reloading:
> >
> >   http://clusterlabs.org/favicon.ico 
> <http://clusterlabs.org/favicon.ico>
> >   http://clusterlabs.org/apple-touch-icon.png
> <http://clusterlabs.org/apple-touch-icon.png>
> >
> > It is also on the wiki banner, though the banner will need some tweaking
> > to make the best use of it. You might not see it there immediately due
> > to browser caching and DNS resolver caching (the wiki IP changed
> > recently as part of an OS upgrade), but it's there. :-)
> >
> > Wishing everyone a happy holiday season,
> 
> Thanks for using my logo! Nice holiday surprise :)
> 
> Cheers,
> Kristoffer
> 
> > --
> > Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>>
> >
> > ___
> > Users mailing list: Users@clusterlabs.org <mailto:Users@clusterlabs.org>
> > http://lists.clusterlabs.org/mailman/listinfo/users
> <http://lists.clusterlabs.org/mailman/listinfo/users>
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
> > Bugs: http://bugs.clusterlabs.org
> 
> --
> // Kristoffer Grönlund
> // kgronl...@suse.com <mailto:kgronl...@suse.com>

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] question about dc-deadtime

[Ken Gaillot]

On 01/10/2017 11:59 AM, Chris Walker wrote:
> 
> 
> On Mon, Jan 9, 2017 at 6:55 PM, Andrew Beekhof <abeek...@redhat.com
> <mailto:abeek...@redhat.com>> wrote:
> 
> On Fri, Dec 16, 2016 at 8:52 AM, Chris Walker
> <christopher.wal...@gmail.com <mailto:christopher.wal...@gmail.com>>
> wrote:
> > Thanks for your response Ken.  I'm puzzled ... in my case node remain
> > UNCLEAN (offline) until dc-deadtime expires, even when both nodes are 
> up and
> > corosync is quorate.
> 
> I'm guessing you're starting both nodes at the same time?
> 
> 
> The nodes power on at the same time, but hardware discovery can vary by
> minutes.
> 
>  
> 
> The behaviour you're seeing is arguably a hangover from the multicast
> days (in which case corosync wouldn't have had a node list).
> 
> 
> That makes sense. 
> 
> 
> But since that's not the common case anymore, we could probably
> shortcut the timeout if we know the complete node list and see that
> they are all online.
> 
> 
> That would be ideal.  It's easy enough to work around this in systemd,
> but it seems like the HA stack should be the authority on node status.

I've open a feature request:
http://bugs.clusterlabs.org/show_bug.cgi?id=5310

FYI the priority list is long at this point, so no idea when it might be
addressed.

> Thanks!
> Chris
> 
> 
> >
> > I see the following from crmd when I have dc-deadtime=2min
> >
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: Quorum acquired
> > Dec 15 21:34:33 max04 crmd[13791]:   notice:
> pcmk_quorum_notification: Node
> > max04[2886730248] - state is now member (was (null))
> > Dec 15 21:34:33 max04 crmd[13791]:   notice:
> pcmk_quorum_notification: Node
> > (null)[2886730249] - state is now member (was (null))
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: Notifications disabled
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: The local CRM is
> operational
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: State transition
> S_STARTING ->
> > S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
> > ...
> > Dec 15 21:36:33 max05 crmd[10365]:  warning: FSA: Input
> I_DC_TIMEOUT from
> > crm_timer_popped() received in state S_PENDING
> > Dec 15 21:36:33 max05 crmd[10365]:   notice: State transition
> S_ELECTION ->
> > S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
> > origin=election_timeout_popped ]
> > Dec 15 21:36:33 max05 crmd[10365]:  warning: FSA: Input
> I_ELECTION_DC from
> > do_election_check() received in state S_INTEGRATION
> > Dec 15 21:36:33 max05 crmd[10365]:   notice: Notifications disabled
> > Dec 15 21:36:33 max04 crmd[13791]:   notice: State transition
> S_PENDING ->
> > S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
> > origin=do_cl_join_finalize_respond ]
> >
> > only after this do the nodes transition to Online.  This is using the
> > vanilla RHEL7.2 cluster stack and the following options:
> >
> > property cib-bootstrap-options: \
> > no-quorum-policy=ignore \
> > default-action-timeout=120s \
> > pe-warn-series-max=1500 \
> > pe-input-series-max=1500 \
> > pe-error-series-max=1500 \
> > stonith-action=poweroff \
> > stonith-timeout=900 \
> > dc-deadtime=2min \
> > maintenance-mode=false \
> > have-watchdog=false \
> > dc-version=1.1.13-10.el7-44eb2dd \
> > cluster-infrastructure=corosync
> >
> > Thanks again,
> > Chris
> >
> > On Thu, Dec 15, 2016 at 3:26 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> >>
> >> On 12/15/2016 02:00 PM, Chris Walker wrote:
> >> > Hello,
> >> >
> >> > I have a quick question about dc-deadtime.  I believe that
> Digimer and
> >> > others on this list might have already addressed this, but I
> want to
> >> > make sure I'm not missing something.
> >> >
> >> > If my understanding is correct, dc-deadtime sets the amount of
> time that
> >> > must elapse before a cluster is formed (DC is elected, etc),
> regardless
> >> > of which nodes have joined the cluster.  In other words, even
> if all
> &

Re: [ClusterLabs] No match for shutdown action on

[Ken Gaillot]

On 01/10/2017 11:38 AM, Denis Gribkov wrote:
> Hi Everyone,
> 
> When I run:
> 
> # pcs resource cleanup resource_name
> 
> I'm getting a block of messages in log on current DC node:
> 
> Jan 10 18:12:13 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:13 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node9
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node10
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node11
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node12
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node13
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node14
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node15
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node16
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node1
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node1
> Jan 10 18:12:23 node1 cib[21630]:  warning: A-Sync reply to crmd failed:
> No message of desired type
> 
> At the same time other nodes didn't getting these messages.
> 
> Does anybody know why this issue happening in such cases and how
> possible to fix it?
> 
> Cluster Properties:
>  cluster-infrastructure: cman
>  cluster-recheck-interval: 5min
>  dc-version: 1.1.14-8.el6-70404b0
>  expected-quorum-votes: 3
>  have-watchdog: false
>  last-lrm-refresh: 1484068350
>  maintenance-mode: false
>  no-quorum-policy: ignore
>  pe-error-series-max: 1000
>  pe-input-series-max: 1000
>  pe-warn-series-max: 1000
>  stonith-action: reboot
>  stonith-enabled: false
>  symmetric-cluster: false
> 
> Thank you.

The message is harmless and can be ignored. It only shows up on the node
that is DC.

The "fix" is to upgrade. :-) In later versions, the message was changed
to the more accurate "No reason to expect node  to be down", and it
is now only printed if the node was actually lost.

In the version you have, the message was printed whenever the cluster
checked to see if any known event would have brought down a node,
regardless of whether there was any actual problem. If there is an
actual problem, there will be other messages about that (e.g. node lost
or fenced).

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] simple setup and resources on different nodes??

[Ken Gaillot]

On 01/11/2017 10:10 AM, lejeczek wrote:
> hi eveyone,
> I have a simple, test setup, like this:
> 
> $ pcs status
> Cluster name: test_cluster
> WARNING: corosync and pacemaker node names do not match (IPs used in
> setup?)
> Stack: corosync
> Current DC: work2.whale.private (version 1.1.15-11.el7_3.2-e174ec8) -
> partition with quorum
> Last updated: Wed Jan 11 16:02:15 2017Last change: Wed Jan 11
> 15:53:36 2017 by root via crm_attribute on work1.whale.private
> 
> 3 nodes and 3 resources configured
> 
> Online: [ work1.whale.private work2.whale.private work3.whale.private ]
> 
> Full list of resources:
> 
>  Fencing(stonith:fence_xvm):Started work1.whale.private
>  virtualIP(ocf::heartbeat:IPaddr2):Started work2.whale.private
>  Website(ocf::heartbeat:apache):Started work3.whale.private
> 
> Daemon Status:
>   corosync: active/enabled
>   pacemaker: active/enabled
>   pcsd: active/enabled
> 
> 
> but it does not make sense (bare in mind a beginner talks) right?
> How come pcs decided to have VirtualIP and Website are started on
> different nodes?
> And it actually did, virtuall IP pings but httpd is not there, indeed
> running on another node.
> What am I missing?
> 
> many thanks,
> L.

You may want to see "Clusters From Scratch":

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html

especially sections 6.4-6.6

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Deleting a variable

[Ken Gaillot]

On 12/01/2016 01:15 AM, Ulrich Windl wrote:
>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 30.11.2016 um 21:39 in 
>>>> Nachricht
> <62cb811f-4396-ff36-ec03-67000b4ed...@redhat.com>:
> 
> [...]
>> Once set, attributes are not truly deleted -- only their values are
>> cleared. And --delay has no effect with --update if the attribute
>> already exists, which is what you see above.
> 
> Is there a difference between a "deleted" variable and a defined variable 
> that has an empty string as value? I feel there should be!
> 
> [...]
> 
> Ulrich

Not currently


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 12/01/2016 06:04 AM, Kostiantyn Ponomarenko wrote:
> OK, now I see. I still have a few questions.
> 1. Is there a good reason to not remove the attribute totally if it
> is "deleted"?

We do this for two reasons:

1. You can "delete" an attribute for just one node, while leaving it on
other nodes. Attrd needs to remember the attribute as a whole, and just
delete the value for the one node. Even deleting an attribute on all
nodes is done by separate deletes for each node, so this matters even then.

2. Information about the attribute needs to continue to exist in order
to reliably process changes from different nodes. I forget the exact
details, but I remember looking into it before.

These limitations could possibly be addressed by keeping the attribute
but setting a "deleted" flag, and changing how those would be reported,
but it's not on the radar right now.

> 2. Does "attrd_updater" sets attributes to "status" configuration
> section only?

Yes (when using corosync 2).

crm_attribute modifies the CIB directly, so there is no way to batch its
changes with a delay.

> 3. Do I need to modify/set "--delay" to 0 before removing or
> changing the attribute? Because now I see that previously set delay
> works when I delete the attribute (--delete).

It depends on what you want. The point of the delay is to write changes
out to the CIB only every so often, to save disk I/O. If you're
potentially changing attribute values several times per second, this
would be a huge performance boost. The delay affects all changes,
including deletes.

If you want a specific change to be written to the CIB immediately, then
yes, you have to set the delay to 0. You can either change the delay
beforehand with attrd_updater --update-delay or change the delay and
value together with --update-both.

> 4. Does a delay set only one time work until it's unset (set to 0)?

Yes

> Thank you,
> Kostia
> 
> On Wed, Nov 30, 2016 at 10:39 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/30/2016 11:31 AM, Kostiantyn Ponomarenko wrote:
> > Hi Ken,
> >
> > I didn't look into the logs, but I experimented with it for a while.
> > Here is what I found.
> >
> > It worked for you because this attribute - "my-attr" - has not ever been
> > set before in that cluster.
> >
> > So if you set an attribute, then remove it, and then set it with
> > "--delay", like:
> >
> > # attrd_updater -N node-0 -n my-attr --update false --delay 20
> >
> > , this delay (dampening) won't work.
> 
> Once set, attributes are not truly deleted -- only their values are
> cleared. And --delay has no effect with --update if the attribute
> already exists, which is what you see above.
> 
> To set a delay on an already existing attribute, you have to use
> attrd_updater --update-delay or --update-both.
> 
>     > Moreover, when you delete this attribute the actual remove will be
> > delayed by that "--delay" which was used when the attribute was set.
> >
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 29, 2016 at 1:08 AM, Ken Gaillot <kgail...@redhat.com 
> <mailto:kgail...@redhat.com>
> > <mailto:kgail...@redhat.com <mailto:kgail...@redhat.com>>> wrote:
> >
> > On 11/24/2016 05:24 AM, Kostiantyn Ponomarenko wrote:
> > > Attribute dampening doesn't work for me also.
> > > To test that I have a script:
> > >
> > > attrd_updater -N node-0 -n my-attr --update false --delay 20
> > > sleep 3
> > > attrd_updater -N node-0 -n my-attr --update true
> > > sleep 7
> > > attrd_updater -N node-1 -n my-attr --update true
> >
> > This sequence works for me -- the attributes are not written
> to the live
> > CIB until the end of the delay, when both are written at the
> same time.
> >
> > The remaining issue must be with the policy engine. You could
> look at
> > the detail log on the DC when these changes were made; you
> should see
> > info-level messages with the CIB change with both values
> together (lines
> > with "cib_perform_op:   ++" and the attribute values), then
> "Transition
> > aborted" with "Transient attribute change", then a bunch of
> "pengine:"
> > lines saying what the clust

Re: [ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

On 12/01/2016 10:13 AM, Jehan-Guillaume de Rorthais wrote:
> On Wed, 30 Nov 2016 14:05:19 -0600
> Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> ClusterLabs is proud to announce the latest release of the Pacemaker
>> cluster resource manager, version 1.1.15.
> 
> 1.1.6 I guess ;)

Whoops!

Well now I don't feel so bad since the correction is wrong too ;)

> But congrats anyway !
> 
>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>> properly passed to multistate resources with notification enabled. This
>> has been fixed. To help resource agents detect when the fix is
>> available, the CRM feature set has been incremented. (Whenever the
>> feature set changes, mixed-version clusters are supported only during
>> rolling upgrades -- nodes with an older version will not be allowed to
>> rejoin once they shut down.)
> 
>   * how could we get the "CRM feature set" version from the RA?
>   * when this "CRM feature set" has been introduced in Pacemaker?
> 
> Regards,
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/30/2016 11:31 AM, Kostiantyn Ponomarenko wrote:
> Hi Ken,
> 
> I didn't look into the logs, but I experimented with it for a while.
> Here is what I found.
> 
> It worked for you because this attribute - "my-attr" - has not ever been
> set before in that cluster.
> 
> So if you set an attribute, then remove it, and then set it with
> "--delay", like:
> 
> # attrd_updater -N node-0 -n my-attr --update false --delay 20
> 
> , this delay (dampening) won't work.

Once set, attributes are not truly deleted -- only their values are
cleared. And --delay has no effect with --update if the attribute
already exists, which is what you see above.

To set a delay on an already existing attribute, you have to use
attrd_updater --update-delay or --update-both.

> Moreover, when you delete this attribute the actual remove will be
> delayed by that "--delay" which was used when the attribute was set.
> 
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 29, 2016 at 1:08 AM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/24/2016 05:24 AM, Kostiantyn Ponomarenko wrote:
> > Attribute dampening doesn't work for me also.
> > To test that I have a script:
> >
> > attrd_updater -N node-0 -n my-attr --update false --delay 20
> > sleep 3
> > attrd_updater -N node-0 -n my-attr --update true
> > sleep 7
> > attrd_updater -N node-1 -n my-attr --update true
> 
> This sequence works for me -- the attributes are not written to the live
> CIB until the end of the delay, when both are written at the same time.
> 
> The remaining issue must be with the policy engine. You could look at
> the detail log on the DC when these changes were made; you should see
> info-level messages with the CIB change with both values together (lines
> with "cib_perform_op:   ++" and the attribute values), then "Transition
> aborted" with "Transient attribute change", then a bunch of "pengine:"
> lines saying what the cluster wants to do with each resource.
> 
> There should be some information about the scores used to place the
> resources.
> 
> >
> > All my resources have this rule in Pacemaker config:
> >
> > crm configure location res1-location-rule res1 \
> > rule 0: my-attr eq true \
> > rule -inf: my-attr ne true
> >
> > On a working two-node cluster I remove "my-attr" from both nodes.
> > Then run my script. And all resources start on node-0.
> > Am I doing something wrong?
> > Or maybe my understanding of an attribute dampening is not correct?
> >
> > My Pacemaker version is 1.1.13. (heh, not the last one, but it is what
> > it is ...)
> >
> > Thank you,
> > Kostia
> >
> > On Wed, Nov 23, 2016 at 7:27 PM, Kostiantyn Ponomarenko
> > <konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>
> > <mailto:konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>>> wrote:
> >
> > Maybe I am doing something wrong, but I cannot set "status" section
> > node attributes to a shadow cib, cluster applies them immediately.
> > To try it out I do in a console:
> >
> > crm_shadow --create test
> > crm_attribute --type nodes --node node-0 --name my-attribute
> > --update 1 --lifetime=reboot
> >
> > And this attribute is set to the live cluster configuration 
> immediately.
> > What am I doing wrong?
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 22, 2016 at 11:33 PM, Kostiantyn Ponomarenko
> > <konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>
> > <mailto:konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>>> wrote:
> >
> > Ken,
> > Thank you for the explanation.
> > I will try this low-level way of shadow cib creation tomorrow.
> > PS: I will sleep much better with this excellent news/idea. =)
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 22, 2016 at 10:53 PM, Ken Gaillot
> > <kgail...@redhat.com <mailto:kgail...@redhat.com>
> 

[ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

ClusterLabs is proud to announce the latest release of the Pacemaker
cluster resource manager, version 1.1.15. The source code is available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16
The most significant enhancements in this release are:

* rsc-pattern may now be used instead of rsc in location constraints, to
allow a single location constraint to apply to all resources whose names
match a regular expression. Sed-like %0 - %9 backreferences let
submatches be used in node attribute names in rules.

* The new ocf:pacemaker:attribute resource agent sets a node attribute
according to whether the resource is running or stopped. This may be
useful in combination with attribute-based rules to model dependencies
that simple constraints can't handle.

* Pacemaker's existing "node health" feature allows resources to move
off nodes that become unhealthy. Now, when using
node-health-strategy=progressive, a new cluster property
node-health-base will be used as the initial health score of newly
joined nodes (defaulting to 0, which is the previous behavior). This
allows a node to be treated as "healthy" even if it has some "yellow"
health attributes, which can be useful to allow clones to run on such nodes.

* Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
properly passed to multistate resources with notification enabled. This
has been fixed. To help resource agents detect when the fix is
available, the CRM feature set has been incremented. (Whenever the
feature set changes, mixed-version clusters are supported only during
rolling upgrades -- nodes with an older version will not be allowed to
rejoin once they shut down.)

* Watchdog-based fencing using sbd now works better on remote nodes.
This capability still likely has some limitations, however.

* The build process now takes advantage of various compiler features
(RELRO, PIE, as-needed linking, etc.) that enhance security and start-up
performance. See the "Hardening flags" comments in the configure.ac file
for more details.

* Python 3 compatibility: The Pacemaker project now targets
compatibility with both python 2 (versions 2.6 and later) and python 3
(versions 3.2 and later). All of the project's python code now meets
this target, with the exception of CTS, which is still python 2 only.

* The Pacemaker coding guidelines have been replaced by a more
comprehensive addition to the documentation set, "Pacemaker
Development". It is intended for developers working on the Pacemaker
code base itself, rather than external code such as resource agents. A
copy is viewable at
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Development/

As usual, the release includes many bugfixes, including a fix for a
serious security vulnerability (CVE-2016-7035). For a more detailed list
of changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Bin Liu, Christian Schneider, Christoph Berg,
David Shane Holden, Ferenc Wágner, Yan Gao, Hideo Yamauchi, Jan Pokorný,
Ken Gaillot, Klaus Wenninger, Kostiantyn Ponomarenko, Kristoffer
Grönlund, Lars Ellenberg, Masatake Yamato, Michal Koutný, Nakahira
Kazutomo, Nate Clark, Nishanth Aravamudan, Oyvind Albrigtsen, Ruben
Kerkhof, Tim Bishop, Vladislav Bogdanov and Yusuke Iida. Apologies if I
have overlooked anyone.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] need some help with failing resources

[Ken Gaillot]

On 12/05/2016 09:30 AM, Darko Gavrilovic wrote:
> On 12/5/2016 10:17 AM, Ken Gaillot wrote:
>> On 12/03/2016 05:19 AM, Darko Gavrilovic wrote:
>>> Here is the output for that resource.. edited
>>>
>>> primitive svc-mysql ocf:heartbeat:mysql \
>>> params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf"
>>> datadir="/var/lib/mysql" user="mysql" group="mysql"
>>> log="/var/log/mysqld.log" pid="/var/run/mysqld/mysqld.pid"
>>> socket="/var/lib/mysql/mysql.sock" test_table="***" test_user="***"
>>> test_passwd="" \
>>> op monitor interval="30s" timeout="60s" OCF_CHECK_LEVEL="5" \
>>> op start interval="0" timeout="120s" \
>>> op stop interval="0" timeout="120s" \
>>> meta target-role="Started" migration-threshold="2"
>>>
>>> ...skipping
>>> order mysql-before-httpd inf: svc-mysql:start svc-httpd:start
>>> order mysql-before-ssh inf: svc-mysql:start svc-ssh:start
>>> property $id="cib-bootstrap-options" \
>>> dc-version="1.0.6-f709c638237cdff7556cb6ab615f32826c0f8c06" \
>>> cluster-infrastructure="openais" \
>>> expected-quorum-votes="2" \
>>> last-lrm-refresh="1480762389" \
>>> no-quorum-policy="ignore" \
>>> stonith-enabled="true" \
>>> ms-drbd0="Master"
>>>
>>>
>>> dg
>>>
>>>
>>> On 12/3/2016 1:25 AM, Kostiantyn Ponomarenko wrote:
>>>> I assume that you are using crmsh.
>>>> If so, I suggest to post an output from "crm configure show" command
>>>> here.
>>>>
>>>> Thank you,
>>>> Kostia
>>>>
>>>> On Sat, Dec 3, 2016 at 5:54 AM, Darko Gavrilovic
>>>> <da...@chass.utoronto.ca <mailto:da...@chass.utoronto.ca>> wrote:
>>>>
>>>> Hello, I have a two node cluster running that seems to be
>>>> failing to
>>>> start resources.
>>>>
>>>>  Resource Group: services
>>>>  svc-mysql  (ocf::heartbeat:mysql) Stopped
>>>>  svc-httpd  (ocf::heartbeat:apache) Stopped
>>>>  svc-ssh(lsb:sshd-virt) Stopped
>>>>  svc-tomcat6(lsb:tomcat6) Stopped
>>>>  svc-plone  (lsb:plone) Stopped
>>>>  svc-bacula (lsb:bacula-fd-virt) Stopped
>>>>
>>>> When I run crm resource start services the service group does not
>>>> start.
>>>>
>>>> I also tried starting the first resource in the group.
>>>> crm resource start svc-mysql
>>>>
>>>> It does not start either.
>>>>
>>>> The error I am seeing is:
>>>> Dec  2 21:59:43  pengine: [25829]: WARN: native_color: Resource
>>>> svc-mysql cannot run anywhere
>>>> Dec  2 22:00:26  pengine: [25829]: WARN: native_color: Resource
>>>> svc-mysql cannot run anywhere
>>
>> The most common reasons for the above message are:
>>
>> * Location or order constraints don't leave any place for the resource
>> to run
>>
>> * The resource has failed the maximum number of times on all nodes.
>> (Does "crm_mon" show any failures?)
> 
> crm_mon does not list any failures for this service group from what I
> can see.
> 
>>
>>>>
>>>> 4b4f-a239-8a10dad40587, cib=0.3857.2) : Resource op removal
>>>> Dec  2 21:59:32 server1 crmd: [25830]: info: te_rsc_command:
> 
> 
>>>> Initiating action 56: monitor svc-mysql_monitor_0 on server2
>>>> Dec  2 21:59:33 server1 crmd: [25830]: WARN: status_from_rc: Action
>>>> 56 (svc-mysql_monitor_0) on server2 failed (target: 7 vs. rc: 0):
>>>> Error
>>
>> The above error indicates that mysql is running on server2 but the
>> cluster didn't start it there. (The "_monitor_0" is called a "probe";
>> it's used to check the status of the service before the cluster starts
>> it. The "target: 7" means it expects the service to be stopped. The "rc:
>> 0" means the service is actually running.)
>>
>> Make sure you're not starting mysql at boot or b

Re: [ClusterLabs] Antwort: Re: clone resource - pacemaker remote

[Ken Gaillot]

On 12/05/2016 09:20 AM, philipp.achmuel...@arz.at wrote:
> Ken Gaillot <kgail...@redhat.com> schrieb am 02.12.2016 19:27:09:
> 
>> Von: Ken Gaillot <kgail...@redhat.com>
>> An: users@clusterlabs.org
>> Datum: 02.12.2016 19:32
>> Betreff: Re: [ClusterLabs] clone resource - pacemaker remote
>>
>> On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
>> > hi,
>> >
>> > what is best way to prevent clone resource trying to run on remote/guest
>> > nodes?
>>
>> location constraints with a negative score:
>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on
>>
>>
>> you can even use a single constraint with a rule based on #kind ne
>> cluster, so you don't need a separate constraint for each node:
>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> Pacemaker_Explained/index.html#_node_attribute_expressions
>>
>>
>> alternatively, you can set symmetric-cluster=false and use positive
>> constraints for cluster nodes only
>>
> 
> set constraint to single primitive, group, or on clone resource?
> are there any advantages/disadvantages using one of these methods?

When a resource is cloned, you want to refer to the clone name in any
constraints, rather than the primitive name.

For a group, it doesn't really matter, but it's simplest to use the
group name in constraints -- mainly that keeps you from accidentally
setting conflicting constraints on different members of the group. And
of course group members are automatically ordered/colocated with each
other, so you don't need individual constraints for that.

> 
>> >
>> > ...
>> > node 167873318: lnx0223a \
>> > attributes maintenance=off
>> > node 167873319: lnx0223b \
>> > attributes maintenance=off
>> > ...
>> > /primitive vm-lnx0107a VirtualDomain \/
>> > /params hypervisor="qemu:///system"
>> > config="/etc/kvm/lnx0107a.xml" \/
>> > /meta remote-node=lnx0107a238 \/
>> > /utilization cpu=1 hv_memory=4096/
>> > /primitive remote-lnx0106a ocf:pacemaker:remote \/
>> > /params server=xx.xx.xx.xx \/
>> > /meta target-role=Started/
>> > /group base-group dlm clvm vg1/
>> > /clone base-clone base-group \/
>> > /meta interleave=true target-role=Started/
>> > /.../
>> >
>> > /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
>> > operation dlm_start_0 on lnx0107a238/
>> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: executing -
>> > rsc:dlm action:start call_id:7/
>> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: finished -
>> > rsc:dlm action:start call_id:7  exit-code:5 exec-time:16ms
> queue-time:0ms/
>> > /Dec  1 14:32:58 lnx0223b crmd[9328]:error: Result of start
>> > operation for dlm on lnx0107a238: Not installed/
>> > /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
>> > on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
>> > /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
>> > on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
>> > /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
>> > start for dlm:2 on lnx0107a238: not installed (5)/
>> > /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
>> > start for dlm:2 on lnx0107a238: not installed (5)/
>> > /.../
>> > /Dec  1 14:32:49 lnx0223a pengine[9824]:   notice: Start  
>> > dlm:3#011(remote-lnx0106a)/
>> > /Dec  1 14:32:49 lnx0223a crmd[9826]:   notice: Initiating monitor
>> > operation dlm_monitor_0 locally on remote-lnx0106a/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:error: Result of probe
>> > operation for dlm on remote-lnx0106a: Not installed/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
>> > on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
>> > on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
>> > /.../
>> >
>> > ---
>> > env: pacemaker-1.1.15-19.15.x86_64
>> >
>> > thank you!


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] clone resource - pacemaker remote

[Ken Gaillot]

On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
> hi,
> 
> what is best way to prevent clone resource trying to run on remote/guest
> nodes?

location constraints with a negative score:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on


you can even use a single constraint with a rule based on #kind ne
cluster, so you don't need a separate constraint for each node:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_node_attribute_expressions


alternatively, you can set symmetric-cluster=false and use positive
constraints for cluster nodes only

> 
> ...
> node 167873318: lnx0223a \
> attributes maintenance=off
> node 167873319: lnx0223b \
> attributes maintenance=off
> ...
> /primitive vm-lnx0107a VirtualDomain \/
> /params hypervisor="qemu:///system"
> config="/etc/kvm/lnx0107a.xml" \/
> /meta remote-node=lnx0107a238 \/
> /utilization cpu=1 hv_memory=4096/
> /primitive remote-lnx0106a ocf:pacemaker:remote \/
> /params server=xx.xx.xx.xx \/
> /meta target-role=Started/
> /group base-group dlm clvm vg1/
> /clone base-clone base-group \/
> /meta interleave=true target-role=Started/
> /.../
> 
> /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
> operation dlm_start_0 on lnx0107a238/
> /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: executing -
> rsc:dlm action:start call_id:7/
> /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: finished -
> rsc:dlm action:start call_id:7  exit-code:5 exec-time:16ms queue-time:0ms/
> /Dec  1 14:32:58 lnx0223b crmd[9328]:error: Result of start
> operation for dlm on lnx0107a238: Not installed/
> /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
> on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
> /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
> on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
> /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
> start for dlm:2 on lnx0107a238: not installed (5)/
> /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
> start for dlm:2 on lnx0107a238: not installed (5)/
> /.../
> /Dec  1 14:32:49 lnx0223a pengine[9824]:   notice: Start  
> dlm:3#011(remote-lnx0106a)/
> /Dec  1 14:32:49 lnx0223a crmd[9826]:   notice: Initiating monitor
> operation dlm_monitor_0 locally on remote-lnx0106a/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:error: Result of probe
> operation for dlm on remote-lnx0106a: Not installed/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
> on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
> on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
> /.../
> 
> ---
> env: pacemaker-1.1.15-19.15.x86_64
> 
> thank you!

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] hawk - pacemaker remote

[Ken Gaillot]

On 12/02/2016 07:38 AM, philipp.achmuel...@arz.at wrote:
> Hi,
> 
> pacemaker remote nodes do not show up in hawk gui.
> regarding documentation, this should work - any hints to activate this?
> 
> thank you!
> 
> env: (SLES12.2)
> pacemaker-1.1.15-19.15.x86_64
> hawk2-2.0.0+git.1468428505.0135e38-24.32.x86_64

I'm not familiar with hawk but based on earlier conversations with
someone who uses it, it may only see nodes in the  CIB section,
so try setting any permanent node attribute on the remote nodes to make
them show up

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

On 12/01/2016 11:58 AM, Jehan-Guillaume de Rorthais wrote:
> 
> 
> Le 1 décembre 2016 17:39:45 GMT+01:00, Ken Gaillot <kgail...@redhat.com> a 
> écrit :
>> On 12/01/2016 10:13 AM, Jehan-Guillaume de Rorthais wrote:
>>> On Wed, 30 Nov 2016 14:05:19 -0600
>>> Ken Gaillot <kgail...@redhat.com> wrote:
>>>
>>>> ClusterLabs is proud to announce the latest release of the Pacemaker
>>>> cluster resource manager, version 1.1.15.
>>>
>>> 1.1.6 I guess ;)
>>
>> Whoops!
>>
>> Well now I don't feel so bad since the correction is wrong too ;)
> 
> Argh !! :-D 
> 
> What about my questions bellow BTW ? :-P 
> 
>>> But congrats anyway !
>>>
>>>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>>>> properly passed to multistate resources with notification enabled.
>>>> This has been fixed. To help resource agents detect when the fix is
>>>> available, the CRM feature set has been incremented. (Whenever the
>>>> feature set changes, mixed-version clusters are supported only during
>>>> rolling upgrades -- nodes with an older version will not be allowed to
>>>> rejoin once they shut down.)
>>>
>>>   * how could we get the "CRM feature set" version from the RA?

it's passed as an environment variable

>>>   * when this "CRM feature set" has been introduced in Pacemaker?

always, see http://wiki.clusterlabs.org/wiki/ReleaseCalendar


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org