Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Klaus Wenninger]

On 06/28/2017 05:29 PM, Klaus Wenninger wrote:
> On 05/08/2017 09:20 PM, Lentes, Bernd wrote:
>> Hi,
>>
>> i remember that digimer often campaigns for a fence delay in a 2-node  
>> cluster.
>> E.g. here: 
>> http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
>> In my eyes it makes sense, so i try to establish that. I have two HP 
>> servers, each with an ILO card.
>> I have to use the stonith:external/ipmi agent, the stonith:external/riloe 
>> refused to work.
>>
>> But i don't have a delay parameter there.
>> crm ra info stonith:external/ipmi:
>>
>> ...
>> pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
>> specify the maximum of random delay
>> This prevents double fencing when using slow devices such as sbd.
>> Use this to enable random delay for stonith actions and specify the 
>> maximum of random delay.
>> ...
>  
> Sorry for resurrecting this but
> https://github.com/ClusterLabs/pacemaker/commit/3e3cf385e0c9780fc95a4491703413b3dad54eb3
> might be of interest.

Or a little bit more readable ;-)

pcmk_delay_max: Enable random delay for stonith actions and specify
the maximum of random delay
This prevents double fencing when using slow devices
such as sbd.
Use this to enable random delay for stonith actions.
The overall delay is derived from a random delay value
adding a static delay so that the sum is kept below
the maximum delay.

pcmk_delay_base: Enable base delay for stonith actions and specify
 base delay value
 This prevents double fencing when different delays
 are configured on the nodes. Use this to enable
 static delay for stonith actions.
 The overall delay is derived from a random delay
 value adding a static delay so
 that the sum is kept below the maximum delay.

Regards,
Klaus

>> This is the only delay parameter i can use. But a random delay does not seem 
>> to be a reliable solution.
>>
>> The stonith:ipmilan agent also provides just a random delay. Same with the 
>> riloe agent.
>>
>> How did anyone solve this problem ?
>>
>> Or do i have to edit the RA (I will get practice in that :-))?
>>
>>
>> Bernd
>>
>>
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


-- 
Klaus Wenninger

Senior Software Engineer, EMEA ENG Openstack Infrastructure

Red Hat

kwenn...@redhat.com   


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Klaus Wenninger]

On 05/08/2017 09:20 PM, Lentes, Bernd wrote:
> Hi,
>
> i remember that digimer often campaigns for a fence delay in a 2-node  
> cluster.
> E.g. here: 
> http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
> In my eyes it makes sense, so i try to establish that. I have two HP servers, 
> each with an ILO card.
> I have to use the stonith:external/ipmi agent, the stonith:external/riloe 
> refused to work.
>
> But i don't have a delay parameter there.
> crm ra info stonith:external/ipmi:
>
> ...
> pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
> specify the maximum of random delay
> This prevents double fencing when using slow devices such as sbd.
> Use this to enable random delay for stonith actions and specify the 
> maximum of random delay.
> ...
 
Sorry for resurrecting this but
https://github.com/ClusterLabs/pacemaker/commit/3e3cf385e0c9780fc95a4491703413b3dad54eb3
might be of interest.

  pcmk_delay_max: Enable random delay for stonith actions and specify
the maximum of random delay
  This prevents double fencing when using slow devices
such as
  sbd. Use this to enable random delay for stonith actions.
  The overall delay is derived from a random delay value
adding a static delay so that
  the sum is kept below the maximum delay.
  pcmk_delay_base: Enable base delay for stonith actions and specify
base delay value
   This prevents double fencing when different delays
are configured on the
   nodes. Use this to enable static delay for stonith
actions.
   The overall delay is derived from a random delay
value adding a static delay so
   that the sum is kept below the maximum delay.

Regards,
Klaus

>
> This is the only delay parameter i can use. But a random delay does not seem 
> to be a reliable solution.
>
> The stonith:ipmilan agent also provides just a random delay. Same with the 
> riloe agent.
>
> How did anyone solve this problem ?
>
> Or do i have to edit the RA (I will get practice in that :-))?
>
>
> Bernd
>
>

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Dmitri Maziuk]

On 2017-05-17 06:24, Lentes, Bernd wrote:



...

I'd like to know what the software is use is doing. Am i the only one having 
that opinion ?


No.


How do you solve the problem of a deathmatch or killing the wrong node ?


*I* live dangerously with fencing disabled. But then my clusters only 
really go down for maintenance reboots, and I usually do those when I'm 
at work and can walk into the server room and push the power button when 
it comes to that.


(More accurately the one cluster that goes down. The others fail over 
without any problems.)


Dima



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Klaus Wenninger]

On 05/17/2017 03:33 PM, Lentes, Bernd wrote:
>
> - On May 17, 2017, at 2:58 PM, Klaus Wenninger kwenn...@redhat.com wrote:
>
>
>>> I don't see that.
>> fence_* are the RHCS-style fence-agents coming mainly from
>> https://github.com/ClusterLabs/fence-agents.
>>
> Ah. Ok, i see that.
>
> Do you know if they cooperate with a SuSE HAE ? I found rpm's for SLES for 
> the fence agents.

There is no conditional-compilation around support for RHCS-fence-agents.
Thus I guess there won't be a technical issue.
Question is just the degree of support you will get / want ...
But there are probably others than me who can give you a more
satisfactory answer.

Regards,
Klaus

>
> Bernd
>  
>
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
> Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons 
> Enhsen
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
>


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Lentes, Bernd]

- On May 17, 2017, at 2:11 PM, Vladislav Bogdanov bub...@hoster-ok.com 
wrote:

> 08.05.2017 22:20, Lentes, Bernd wrote:
>> Hi,
>>
>> i remember that digimer often campaigns for a fence delay in a 2-node  
>> cluster.
>> E.g. here: 
>> http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
>> In my eyes it makes sense, so i try to establish that. I have two HP servers,
>> each with an ILO card.
>> I have to use the stonith:external/ipmi agent, the stonith:external/riloe
>> refused to work.
>>
>> But i don't have a delay parameter there.
>> crm ra info stonith:external/ipmi:
> 
> Hi,
> 
> There is another ipmi fence agent - fence_ipmilan (part of fence-agents
> package). It has 'delay' parameter.
> 
>>

I don't see that.


crm(live)# ra info stonith:ipmilan
IPMI Over LAN (stonith:ipmilan)

IPMI LAN STONITH device

Parameters (*: required, []: default):

hostname* (string):
The hostname of the STONITH device

ipaddr* (string): IP Address
The IP address of the STONITH device

port* (string):
The port number to where the IPMI message is sent

auth* (string):
The authorization type of the IPMI session ("none", "straight", "md2", or 
"md5")

priv* (string):
The privilege level of the user ("operator" or "admin")

login* (string): Login
The username used for logging in to the STONITH device

password* (string): Password
The password used for logging in to the STONITH device

priority (integer, [0]): The priority of the stonith resource. Devices are 
tried in order of highest priority to lowest.
pcmk_host_argument (string, [port]): Advanced use only: An alternate parameter 
to supply instead of 'port'
Some devices do not support the standard 'port' parameter or may provide 
additional ones.
Use this to specify an alternate, device-specific, parameter that should 
indicate the machine to be fenced.
A value of 'none' can be used to tell the cluster not to supply any 
additional parameters.

pcmk_host_map (string): A mapping of host names to ports numbers for devices 
that do not support host names.
Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and 
ports 2 and 3 for node2

pcmk_host_list (string): A list of machines controlled by this device (Optional 
unless pcmk_host_check=static-list).
pcmk_host_check (string, [dynamic-list]): How to determine which machines are 
controlled by the device.
Allowed values: dynamic-list (query the device), static-list (check the 
pcmk_host_list attribute), none (assume every device can fence every machine)
...


There is no delay parameter, and all the pcmk_*** parameters are the ones from 
stonithd, and that one does not have a dedicated delay parameter,
just the pcmk_delay_max parameter which is not fixed but random. Do you have 
another ipmilan RA ?

I have SLES 11 SP4 boxes, maybe my RA is not recent enough ?

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Vladislav Bogdanov]

08.05.2017 22:20, Lentes, Bernd wrote:

Hi,

i remember that digimer often campaigns for a fence delay in a 2-node  cluster.
E.g. here: http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
In my eyes it makes sense, so i try to establish that. I have two HP servers, 
each with an ILO card.
I have to use the stonith:external/ipmi agent, the stonith:external/riloe 
refused to work.

But i don't have a delay parameter there.
crm ra info stonith:external/ipmi:


Hi,

There is another ipmi fence agent - fence_ipmilan (part of fence-agents 
package). It has 'delay' parameter.




...
pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
specify the maximum of random delay
This prevents double fencing when using slow devices such as sbd.
Use this to enable random delay for stonith actions and specify the maximum 
of random delay.
...

This is the only delay parameter i can use. But a random delay does not seem to 
be a reliable solution.

The stonith:ipmilan agent also provides just a random delay. Same with the 
riloe agent.

How did anyone solve this problem ?

Or do i have to edit the RA (I will get practice in that :-))?


Bernd





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Lentes, Bernd]

- On May 10, 2017, at 9:15 PM, Dimitri Maziuk dmaz...@bmrb.wisc.edu wrote:

> On 05/10/2017 01:54 PM, Ken Gaillot wrote:
>> On 05/10/2017 12:26 PM, Dimitri Maziuk wrote:
> 
>>> - fencing in 2-node clusters does not work reliably without fixed delay
>> 
>> Not quite. Fixed delay allows a particular method for avoiding a death
>> match in a two-node cluster. Pacemaker's built-in random delay
>> capability is another method.
> 
> Deathmatch is one problem, killing the wrong node (2 nodes, no quorum)
> is another. Fixed delay is digimer's attempt to alleviate the latter,
> so... apples and fruits not entirely unlike apples.
> 
> --

Hi,

so what should i do ? Using pcmk_delay_max does not seem to be really reliable.
I don't like the idea of being dependent from a software thinking "which delay 
i should choose, depending on the ... weather conditions, any mood ..."
I'd like to know what the software is use is doing. Am i the only one having 
that opinion ?

How do you solve the problem of a deathmatch or killing the wrong node ?

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Dimitri Maziuk]

On 05/10/2017 01:54 PM, Ken Gaillot wrote:
> On 05/10/2017 12:26 PM, Dimitri Maziuk wrote:

>> - fencing in 2-node clusters does not work reliably without fixed delay
> 
> Not quite. Fixed delay allows a particular method for avoiding a death
> match in a two-node cluster. Pacemaker's built-in random delay
> capability is another method.

Deathmatch is one problem, killing the wrong node (2 nodes, no quorum)
is another. Fixed delay is digimer's attempt to alleviate the latter,
so... apples and fruits not entirely unlike apples.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Ken Gaillot]

On 05/10/2017 12:26 PM, Dimitri Maziuk wrote:
> 
> i remember that digimer often campaigns for a fence delay in a 2-node  
> cluster.
> ...
> But  ... a random delay does not seem to
> be a reliable solution.
> 
>> Some fence agents implement a delay parameter of their own, to set a
>> fixed delay. I believe that's what digimer uses.
> 
> Is it just me or does this sound like catch-22:
> - pacemaker does not work reliably without fencing

Correct -- more specifically, some failure scenarios can't be safely
handled without fencing.

> - fencing in 2-node clusters does not work reliably without fixed delay

Not quite. Fixed delay allows a particular method for avoiding a death
match in a two-node cluster. Pacemaker's built-in random delay
capability is another method.

> - code that ships with pacemaker does not implement fixed delay.

Fence agents are used with pacemaker but not shipped as part of it. They
have their own packages distributed separately. Anyone can write a fence
agent and make it available to the community.

It would be nice if every fence agent supported a delay parameter, but
there's no requirement to do so, and even if there were, it would just
be a guideline -- it's up to the developer.

There's certainly an argument to be made for supporting a fixed delay at
the pacemaker level. There's an idea floating around to do this based on
node health, which could allow a lot of flexibility.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Dimitri Maziuk]

 i remember that digimer often campaigns for a fence delay in a 2-node  
 cluster.
...
 But  ... a random delay does not seem to
 be a reliable solution.

> Some fence agents implement a delay parameter of their own, to set a
> fixed delay. I believe that's what digimer uses.

Is it just me or does this sound like catch-22:
- pacemaker does not work reliably without fencing
- fencing in 2-node clusters does not work reliably without fixed delay
- code that ships with pacemaker does not implement fixed delay.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Ken Gaillot]

On 05/10/2017 12:20 AM, Kristoffer Grönlund wrote:
> "Lentes, Bernd"  writes:
> 
>> - On May 8, 2017, at 9:20 PM, Bernd Lentes 
>> bernd.len...@helmholtz-muenchen.de wrote:
>>
>>> Hi,
>>>
>>> i remember that digimer often campaigns for a fence delay in a 2-node  
>>> cluster.
>>> E.g. here: 
>>> http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
>>> In my eyes it makes sense, so i try to establish that. I have two HP 
>>> servers,
>>> each with an ILO card.
>>> I have to use the stonith:external/ipmi agent, the stonith:external/riloe
>>> refused to work.
>>>
>>> But i don't have a delay parameter there.
>>> crm ra info stonith:external/ipmi:
>>>
>>> ...
>>> pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
>>> specify
>>> the maximum of random delay
>>>This prevents double fencing when using slow devices such as sbd.
>>>Use this to enable random delay for stonith actions and specify the 
>>> maximum of
>>>random delay.
>>> ...
>>>
>>> This is the only delay parameter i can use. But a random delay does not 
>>> seem to
>>> be a reliable solution.
>>>
>>> The stonith:ipmilan agent also provides just a random delay. Same with the 
>>> riloe
>>> agent.
>>>
>>> How did anyone solve this problem ?
>>>
>>> Or do i have to edit the RA (I will get practice in that :-))?
>>>
>>>
>>
>> crm ra info stonith:external/ipmi says there exists a parameter 
>> pcmk_delay_max.
>> Having a look in  /usr/lib64/stonith/plugins/external/ipmi i don't find 
>> anything about delay.
>> Also "crm_resource --show-metadata=stonith:external/ipmi" does not say 
>> anything about a delay.
>>
>> Is this "pcmk_delay_max" not implemented ? From where does "crm ra info 
>> stonith:external/ipmi" get this info ?
>>
> 
> pcmk_delay_max is implemented by Pacemaker. crmsh gets the information
> about available parameters by querying stonithd directly.
> 
> Cheers,
> Kristoffer

The various pcmk_* parameters are documented in the stonithd(7) man page.

Some fence agents implement a delay parameter of their own, to set a
fixed delay. I believe that's what digimer uses.

>>
>> Bernd
>>  
>>
>> Helmholtz Zentrum Muenchen
>> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
>> Ingolstaedter Landstr. 1
>> 85764 Neuherberg
>> www.helmholtz-muenchen.de
>> Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
>> Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons 
>> Enhsen
>> Registergericht: Amtsgericht Muenchen HRB 6466
>> USt-IdNr: DE 129521671

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Lentes, Bernd]

- On May 8, 2017, at 9:20 PM, Bernd Lentes 
bernd.len...@helmholtz-muenchen.de wrote:

> Hi,
> 
> i remember that digimer often campaigns for a fence delay in a 2-node  
> cluster.
> E.g. here: 
> http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
> In my eyes it makes sense, so i try to establish that. I have two HP servers,
> each with an ILO card.
> I have to use the stonith:external/ipmi agent, the stonith:external/riloe
> refused to work.
> 
> But i don't have a delay parameter there.
> crm ra info stonith:external/ipmi:
> 
> ...
> pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
> specify
> the maximum of random delay
>This prevents double fencing when using slow devices such as sbd.
>Use this to enable random delay for stonith actions and specify the 
> maximum of
>random delay.
> ...
> 
> This is the only delay parameter i can use. But a random delay does not seem 
> to
> be a reliable solution.
> 
> The stonith:ipmilan agent also provides just a random delay. Same with the 
> riloe
> agent.
> 
> How did anyone solve this problem ?
> 
> Or do i have to edit the RA (I will get practice in that :-))?
> 
> 

crm ra info stonith:external/ipmi says there exists a parameter pcmk_delay_max.
Having a look in  /usr/lib64/stonith/plugins/external/ipmi i don't find 
anything about delay.
Also "crm_resource --show-metadata=stonith:external/ipmi" does not say anything 
about a delay.

Is this "pcmk_delay_max" not implemented ? From where does "crm ra info 
stonith:external/ipmi" get this info ?


Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org