Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-19 Thread Jehan-Guillaume de Rorthais 
On Mon, 19 Dec 2016 13:37:09 +0100
Klaus Wenninger  wrote:

> On 12/17/2016 11:55 PM, Jehan-Guillaume de Rorthais wrote:
> > On Wed, 14 Dec 2016 14:52:41 +0100
> > Klaus Wenninger  wrote:
> >  
> >> On 12/14/2016 01:26 PM, Jehan-Guillaume de Rorthais wrote:  
> >>> On Thu, 8 Dec 2016 11:47:20 +0100
> >>> Jehan-Guillaume de Rorthais  wrote:
> >>>
>  Hello,
> 
>  While setting this various parameters, I couldn't find documentation and
>  details about them. Bellow some questions.
> 
>  Considering the watchdog module used on a server is set up with a 30s
>  timer (lets call it the wdt, the "watchdog timer"), how should
>  "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout"
>  be set?
> 
>  Here is my thinking so far:
> 
>  "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer
>  before the wdt expire so the server stay alive. Online resources and
>  default values are usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But
>  what if sbd fails to reset the timer multiple times (eg. because of
>  excessive load, swap storm etc)? The server will not reset before
>  random*SBD_WATCHDOG_TIMEOUT or wdt, right? 
> >> SBD_WATCHDOG_TIMEOUT (e.g. in /etc/sysconfig/sbd) is already the
> >> timeout the hardware watchdog is configured to by sbd-daemon.  
> > Oh, ok, I did not realized sbd was actually setting the hardware watchdog
> > timeout itself based on this variable. After some quick search to make sure
> > I understand it right, I suppose it is done there?
> > https://github.com/ClusterLabs/sbd/blob/172dcd03eaf26503a10a18501aa1b9f30eed7ee2/src/sbd-common.c#L123
> >  
> >> sbd-daemon is triggering faster - timeout_loop defaults to 1s but
> >> is configurable.
> >>
> >> SBD_WATCHDOG_TIMEOUT (and maybe the loop timeout as well
> >> but significantly shorter should be sufficient)
> >> has to be configured so that failing to trigger within time means
> >> a failure with high enough certainty or the machine showing
> >> comparable response-times would anyway violate timing requirements
> >> of the services running on itself and in the cluster.  
> > OK. So I understand now why 5s is fine as a default value then.
> >  
> >> Have in mind that sbd-daemon defaults to running realtime-scheduled
> >> and thus is gonna be more responsive than the usual services
> >> on the system. Although you of course have to consider that
> >> the watchers (child-processes of sbd that are observing e.g.
> >> the block-device(s), corosync, pacemaker_remoted or
> >> pacemaker node-health) might be significantly less responsive
> >> due to their communication partners.  
> > I'm not sure yet to understand clearly the mechanism and interactions of sbd
> > with other daemons. So far, I understood that Pacemaker/stonithd was able to
> > poke sbd to ask it to trigger a node reset through the wd device. I'm very
> > new to this area and I still lake of self documentation.  
> 
> Pacemaker is setting the node unclean which pacemaker-watcher
> (one of sbd daemons) sees as it is connected to the cib.
> This is why the mechanism is working (sort of - see the discussion
> in my pull request in the sbd-repo) on nodes without stonithd as
> well (remote-nodes).
> If you are running sbd with a block-device there is of course this
> way of communication as well between pacemaker and sbd.
> (e.g. via fence_sbd fence-agent)
> Be aware that there are different levels of support for these
> features in the distributions. (RHEL more on the watchdog-side,
> SLES more on the block-device side ... roughly as far as I got it)

OK, I have a better understanding of the need for various sbd watchers and
how it all sounds to works.

>  "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure
>  what is stonith-watchdog-timeout. Is it the maximum time to wait from
>  stonithd after it asked for a node fencing before it considers the
>  watchdog was actually triggered and the node reseted, even with no
>  confirmation? I suppose "stonith-watchdog-timeout" is mostly useful to
>  stonithd, right?
> >> Yes, the time we can assume a node to be killed by the hardware-watchdog...
> >> Double the hardware-watchdog-timeout is a good choice.  
> > OK, thank you
> >  
>  "stonith-watchdog-timeout < stonith-timeout". I understand the stonith
>  action timeout should be at least greater than the wdt so stonithd will
>  not raise a timeout before the wdt had a chance to exprire and reset the
>  node. Is it right?
> >> stonith-timeout is the cluster-wide-defaut to wait for stonith-devices
> >> to carry out their duty. In the sbd-case without a block-device (sbd used
> >> for pacemaker to be observed by a hardware-watchdog) it shouldn't
> >> play a role.  
> > I thought self-fencing through sbd/wd was carried by stonithd because of
> > such messages in my

Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-19 Thread Klaus Wenninger 
On 12/17/2016 11:55 PM, Jehan-Guillaume de Rorthais wrote:
> On Wed, 14 Dec 2016 14:52:41 +0100
> Klaus Wenninger  wrote:
>
>> On 12/14/2016 01:26 PM, Jehan-Guillaume de Rorthais wrote:
>>> On Thu, 8 Dec 2016 11:47:20 +0100
>>> Jehan-Guillaume de Rorthais  wrote:
>>>  
 Hello,

 While setting this various parameters, I couldn't find documentation and
 details about them. Bellow some questions.

 Considering the watchdog module used on a server is set up with a 30s timer
 (lets call it the wdt, the "watchdog timer"), how should
 "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout" be
 set?

 Here is my thinking so far:

 "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer before
 the wdt expire so the server stay alive. Online resources and default
 values are usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But what if
 sbd fails to reset the timer multiple times (eg. because of excessive
 load, swap storm etc)? The server will not reset before
 random*SBD_WATCHDOG_TIMEOUT or wdt, right?   
>> SBD_WATCHDOG_TIMEOUT (e.g. in /etc/sysconfig/sbd) is already the
>> timeout the hardware watchdog is configured to by sbd-daemon.
> Oh, ok, I did not realized sbd was actually setting the hardware watchdog
> timeout itself based on this variable. After some quick search to make sure I
> understand it right, I suppose it is done there?
> https://github.com/ClusterLabs/sbd/blob/172dcd03eaf26503a10a18501aa1b9f30eed7ee2/src/sbd-common.c#L123
>
>> sbd-daemon is triggering faster - timeout_loop defaults to 1s but
>> is configurable.
>>
>> SBD_WATCHDOG_TIMEOUT (and maybe the loop timeout as well
>> but significantly shorter should be sufficient)
>> has to be configured so that failing to trigger within time means
>> a failure with high enough certainty or the machine showing
>> comparable response-times would anyway violate timing requirements
>> of the services running on itself and in the cluster.
> OK. So I understand now why 5s is fine as a default value then.
>
>> Have in mind that sbd-daemon defaults to running realtime-scheduled
>> and thus is gonna be more responsive than the usual services
>> on the system. Although you of course have to consider that
>> the watchers (child-processes of sbd that are observing e.g.
>> the block-device(s), corosync, pacemaker_remoted or
>> pacemaker node-health) might be significantly less responsive
>> due to their communication partners.
> I'm not sure yet to understand clearly the mechanism and interactions of sbd
> with other daemons. So far, I understood that Pacemaker/stonithd was able to
> poke sbd to ask it to trigger a node reset through the wd device. I'm very new
> to this area and I still lake of self documentation.

Pacemaker is setting the node unclean which pacemaker-watcher
(one of sbd daemons) sees as it is connected to the cib.
This is why the mechanism is working (sort of - see the discussion
in my pull request in the sbd-repo) on nodes without stonithd as
well (remote-nodes).
If you are running sbd with a block-device there is of course this
way of communication as well between pacemaker and sbd.
(e.g. via fence_sbd fence-agent)
Be aware that there are different levels of support for these
features in the distributions. (RHEL more on the watchdog-side,
SLES more on the block-device side ... roughly as far as I got it)

>
 "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure what
 is stonith-watchdog-timeout. Is it the maximum time to wait from stonithd
 after it asked for a node fencing before it considers the watchdog was
 actually triggered and the node reseted, even with no confirmation? I
 suppose "stonith-watchdog-timeout" is mostly useful to stonithd, right?  
>> Yes, the time we can assume a node to be killed by the hardware-watchdog...
>> Double the hardware-watchdog-timeout is a good choice.
> OK, thank you
>
 "stonith-watchdog-timeout < stonith-timeout". I understand the stonith
 action timeout should be at least greater than the wdt so stonithd will
 not raise a timeout before the wdt had a chance to exprire and reset the
 node. Is it right?  
>> stonith-timeout is the cluster-wide-defaut to wait for stonith-devices
>> to carry out their duty. In the sbd-case without a block-device (sbd used
>> for pacemaker to be observed by a hardware-watchdog) it shouldn't
>> play a role.
> I thought self-fencing through sbd/wd was carried by stonithd because of such
> messages in my PoC log files:
>
>   stonith-ng: notice: unpack_config: Relying on watchdog integration for 
> fencing

see above ... or read as sit still and wait for the watchdog to do the
job ;-)

>
> That's why I thought "stonith-timeout" might have a role there, as it looks
> like a stonith device then...
>
> By pure tech interest here, some more input or documentation to read about how
> it works

Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-17 Thread Jehan-Guillaume de Rorthais 
On Wed, 14 Dec 2016 14:52:41 +0100
Klaus Wenninger  wrote:

> On 12/14/2016 01:26 PM, Jehan-Guillaume de Rorthais wrote:
> > On Thu, 8 Dec 2016 11:47:20 +0100
> > Jehan-Guillaume de Rorthais  wrote:
> >  
> >> Hello,
> >>
> >> While setting this various parameters, I couldn't find documentation and
> >> details about them. Bellow some questions.
> >>
> >> Considering the watchdog module used on a server is set up with a 30s timer
> >> (lets call it the wdt, the "watchdog timer"), how should
> >> "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout" be
> >> set?
> >>
> >> Here is my thinking so far:
> >>
> >> "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer before
> >> the wdt expire so the server stay alive. Online resources and default
> >> values are usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But what if
> >> sbd fails to reset the timer multiple times (eg. because of excessive
> >> load, swap storm etc)? The server will not reset before
> >> random*SBD_WATCHDOG_TIMEOUT or wdt, right?   
> 
> SBD_WATCHDOG_TIMEOUT (e.g. in /etc/sysconfig/sbd) is already the
> timeout the hardware watchdog is configured to by sbd-daemon.

Oh, ok, I did not realized sbd was actually setting the hardware watchdog
timeout itself based on this variable. After some quick search to make sure I
understand it right, I suppose it is done there?
https://github.com/ClusterLabs/sbd/blob/172dcd03eaf26503a10a18501aa1b9f30eed7ee2/src/sbd-common.c#L123

> sbd-daemon is triggering faster - timeout_loop defaults to 1s but
> is configurable.
> 
> SBD_WATCHDOG_TIMEOUT (and maybe the loop timeout as well
> but significantly shorter should be sufficient)
> has to be configured so that failing to trigger within time means
> a failure with high enough certainty or the machine showing
> comparable response-times would anyway violate timing requirements
> of the services running on itself and in the cluster.

OK. So I understand now why 5s is fine as a default value then.

> Have in mind that sbd-daemon defaults to running realtime-scheduled
> and thus is gonna be more responsive than the usual services
> on the system. Although you of course have to consider that
> the watchers (child-processes of sbd that are observing e.g.
> the block-device(s), corosync, pacemaker_remoted or
> pacemaker node-health) might be significantly less responsive
> due to their communication partners.

I'm not sure yet to understand clearly the mechanism and interactions of sbd
with other daemons. So far, I understood that Pacemaker/stonithd was able to
poke sbd to ask it to trigger a node reset through the wd device. I'm very new
to this area and I still lake of self documentation.

> >> "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure what
> >> is stonith-watchdog-timeout. Is it the maximum time to wait from stonithd
> >> after it asked for a node fencing before it considers the watchdog was
> >> actually triggered and the node reseted, even with no confirmation? I
> >> suppose "stonith-watchdog-timeout" is mostly useful to stonithd, right?  
> 
> Yes, the time we can assume a node to be killed by the hardware-watchdog...
> Double the hardware-watchdog-timeout is a good choice.

OK, thank you

> >> "stonith-watchdog-timeout < stonith-timeout". I understand the stonith
> >> action timeout should be at least greater than the wdt so stonithd will
> >> not raise a timeout before the wdt had a chance to exprire and reset the
> >> node. Is it right?  
> 
> stonith-timeout is the cluster-wide-defaut to wait for stonith-devices
> to carry out their duty. In the sbd-case without a block-device (sbd used
> for pacemaker to be observed by a hardware-watchdog) it shouldn't
> play a role.

I thought self-fencing through sbd/wd was carried by stonithd because of such
messages in my PoC log files:

  stonith-ng: notice: unpack_config: Relying on watchdog integration for fencing

That's why I thought "stonith-timeout" might have a role there, as it looks
like a stonith device then...

By pure tech interest here, some more input or documentation to read about how
it works would be really appreciated.

> When a block-device is being used it guards the
> communication with the fence-agent communicating with the
> block-device.

OK


Thank you for your help!
-- 
Jehan-Guillaume de Rorthais
Dalibo

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-14 Thread Klaus Wenninger 
On 12/14/2016 01:26 PM, Jehan-Guillaume de Rorthais wrote:
> On Thu, 8 Dec 2016 11:47:20 +0100
> Jehan-Guillaume de Rorthais  wrote:
>
>> Hello,
>>
>> While setting this various parameters, I couldn't find documentation and
>> details about them. Bellow some questions.
>>
>> Considering the watchdog module used on a server is set up with a 30s timer
>> (lets call it the wdt, the "watchdog timer"), how should
>> "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout" be
>> set?
>>
>> Here is my thinking so far:
>>
>> "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer before 
>> the
>> wdt expire so the server stay alive. Online resources and default values are
>> usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But what if sbd fails to
>> reset the timer multiple times (eg. because of excessive load, swap storm
>> etc)? The server will not reset before random*SBD_WATCHDOG_TIMEOUT or wdt,
>> right? 

SBD_WATCHDOG_TIMEOUT (e.g. in /etc/sysconfig/sbd) is already the
timeout the hardware watchdog is configured to by sbd-daemon.
sbd-daemon is triggering faster - timeout_loop defaults to 1s but
is configurable.

SBD_WATCHDOG_TIMEOUT (and maybe the loop timeout as well
but significantly shorter should be sufficient)
has to be configured so that failing to trigger within time means
a failure with high enough certainty or the machine showing
comparable response-times would anyway violate timing requirements
of the services running on itself and in the cluster.

Have in mind that sbd-daemon defaults to running realtime-scheduled
and thus is gonna be more responsive than the usual services
on the system. Although you of course have to consider that
the watchers (child-processes of sbd that are observing e.g.
the block-device(s), corosync, pacemaker_remoted or
pacemaker node-health) might be significantly less responsive
due to their communication partners.
 
>>
>> "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure what is
>> stonith-watchdog-timeout. Is it the maximum time to wait from stonithd after
>> it asked for a node fencing before it considers the watchdog was actually
>> triggered and the node reseted, even with no confirmation? I suppose
>> "stonith-watchdog-timeout" is mostly useful to stonithd, right?

Yes, the time we can assume a node to be killed by the hardware-watchdog...
Double the hardware-watchdog-timeout is a good choice.

>>
>> "stonith-watchdog-timeout < stonith-timeout". I understand the stonith action
>> timeout should be at least greater than the wdt so stonithd will not raise a
>> timeout before the wdt had a chance to exprire and reset the node. Is it
>> right?

stonith-timeout is the cluster-wide-defaut to wait for stonith-devices
to carry out their duty. In the sbd-case without a block-device (sbd used
for pacemaker to be observed by a hardware-watchdog) it shouldn't
play a role. When a block-device is being used it guards the
communication with the fence-agent communicating with the
block-device.

> Anyone on these questions? I am currently writing some more doc/cookbook for
> the PAF project[1], I would prefer being sure of what is written there :)
>
> [1] http://dalibo.github.io/PAF/documentation.html
>
> Regards,
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-14 Thread Jehan-Guillaume de Rorthais 
On Thu, 8 Dec 2016 11:47:20 +0100
Jehan-Guillaume de Rorthais  wrote:

> Hello,
> 
> While setting this various parameters, I couldn't find documentation and
> details about them. Bellow some questions.
> 
> Considering the watchdog module used on a server is set up with a 30s timer
> (lets call it the wdt, the "watchdog timer"), how should
> "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout" be
> set?
> 
> Here is my thinking so far:
> 
> "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer before the
> wdt expire so the server stay alive. Online resources and default values are
> usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But what if sbd fails to
> reset the timer multiple times (eg. because of excessive load, swap storm
> etc)? The server will not reset before random*SBD_WATCHDOG_TIMEOUT or wdt,
> right? 
> 
> "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure what is
> stonith-watchdog-timeout. Is it the maximum time to wait from stonithd after
> it asked for a node fencing before it considers the watchdog was actually
> triggered and the node reseted, even with no confirmation? I suppose
> "stonith-watchdog-timeout" is mostly useful to stonithd, right?
> 
> "stonith-watchdog-timeout < stonith-timeout". I understand the stonith action
> timeout should be at least greater than the wdt so stonithd will not raise a
> timeout before the wdt had a chance to exprire and reset the node. Is it
> right?

Anyone on these questions? I am currently writing some more doc/cookbook for
the PAF project[1], I would prefer being sure of what is written there :)

[1] http://dalibo.github.io/PAF/documentation.html

Regards,

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] setting up SBD_WATCHDOG_TIMEOUT, stonith-timeout and stonith-watchdog-timeout

2016-12-08 Thread emmanuel segura 
the only thing that I can say is: sbd is a realtime process

2016-12-08 11:47 GMT+01:00 Jehan-Guillaume de Rorthais :
> Hello,
>
> While setting this various parameters, I couldn't find documentation and
> details about them. Bellow some questions.
>
> Considering the watchdog module used on a server is set up with a 30s timer
> (lets call it the wdt, the "watchdog timer"), how should
> "SBD_WATCHDOG_TIMEOUT", "stonith-timeout" and "stonith-watchdog-timeout" be 
> set?
>
> Here is my thinking so far:
>
> "SBD_WATCHDOG_TIMEOUT < wdt". The sbd daemon should reset the timer before the
> wdt expire so the server stay alive. Online resources and default values are
> usually "SBD_WATCHDOG_TIMEOUT=5s" and "wdt=30s". But what if sbd fails to 
> reset
> the timer multiple times (eg. because of excessive load, swap storm etc)? The
> server will not reset before random*SBD_WATCHDOG_TIMEOUT or wdt, right?
>
> "stonith-watchdog-timeout > SBD_WATCHDOG_TIMEOUT". I'm not quite sure what is
> stonith-watchdog-timeout. Is it the maximum time to wait from stonithd after 
> it
> asked for a node fencing before it considers the watchdog was actually
> triggered and the node reseted, even with no confirmation? I suppose
> "stonith-watchdog-timeout" is mostly useful to stonithd, right?
>
> "stonith-watchdog-timeout < stonith-timeout". I understand the stonith action
> timeout should be at least greater than the wdt so stonithd will not raise a
> timeout before the wdt had a chance to exprire and reset the node. Is it 
> right?
>
> Any other comments?
>
> Regards,
> --
> Jehan-Guillaume de Rorthais
> Dalibo
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



-- 
  .~.
  /V\
 //  \\
/(   )\
^`~'^

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org