[ClusterLabs] Antw: Re: SBD as watchdog daemon
>>> schrieb am 15.04.2019 um 13:03 in Nachricht <566fe1cd-b8fd-41e0-bc07-1722be14e...@ya.ru>: > >> 14 апр. 2019 г., в 10:12, Andrei Borzenkov написал(а): > > Thanks for explanation, I think this will be good addition to the SBD > manual. (SBD manual need in this.) But my problem lies in other plain. > > I investigated SBD. A common watchdog is a much simple. One infinite loop, > checks some tests and write to the watchdog device. Any mistakes, freeze or > segfault and watchdog will fire. But SBD has another design. First of all > there is not one infinite loop. There are three different processes, one is > «inquisitor" and to other «servants» for corosync and pacemaker. And there is > complex logic to check each other inside SBD. But the problem even is not > here. Both the servants send to the inquisitor health heartbeat every second. > But… They send health heartbeat not as result of checking corosync or > pacemaker, as expected to be, but from the internal buffer variable > «servant_health». And if corosync or pacemaker is frozen (can be emulated by > `kill -s STOP`), this variable is never changed and the servants continue > send to the inquisitor a good health status always. And this is a bug. I am > looking a way to fix this. I had a similar design, where a monitor updated some measurements in shared value, and another thread was reading them. The design idea was to avoid blocking the read. But if the monitor block, you read old data all the time. So I simply added a modulo-16 count that is incremented whenever the data was updated. So if the counter does not change, there's a 15/16 probability that the data wasn't actually updated. As everything was performance- critical, I did no use the luxury of writing a stime-stamp into shared memory... ;-) Regards, Ulrich > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Resource not starting correctly
Thanks. See my comments interspersed below. On Mon, Apr 15, 2019 at 4:30 PM Ken Gaillot wrote: > On Mon, 2019-04-15 at 14:15 -0600, JCA wrote: > > I have a simple two-node cluster, node one and node two, with a > > single resource, ClusterMyApp. The nodes are CentOS 7 VMs. The > > resource is created executing the following line in node one: > > > ># pcs resource create ClusterMyApp ocf:myapp:myapp-script op > > monitor interval=30s > > FYI, it doesn't matter which node you change the configuration on -- > it's automatically sync'd across all nodes. > > OK. > This invokes myapp-script, which I installed under > > /usr/lib/ocf/resource.d/myapp/myapp-script, both in one and two - > > i.e. it is exactly the same script in both nodes. > > > > On executing the command above in node one, I get the following log > > entries in node one itself: > > > > Apr 15 13:40:12 one crmd[13670]: notice: Result of probe operation > > for ClusterMyApp on one: 7 (not running) > > Apr 15 13:40:12 one crmd[13670]: notice: Result of start operation > > for ClusterMyApp on one: 0 (ok) > > Whenever starting the cluster on a node, or adding a resource, > pacemaker probes the state of all resources on the node (or all nodes > in the case of adding a resource) by calling the agent's "monitor" > action once. You'll see this "Result of probe operation" for all > resources on all nodes. > > This allows pacemaker to detect if and where a service is already > running. > > OK. > > This is in line with what I expect from myapp-script when invoked > > with the 'start' option (which is what the command above is doing.) > > myapp-script first checks out whether my app is running, and if it is > > not then launches it. The rest of the log entries are to do with my > > app, indicating that it started without any problems. > > > > In node two, when the command above is executed in one, the following > > log entries are generated: > > > > Apr 15 13:40:12 two crmd[4293]: notice: State transition S_IDLE -> > > S_POLICY_ENGINE > > Apr 15 13:40:12 two pengine[4292]: notice: * Start > > ClusterMyApp (one ) > > Apr 15 13:40:12 two pengine[4292]: notice: Calculated transition 16, > > saving inputs in /var/lib/pacemaker/pengine/pe-input-66.bz2 > > At any given time, one node in the cluster is elected the "Designated > Controller" (DC). That node will calculate what (if anything) needs to > be done, and tell the right nodes to do it. Above, it has determined > that ClusterMyApp needs to be started on node one. > OK. > > > Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation > > ClusterMyApp_monitor_0 locally on two > > Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation > > ClusterMyApp_monitor_0 on one > > The cluster first probes the current state of the service on both > nodes, before any actions have been taken. The expected result is "not > running". > OK. > > > Apr 15 13:40:12 two crmd[4293]: notice: Result of probe operation > > for ClusterMyApp on two: 7 (not running) > > Apr 15 13:40:12 two crmd[4293]: notice: Initiating start operation > > ClusterMyApp_start_0 on one > > Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation > > ClusterMyApp_monitor_3 on one > > Apr 15 13:40:12 two crmd[4293]: warning: Action 4 > > (ClusterMyApp_monitor_3) on one failed (target: 0 vs. rc: 7): > > Error > > Apr 15 13:40:12 two crmd[4293]: notice: Transition aborted by > > operation ClusterMyApp_monitor_3 'create' on one: Event failed > > The cluster successfully probed the service on both nodes, and started > it on node one. It then tried to start a 30-second recurring monitor > for the service, but the monitor immediately failed (the expected > result was running, but the monitor said it was not running). > It failed, where? In one, I know for a fact that my app is running, as reported by ps. I also know it has started correctly and is sitting there for stuff to do - it depends on timers and external events. In two, of course, it is not running. > > > After doing all of the above, pcs status returns the following, when > > invoked in either node: > > > > Cluster name: MyCluster > > Stack: corosync > > Current DC: two (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition > > with quorum > > Last updated: Mon Apr 15 13:45:14 2019 > > Last change: Mon Apr 15 13:40:11 2019 by root via cibadmin on one > > > > 2 nodes configured > > 1 resource configured > > > > Online: [ one two ] > > > > Full list of resources: > > > > ClusterMyApp (ocf::myapp:myapp-script): Started one > > > > Failed Actions: > > * ClusterMyApp_monitor_3 on one 'not running' (7): call=37, > > status=complete, exitreason='', > > last-rc-change='Mon Apr 15 13:40:12 2019', queued=0ms, exec=105ms > > > > > > Daemon Status: > > corosync: active/disabled > > pacemaker: active/disabled > > pcsd: active/enabled > > > > The start function in this script is as follows: > > > >
Re: [ClusterLabs] Resource not starting correctly
On Mon, 2019-04-15 at 14:15 -0600, JCA wrote:
> I have a simple two-node cluster, node one and node two, with a
> single resource, ClusterMyApp. The nodes are CentOS 7 VMs. The
> resource is created executing the following line in node one:
>
># pcs resource create ClusterMyApp ocf:myapp:myapp-script op
> monitor interval=30s
FYI, it doesn't matter which node you change the configuration on --
it's automatically sync'd across all nodes.
> This invokes myapp-script, which I installed under
> /usr/lib/ocf/resource.d/myapp/myapp-script, both in one and two -
> i.e. it is exactly the same script in both nodes.
>
> On executing the command above in node one, I get the following log
> entries in node one itself:
>
> Apr 15 13:40:12 one crmd[13670]: notice: Result of probe operation
> for ClusterMyApp on one: 7 (not running)
> Apr 15 13:40:12 one crmd[13670]: notice: Result of start operation
> for ClusterMyApp on one: 0 (ok)
Whenever starting the cluster on a node, or adding a resource,
pacemaker probes the state of all resources on the node (or all nodes
in the case of adding a resource) by calling the agent's "monitor"
action once. You'll see this "Result of probe operation" for all
resources on all nodes.
This allows pacemaker to detect if and where a service is already
running.
> This is in line with what I expect from myapp-script when invoked
> with the 'start' option (which is what the command above is doing.)
> myapp-script first checks out whether my app is running, and if it is
> not then launches it. The rest of the log entries are to do with my
> app, indicating that it started without any problems.
>
> In node two, when the command above is executed in one, the following
> log entries are generated:
>
> Apr 15 13:40:12 two crmd[4293]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE
> Apr 15 13:40:12 two pengine[4292]: notice: * Start
> ClusterMyApp (one )
> Apr 15 13:40:12 two pengine[4292]: notice: Calculated transition 16,
> saving inputs in /var/lib/pacemaker/pengine/pe-input-66.bz2
At any given time, one node in the cluster is elected the "Designated
Controller" (DC). That node will calculate what (if anything) needs to
be done, and tell the right nodes to do it. Above, it has determined
that ClusterMyApp needs to be started on node one.
> Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
> ClusterMyApp_monitor_0 locally on two
> Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
> ClusterMyApp_monitor_0 on one
The cluster first probes the current state of the service on both
nodes, before any actions have been taken. The expected result is "not
running".
> Apr 15 13:40:12 two crmd[4293]: notice: Result of probe operation
> for ClusterMyApp on two: 7 (not running)
> Apr 15 13:40:12 two crmd[4293]: notice: Initiating start operation
> ClusterMyApp_start_0 on one
> Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
> ClusterMyApp_monitor_3 on one
> Apr 15 13:40:12 two crmd[4293]: warning: Action 4
> (ClusterMyApp_monitor_3) on one failed (target: 0 vs. rc: 7):
> Error
> Apr 15 13:40:12 two crmd[4293]: notice: Transition aborted by
> operation ClusterMyApp_monitor_3 'create' on one: Event failed
The cluster successfully probed the service on both nodes, and started
it on node one. It then tried to start a 30-second recurring monitor
for the service, but the monitor immediately failed (the expected
result was running, but the monitor said it was not running).
> After doing all of the above, pcs status returns the following, when
> invoked in either node:
>
> Cluster name: MyCluster
> Stack: corosync
> Current DC: two (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition
> with quorum
> Last updated: Mon Apr 15 13:45:14 2019
> Last change: Mon Apr 15 13:40:11 2019 by root via cibadmin on one
>
> 2 nodes configured
> 1 resource configured
>
> Online: [ one two ]
>
> Full list of resources:
>
> ClusterMyApp (ocf::myapp:myapp-script): Started one
>
> Failed Actions:
> * ClusterMyApp_monitor_3 on one 'not running' (7): call=37,
> status=complete, exitreason='',
> last-rc-change='Mon Apr 15 13:40:12 2019', queued=0ms, exec=105ms
>
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
>
> The start function in this script is as follows:
>
> myapp_start() {
> myapp_conf_check
> local diagnostic=$?
>
> if [ $diagnostic -ne $OCF_SUCCESS ]; then
> return $diagnostic
> fi
>
> myapp_monitor
>
> local state=$?
>
> case $state in
> $OCF_SUCCESS)
> return $OCF_SUCCESS
> ;;
>
> $OCF_NOT_RUNNING)
> myapp_launch > /dev/null 2>&1
> if [ $? -eq 0 ]; then
> return $OCF_SUCCESS
> fi
>
> return $OCF_ERR_GENERIC
> ;;
>
> *)
> return $state
> ;;
> esac
> }
>
> I know for a fact that, in one, myapp_launch gets invoked, and that
> its exit
[ClusterLabs] Resource not starting correctly III
This is weird. Further experiments, consisting of creating and deleting the resource, reveal that, on creating the resource, myapp-script may be invoked multiple times - sometimes four, sometimes twenty or so, sometimes returning OCF_SUCCESS, some other times returning OCF_NOT_RUNNING. And whether or not it succeeds, as per pcs status, this seems to be something completely random. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Resource not starting correctly II
Well, I remain puzzled. I added a statement to the end of my script in order to capture its return value. Much to my surprise, when I create the associated resource (as described in my previous post) myapp-script gets invoked four times in node one (where the resource is created) and two in node two. Just as intriguing, the return value in one is, for each instance OCF_NOT_RUNNING OCF_SUCCESS OCF_SUCCESS OCF_NOT_RUNNING while in node two I get OCF_NOT_RUNNING OCF_SUCCESS And now things work, as can be seen in the output from pcs status: Cluster name: MyCluster Stack: corosync Current DC: two (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum Last updated: Mon Apr 15 15:21:08 2019 Last change: Mon Apr 15 15:20:47 2019 by root via cibadmin on one 2 nodes configured 1 resource configured Online: [ one two ] Full list of resources: ClusterMyApp (ocf::myapp:myapp-script): Started one Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled Can anybody clarify what is going on? And how to guarantee that, assuming that my app starts correctly, the creation resource will predictably succeed? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Resource not starting correctly
I have a simple two-node cluster, node one and node two, with a single
resource, ClusterMyApp. The nodes are CentOS 7 VMs. The resource is created
executing the following line in node one:
# pcs resource create ClusterMyApp ocf:myapp:myapp-script op monitor
interval=30s
This invokes myapp-script, which I installed under
/usr/lib/ocf/resource.d/myapp/myapp-script, both in one and two - i.e. it
is exactly the same script in both nodes.
On executing the command above in node one, I get the following log entries
in node one itself:
Apr 15 13:40:12 one crmd[13670]: notice: Result of probe operation for
ClusterMyApp on one: 7 (not running)
Apr 15 13:40:12 one crmd[13670]: notice: Result of start operation for
ClusterMyApp on one: 0 (ok)
This is in line with what I expect from myapp-script when invoked with the
'start' option (which is what the command above is doing.) myapp-script
first checks out whether my app is running, and if it is not then launches
it. The rest of the log entries are to do with my app, indicating that it
started without any problems.
In node two, when the command above is executed in one, the following log
entries are generated:
Apr 15 13:40:12 two crmd[4293]: notice: State transition S_IDLE ->
S_POLICY_ENGINE
Apr 15 13:40:12 two pengine[4292]: notice: * Start ClusterMyApp
(one )
Apr 15 13:40:12 two pengine[4292]: notice: Calculated transition 16,
saving inputs in /var/lib/pacemaker/pengine/pe-input-66.bz2
Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
ClusterMyApp_monitor_0 locally on two
Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
ClusterMyApp_monitor_0 on one
Apr 15 13:40:12 two crmd[4293]: notice: Result of probe operation for
ClusterMyApp on two: 7 (not running)
Apr 15 13:40:12 two crmd[4293]: notice: Initiating start operation
ClusterMyApp_start_0 on one
Apr 15 13:40:12 two crmd[4293]: notice: Initiating monitor operation
ClusterMyApp_monitor_3 on one
Apr 15 13:40:12 two crmd[4293]: warning: Action 4
(ClusterMyApp_monitor_3) on one failed (target: 0 vs. rc: 7): Error
Apr 15 13:40:12 two crmd[4293]: notice: Transition aborted by operation
ClusterMyApp_monitor_3 'create' on one: Event failed
After doing all of the above, pcs status returns the following, when
invoked in either node:
Cluster name: MyCluster
Stack: corosync
Current DC: two (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with
quorum
Last updated: Mon Apr 15 13:45:14 2019
Last change: Mon Apr 15 13:40:11 2019 by root via cibadmin on one
2 nodes configured
1 resource configured
Online: [ one two ]
Full list of resources:
ClusterMyApp (ocf::myapp:myapp-script): Started one
Failed Actions:
* ClusterMyApp_monitor_3 on one 'not running' (7): call=37,
status=complete, exitreason='',
last-rc-change='Mon Apr 15 13:40:12 2019', queued=0ms, exec=105ms
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
The start function in this script is as follows:
myapp_start() {
myapp_conf_check
local diagnostic=$?
if [ $diagnostic -ne $OCF_SUCCESS ]; then
return $diagnostic
fi
myapp_monitor
local state=$?
case $state in
$OCF_SUCCESS)
return $OCF_SUCCESS
;;
$OCF_NOT_RUNNING)
myapp_launch > /dev/null 2>&1
if [ $? -eq 0 ]; then
return $OCF_SUCCESS
fi
return $OCF_ERR_GENERIC
;;
*)
return $state
;;
esac
}
I know for a fact that, in one, myapp_launch gets invoked, and that its
exit value is 0. The function therefore returns OCF_SUCCESS, as it should.
However, if I understand things correctly, the log entries in two seem to
claim that the exit value of the script in one is OCF_NOT_RUNNING.
What's going on here? It's obviously something to do with myapp-script -
but, what?
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] [Announce] libqb 1.0.4 release
We are pleased to announce the release of libqb 1.0.4 Source code is available at: https://github.com/ClusterLabs/libqb/releases/download/v1.0.4/libqb-1.0.4.tar.xz Please used the signed .tar.gz or .tar.xz files with the version number in rather than the github-generated "Source Code" ones. This is a security update to 1.0.3. Files are now opened with O_EXCL and are placed in directories created by mkdtemp(). It is backwards compatible with 1.0.3. Chrissie ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] SBD as watchdog daemon
> 14 апр. 2019 г., в 10:12, Andrei Borzenkov написал(а): Thanks for explanation, I think this will be good addition to the SBD manual. (SBD manual need in this.) But my problem lies in other plain. I investigated SBD. A common watchdog is a much simple. One infinite loop, checks some tests and write to the watchdog device. Any mistakes, freeze or segfault and watchdog will fire. But SBD has another design. First of all there is not one infinite loop. There are three different processes, one is «inquisitor" and to other «servants» for corosync and pacemaker. And there is complex logic to check each other inside SBD. But the problem even is not here. Both the servants send to the inquisitor health heartbeat every second. But… They send health heartbeat not as result of checking corosync or pacemaker, as expected to be, but from the internal buffer variable «servant_health». And if corosync or pacemaker is frozen (can be emulated by `kill -s STOP`), this variable is never changed and the servants continue send to the inquisitor a good health status always. And this is a bug. I am looking a way to fix this. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
