Hi Ken,

The cause turned out.

When stonith is executed, stonithd sends results and notifications to crmd.
https://github.com/ClusterLabs/pacemaker/blob/0459f409580f41b35ce8ae31fb22e6370a508dab/fencing/remote.c#L402-L406

- when "result" is sent (calling do_local_reply()), too many stonith failures 
is checked in too_many_st_failures().
  
https://github.com/ClusterLabs/pacemaker/blob/0459f409580f41b35ce8ae31fb22e6370a508dab/crmd/te_callbacks.c#L638-L669
- when "notification" is sent (calling do_stonith_notify()), the number of 
failures is incremented in st_fail_count_increment().
  
https://github.com/ClusterLabs/pacemaker/blob/0459f409580f41b35ce8ae31fb22e6370a508dab/crmd/te_callbacks.c#L704-L726
From this, since checking is done before incrementing, the number of failures 
in "Too many failures (10) to fence" log does not match the number of actual 
failures.

I confirmed that the expected result will be obtained from the following 
changes.

# git diff
diff --git a/fencing/remote.c b/fencing/remote.c
index 4a47d49..3ff324e 100644
--- a/fencing/remote.c
+++ b/fencing/remote.c
@@ -399,12 +399,12 @@ handle_local_reply_and_notify(remote_fencing_op_t * op, 
xmlNode * data, int rc)
     reply = stonith_construct_reply(op->request, NULL, data, rc);
     crm_xml_add(reply, F_STONITH_DELEGATE, op->delegate);

-    /* Send fencing OP reply to local client that initiated fencing */
-    do_local_reply(reply, op->client_id, op->call_options & st_opt_sync_call, 
FALSE);
-
     /* bcast to all local clients that the fencing operation happend */
     do_stonith_notify(0, T_STONITH_NOTIFY_FENCE, rc, notify_data);

+    /* Send fencing OP reply to local client that initiated fencing */
+    do_local_reply(reply, op->client_id, op->call_options & st_opt_sync_call, 
FALSE);
+
     /* mark this op as having notify's already sent */
     op->notify_sent = TRUE;
     free_xml(reply);

Regards,
Kazunori INOUE

> -----Original Message-----
> From: Ken Gaillot [mailto:[email protected]]
> Sent: Wednesday, May 17, 2017 11:09 PM
> To: [email protected]
> Subject: Re: [ClusterLabs] Pacemaker's "stonith too many failures" log is not 
> accurate
> 
> On 05/17/2017 04:56 AM, Klaus Wenninger wrote:
> > On 05/17/2017 11:28 AM, 井上 和徳 wrote:
> >> Hi,
> >> I'm testing Pacemaker-1.1.17-rc1.
> >> The number of failures in "Too many failures (10) to fence" log does not 
> >> match the number of actual failures.
> >
> > Well it kind of does as after 10 failures it doesn't try fencing again
> > so that is what
> > failures stay at ;-)
> > Of course it still sees the need to fence but doesn't actually try.
> >
> > Regards,
> > Klaus
> 
> This feature can be a little confusing: it doesn't prevent all further
> fence attempts of the target, just *immediate* fence attempts. Whenever
> the next transition is started for some other reason (a configuration or
> state change, cluster-recheck-interval, node failure, etc.), it will try
> to fence again.
> 
> Also, it only checks this threshold if it's aborting a transition
> *because* of this fence failure. If it's aborting the transition for
> some other reason, the number can go higher than the threshold. That's
> what I'm guessing happened here.
> 
> >> After the 11th time fence failure, "Too many failures (10) to fence" is 
> >> output.
> >> Incidentally, stonith-max-attempts has not been set, so it is 10 by 
> >> default..
> >>
> >> [root@x3650f log]# egrep "Requesting fencing|error: Operation 
> >> reboot|Stonith failed|Too many failures"
> >> ##Requesting fencing : 1st time
> >> May 12 05:51:47 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:52:52 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:52:52 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 2nd time
> >> May 12 05:52:52 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:53:56 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:53:56 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 3rd time
> >> May 12 05:53:56 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:55:01 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:55:01 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 4th time
> >> May 12 05:55:01 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:56:05 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:56:05 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 5th time
> >> May 12 05:56:05 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:57:10 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:57:10 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 6th time
> >> May 12 05:57:10 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:58:14 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for [email protected]:
> No data available
> >> May 12 05:58:14 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 7th time
> >> May 12 05:58:14 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 05:59:19 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 05:59:19 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 8th time
> >> May 12 05:59:19 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 06:00:24 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for [email protected]:
> No data available
> >> May 12 06:00:24 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 9th time
> >> May 12 06:00:24 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 06:01:28 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 06:01:28 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 10th time
> >> May 12 06:01:28 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 06:02:33 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for
> [email protected]: No data available
> >> May 12 06:02:33 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >> ## 11th time
> >> May 12 06:02:33 rhel73-1 crmd[5269]:  notice: Requesting fencing (reboot) 
> >> of node rhel73-2
> >> May 12 06:03:37 rhel73-1 stonith-ng[5265]:   error: Operation reboot of 
> >> rhel73-2 by rhel73-1 for [email protected]:
> No data available
> >> May 12 06:03:37 rhel73-1 crmd[5269]: warning: Too many failures (10) to 
> >> fence rhel73-2, giving up
> >> May 12 06:03:37 rhel73-1 crmd[5269]:  notice: Transition aborted: Stonith 
> >> failed
> >>
> >> Regards,
> >> Kazunori INOUE
> 
> _______________________________________________
> Users mailing list: [email protected]
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Reply via email to