Re: [Pacemaker] meta failure-timeout: crashed resource is assumed to be Started?
FYI: I cannot reproduce this problem right now. I guess I made a mistake analyzing the logs. -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] meta failure-timeout: crashed resource is assumed to be Started?
Dear all, I did not get any response so far. Could you please find the time and tell me how the meta failure-timeout is supposed to work, in combination with monitor operations? Thanks, Carsten On Thu, Oct 16, 2014 at 05:06:41PM +0200, Carsten Otto wrote: Dear all, I configured meta failure-timeout=60sec on all of my resources. For the sake of simplicity, assume I have a group of two resources FIRST and SECOND (where SECOND is started after FIRST, surprise!). If now FIRST crashes, I see a failure, as expected. I also see that SECOND is stopped, as expected. Sadly, SECOND needs more than 60 seconds to stop. Thus, it can happen that the failure-timeout for FIRST is reached, and its failure is cleaned. This also is expected. The problem now is that after the 60sec timeout pacemaker assumes that FIRST is in the Started state. There is no indication about that in the log files, and the last monitor operation which ran just a few seconds before also indicated that FIRST is actually not running. As a consequence of the bug, pacemaker tries to re-start SECOND on the same system, which fails to start (as it depends on FIRST, which actually is not running). Only then the resources are started on the other system. So, my question is: Why does pacemaker assume that a previously failed resource is Started when the meta failure-timeout is triggered? Why is the monitor operation not invoked to determine the correct state? The corresponding lines of the log file, about a minute after FIRST crashed and the stop operation for SECOND was triggered: Oct 16 16:27:20 [2100] HOSTNAME [...] (monitor operation indicating that FIRST is not running) [...] Oct 16 16:27:23 [2104] HOSTNAME lrmd: info: log_finished: finished - rsc:SECOND action:stop call_id:123 pid:29314 exit-code:0 exec-time:62827ms queue-time:0ms Oct 16 16:27:23 [2107] HOSTNAME crmd: notice: process_lrm_event: LRM operation SECOND_stop_0 (call=123, rc=0, cib-update=225, confirmed=true) ok Oct 16 16:27:23 [2107] HOSTNAME crmd: info: match_graph_event: Action SECOND_stop_0 (74) confirmed on HOSTNAME (rc=0) Oct 16 16:27:23 [2107] HOSTNAME crmd: notice: run_graph: Transition 40 (Complete=5, Pending=0, Fired=0, Skipped=31, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-input-2937.bz2): Stopped Oct 16 16:27:23 [2107] HOSTNAME crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE - S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/225, version=0.1450.89) Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/226, version=0.1450.89) Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_config: On loss of CCM Quorum: Ignore Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: determine_online_status_fencing: Node HOSTNAME is active Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: determine_online_status: Node HOSTNAME is online [...] Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Re-initiated expired calculated failure FIRST_last_failure_0 (rc=7, magic=0:7;68:31:0:28c68203-6990-48fd-96cc-09f86e2b21f9) on HOSTNAME [...] Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: group_print: Resource Group: GROUP Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: native_print: FIRST (ocf::heartbeat:xxx): Started HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: native_print: SECOND (ocf::heartbeat:yyy):Stopped Thank you, Carsten ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- andrena objects ag Büro Frankfurt
[Pacemaker] meta failure-timeout: crashed resource is assumed to be Started?
Dear all, I configured meta failure-timeout=60sec on all of my resources. For the sake of simplicity, assume I have a group of two resources FIRST and SECOND (where SECOND is started after FIRST, surprise!). If now FIRST crashes, I see a failure, as expected. I also see that SECOND is stopped, as expected. Sadly, SECOND needs more than 60 seconds to stop. Thus, it can happen that the failure-timeout for FIRST is reached, and its failure is cleaned. This also is expected. The problem now is that after the 60sec timeout pacemaker assumes that FIRST is in the Started state. There is no indication about that in the log files, and the last monitor operation which ran just a few seconds before also indicated that FIRST is actually not running. As a consequence of the bug, pacemaker tries to re-start SECOND on the same system, which fails to start (as it depends on FIRST, which actually is not running). Only then the resources are started on the other system. So, my question is: Why does pacemaker assume that a previously failed resource is Started when the meta failure-timeout is triggered? Why is the monitor operation not invoked to determine the correct state? The corresponding lines of the log file, about a minute after FIRST crashed and the stop operation for SECOND was triggered: Oct 16 16:27:20 [2100] HOSTNAME [...] (monitor operation indicating that FIRST is not running) [...] Oct 16 16:27:23 [2104] HOSTNAME lrmd: info: log_finished: finished - rsc:SECOND action:stop call_id:123 pid:29314 exit-code:0 exec-time:62827ms queue-time:0ms Oct 16 16:27:23 [2107] HOSTNAME crmd: notice: process_lrm_event:LRM operation SECOND_stop_0 (call=123, rc=0, cib-update=225, confirmed=true) ok Oct 16 16:27:23 [2107] HOSTNAME crmd: info: match_graph_event: Action SECOND_stop_0 (74) confirmed on HOSTNAME (rc=0) Oct 16 16:27:23 [2107] HOSTNAME crmd: notice: run_graph:Transition 40 (Complete=5, Pending=0, Fired=0, Skipped=31, Incomplete=10, Source=/var/lib/pacemaker/pengine/pe-input-2937.bz2): Stopped Oct 16 16:27:23 [2107] HOSTNAME crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE - S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/225, version=0.1450.89) Oct 16 16:27:23 [2100] HOSTNAMEcib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/226, version=0.1450.89) Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_config:On loss of CCM Quorum: Ignore Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: determine_online_status_fencing: Node HOSTNAME is active Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: determine_online_status: Node HOSTNAME is online [...] Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: get_failcount_full: FIRST has failed 1 times on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Clearing expired failcount for FIRST on HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: notice: unpack_rsc_op: Re-initiated expired calculated failure FIRST_last_failure_0 (rc=7, magic=0:7;68:31:0:28c68203-6990-48fd-96cc-09f86e2b21f9) on HOSTNAME [...] Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: group_print: Resource Group: GROUP Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: native_print: FIRST (ocf::heartbeat:xxx): Started HOSTNAME Oct 16 16:27:23 [2106] HOSTNAMEpengine: info: native_print: SECOND (ocf::heartbeat:yyy):Stopped Thank you, Carsten -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started:
Re: [Pacemaker] Pacemaker on system with disk failure
Dear Andrew, please find the time to have a look at this. Thank you, Carsten -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
On Tue, Sep 23, 2014 at 10:14:33AM -0400, Digimer wrote: You don't have real fencing configured, by the looks of it. Without real, working fencing, recovery can be unpredictable. Can you set that up and see if the problem goes away? I now have real fencing which also is fully automatic, non-manual. The problem still exists just as described in previous mails. The only error seen by the surviving node is that the stonith resource of the diskless node failed. However, this does not cause a switchover. As said earlier, the services are not monitored anymore, and they do not work. Yet, the resources stay on the diskless node, as if nothing happened. As far as I understand the situation, this is a grave error. A mere disk failure causes the whole setup to be in a failed state. There is no working monitoring, no switchover happens, ... From a client's perspective there's no difference to a non-redundant setup. If you find the time, please try it out yourself. Just pull the cable from the/all disks which provide the root filesystem. Best regards, Carsten -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
Dear John, thank you for confirming the problem. Your script might do the job, although I see that some files like echo/true/sleep/cron might not be available - but I can really work with that. Best regards, Carsten -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
Dear John,
On Thu, Sep 25, 2014 at 10:03:27AM -0400, John Lauro wrote:
One of the reasons I like ksh is that true, echo, and sleep (among
many others) are all builtin, so you don't need those commands on the
filesystem, so the script is less likely to fail if the filesystem
fails... that said you probably don't have ksh installed by default.
Thanks for the hint! I just wrote a simple watchdog resource agent and
the corresponding shell script which successfully reboots a server when
the disk fails.
I provided my solution in the attachment.
Put crude-watchdog.sh in /root/, and put crude-watchdog in
/usr/lib/ocf/resource.d/heartbeat/.
In my two node cluster I used these commands to let this watchdog run on
all two machines:
pcs resource create WATCHDOG ocf:heartbeat:crude-watchdog
pcs resource clone WATCHDOG
Best regards,
Carsten
--
andrena objects ag
Büro Frankfurt
Clemensstr. 8
60487 Frankfurt
Tel: +49 (0) 69 977 860 38
Fax: +49 (0) 69 977 860 39
http://www.andrena.de
Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn
Aufsichtsratsvorsitzender: Rolf Hetzelberger
Sitz der Gesellschaft: Karlsruhe
Amtsgericht Mannheim, HRB 109694
USt-IdNr. DE174314824
Bitte beachten Sie auch unsere anstehenden Veranstaltungen:
http://www.andrena.de/events
crude-watchdog.sh
Description: Bourne shell script
#!/bin/sh
: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}
. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs
SCRIPT=/root/crude-watchdog.sh
meta_data() {
cat END
?xml version=1.0?
!DOCTYPE resource-agent SYSTEM ra-api-1.dtd
resource-agent name=crude-watchdog version=1.0
version1.0/version
longdesc lang=en
This agent reboots the system if the root file system stops working.
/longdesc
shortdesc lang=en
This agent reboots the system if the root file system stops working.
/shortdesc
parameters
/parameters
actions
action name=starttimeout=20 /
action name=stop timeout=20 /
action name=monitor timeout=20 interval=10 depth=0 /
action name=reload timeout=20 /
action name=migrate_to timeout=20 /
action name=migrate_from timeout=20 /
action name=meta-datatimeout=5 /
action name=validate-all timeout=20 /
/actions
/resource-agent
END
}
###
watchdog_usage() {
cat END
usage: $0 {start|stop|monitor|migrate_to|migrate_from|validate-all|meta-data}
Expects to have a fully populated OCF RA-compliant environment set.
END
}
watchdog_start() {
watchdog_monitor
if [ $? = $OCF_SUCCESS ]; then
return $OCF_SUCCESS
fi
nohup $SCRIPT
}
watchdog_stop() {
watchdog_monitor
if [ $? = $OCF_SUCCESS ]; then
killall crude-watchdog.sh
fi
watchdog_monitor
if [ $? = $OCF_SUCCESS ]; then
return $OCF_ERR_GENERIC
fi
return $OCF_SUCCESS
}
watchdog_monitor() {
RES=`ps aux | grep crude-watchdog.sh | grep -v grep -q`
if [ $? = 0 ]; then
return $OCF_SUCCESS
fi
return $OCF_NOT_RUNNING
}
watchdog_validate() {
if [ -x $SCRIPT ]; then
return $OCF_SUCCESS
fi
return $OCF_ERR_ARGS
}
case $__OCF_ACTION in
meta-data) meta_data
exit $OCF_SUCCESS
;;
start) watchdog_start;;
stop) watchdog_stop;;
monitor)watchdog_monitor;;
migrate_to) ocf_log info Migrating ${OCF_RESOURCE_INSTANCE} to
${OCF_RESKEY_CRM_meta_migrate_target}.
watchdog_stop
;;
migrate_from) ocf_log info Migrating ${OCF_RESOURCE_INSTANCE} from
${OCF_RESKEY_CRM_meta_migrate_source}.
watchdog_start
;;
reload) ocf_log info Reloading ${OCF_RESOURCE_INSTANCE} ...
;;
validate-all) watchdog_validate;;
usage|help) watchdog_usage
exit $OCF_SUCCESS
;;
*) watchdog_usage
exit $OCF_ERR_UNIMPLEMENTED
;;
esac
rc=$?
ocf_log debug ${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc
exit $rc
signature.asc
Description: Digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
The web archive seems to not like my shell script, so here it is again.
#!/bin/ksh
TESTFILE=/root/.testing
TRIGGER=/proc/sysrq-trigger
while true ; do
/bin/touch ${TESTFILE} || echo b ${TRIGGER}
[ ! -f ${TESTFILE} ] echo b ${TRIGGER}
sleep 1
done
--
andrena objects ag
Büro Frankfurt
Clemensstr. 8
60487 Frankfurt
Tel: +49 (0) 69 977 860 38
Fax: +49 (0) 69 977 860 39
http://www.andrena.de
Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn
Aufsichtsratsvorsitzender: Rolf Hetzelberger
Sitz der Gesellschaft: Karlsruhe
Amtsgericht Mannheim, HRB 109694
USt-IdNr. DE174314824
Bitte beachten Sie auch unsere anstehenden Veranstaltungen:
http://www.andrena.de/events
signature.asc
Description: Digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
[Pacemaker] Pacemaker on system with disk failure
Hello, I run Corosync + Pacemaker + DRBD in a two node cluster, where all resources are part of a group/colocated with DRBD (DRBD + virtual IP + filesystem + ...). To test my configuration, I currently have two nodes with only a single disk drive. This drive is the only LVM physical drive in a LVM volume group, where the Linux system resides on some logical volumes and the disk exported by DRBD is another logical volume. When I now unplug power of the disk drive on the node running the resources (DRBD is primary), this gets noticed by DRBD (diskless). Furthermore, I notice that my services do not work anymore (which is understandable without a working disk drive). However, in my experiments one of the following problems occurs: 1) The services are stopped and DRBD is demoted (according to pcs status and pacemaker.log), however according to /proc/drbd on the surviving node, the diskless node still is running as primary. As a consequence, I see failing attempts to promote on the surviver node: drbd(DRBD)[1797]: 2014/09/23_14:35:56 ERROR: disk0: Called drbdadm -c /etc/drbd.conf primary disk0 drbd(DRBD)[1797]: 2014/09/23_14:35:56 ERROR: disk0: Exit code 11 The problem here seems to be: crmd: info: match_graph_event: Action DRBD_demote_0 (12) confirmed on diskless_node (rc=0) While this demote operation obviously should not be confirmed, I also strongly believe that running the stop operations of the standard resources works without having access to the resource agent scripts (which are on the failed disk) and the tools used by them. 2) My services do not work anymore, but nothing happens in the cluster. Everything looks like it did before the failure, with the only difference that /proc/drbd shows Diskless and some oos. It seems corosync/pacemaker sends out all is well to the DC, while internally (due to the missing disk) nothing works. I guess that running all sorts of monitor scripts is problematic without having access to the actual files, so I'd like to see some sort of failure communicated from the diskless node to the surviving node (or, having the surviving node come to the same conclusion due to some timeout). Is this buggy behaviour? How should a node behave if all disks stopped working? I can reproduce this. If you need details about the configuration or more output from pacemaker.log, please just tell me so. The versions reported by Centos 7: corosync 2.3.3-2.el7 pacemaker 1.1.10-32.el7_0 drbd 8.4.5-1.el7.elrepo Thank you, Carsten -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
On Tue, Sep 23, 2014 at 03:39:45PM +0200, Carsten Otto wrote: While this demote operation obviously should not be confirmed, I also strongly believe that running the stop operations of the standard ^^^ disbelieve resources works without having access to the resource agent scripts (which are on the failed disk) and the tools used by them. -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] Pacemaker on system with disk failure
On Tue, Sep 23, 2014 at 09:50:12AM -0400, Digimer wrote:
Can you share your pacemaker and drbd configurations please?
drbd.d/global_comman.conf:
global {
usage-count no;
}
common {
protocol C;
handlers {
split-brain /usr/lib/drbd/notify-split-brain.sh root;
out-of-sync /usr/lib/drbd/notify-out-of-sync.sh root;
}
}
drbd.d/disk0.res:
resource disk0 {
syncer {
rate 10M;
csums-alg sha1;
}
disk {
on-io-error detach;
fencing resource-only;
}
handlers {
before-resync-target
/usr/lib/drbd/snapshot-resync-target-lvm.sh;
after-resync-target
/usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
/usr/lib/drbd/crm-unfence-peer.sh;
fence-peer /usr/lib/drbd/crm-fence-peer.sh;
split-brain /usr/lib/drbd/notify-split-brain.sh root;
}
net {
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}
device/dev/drbd0;
disk /dev/centos/drbd-lv;
meta-disk internal;
on node_a {
address 192.168.69.89:7789;
}
on node_b {
address 192.168.69.90:7789;
}
}
pcs resource --full:
Master: DRBD_MASTER
Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
notify=true failure-timeout=60sec
Resource: DRBD (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=disk0
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=240 (DRBD-start-timeout-240)
promote interval=0s timeout=90 (DRBD-promote-timeout-90)
demote interval=0s timeout=90 (DRBD-demote-timeout-90)
stop interval=0s timeout=100 (DRBD-stop-timeout-100)
monitor interval=9 role=Master (DRBD-monitor-interval-9)
monitor interval=11 role=Slave (DRBD-monitor-interval-11)
Group: GROUP
Resource: VIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.69.48 cidr_netmask=32
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=20s (VIP-start-timeout-20s)
stop interval=0s timeout=5s (VIP-stop-timeout-5s)
monitor interval=10sec (VIP-monitor-interval-10sec)
Resource: FS (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd0 directory=/mnt/drbd options=noatime,nodiratime
fstype=ext4
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=60 (FS-start-timeout-60)
stop interval=0s timeout=10s (FS-stop-timeout-10s)
monitor interval=5sec (FS-monitor-interval-5sec)
Resource: PGSQL (class=ocf provider=heartbeat type=pgsql)
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=120 (PGSQL-start-timeout-120)
stop interval=0s timeout=120 (PGSQL-stop-timeout-120)
promote interval=0s timeout=120 (PGSQL-promote-timeout-120)
demote interval=0s timeout=120 (PGSQL-demote-timeout-120)
monitor interval=10sec (PGSQL-monitor-interval-10sec)
Resource: ASTERISK (class=ocf provider=heartbeat type=asterisk)
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=20 (ASTERISK-start-timeout-20)
monitor interval=10sec (ASTERISK-monitor-interval-10sec)
stop interval=0s timeout=1 (ASTERISK-stop-timeout-1)
Resource: TOMCAT (class=ocf provider=heartbeat type=tomcat)
Attributes: java_home=/usr/java/latest/ catalina_home=/usr/share/tomcat
statusurl=http://localhost:8080/xxx/
Meta Attrs: failure-timeout=60sec
Operations: start interval=0s timeout=60s (TOMCAT-start-timeout-60s)
stop interval=0s timeout=20s (TOMCAT-stop-timeout-20s)
monitor interval=10sec (TOMCAT-monitor-interval-10sec)
pcs constraint --full:
Location Constraints:
Resource: DRBD_MASTER
Constraint: drbd-fence-by-handler-disk0-DRBD_MASTER
Rule: score=-INFINITY role=Master
(id:drbd-fence-by-handler-disk0-rule-DRBD_MASTER)
Expression: #uname ne node_a
(id:drbd-fence-by-handler-disk0-expr-DRBD_MASTER)
Resource: STONITH_A
Disabled on: node_b (score:-INFINITY)
(id:location-STONITH_A-node_b--INFINITY)
Ordering Constraints:
promote DRBD_MASTER then start GROUP (Mandatory)
(id:order-DRBD_MASTER-GROUP-mandatory)
Colocation Constraints:
GROUP with DRBD_MASTER (INFINITY) (rsc-role:Started) (with-rsc-role:Master)
(id:colocation-GROUP-DRBD_MASTER-INFINITY)
pcs stonith --full:
STONITH_A (stonith:fence_dummy): Started
Resource: STONITH_A (class=stonith type=fence_dummy)
Attributes: passwd=x pcmk_host_list=node_b
Operations: monitor interval=60s (STONITH_A-monitor-interval-60s)
[Note: The problem also happens without stonith and with a proper stonith
Re: [Pacemaker] Pacemaker on system with disk failure
On Tue, Sep 23, 2014 at 10:14:33AM -0400, Digimer wrote: You don't have real fencing configured, by the looks of it. Without real, working fencing, recovery can be unpredictable. Can you set that up and see if the problem goes away? I do have real, working fencing - although manual for testing purposes. However, as I said, the problem also appears without any fencing configured in the cluster. Furthermore, according to the logs and the dummy script itself, it is never tried to fence. (To clarify, the dummy always reports the correct status and the script works, although I need to press the power button manually) -- andrena objects ag Büro Frankfurt Clemensstr. 8 60487 Frankfurt Tel: +49 (0) 69 977 860 38 Fax: +49 (0) 69 977 860 39 http://www.andrena.de Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn Aufsichtsratsvorsitzender: Rolf Hetzelberger Sitz der Gesellschaft: Karlsruhe Amtsgericht Mannheim, HRB 109694 USt-IdNr. DE174314824 Bitte beachten Sie auch unsere anstehenden Veranstaltungen: http://www.andrena.de/events signature.asc Description: Digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
