Thanks for your comments. I have attached an updated version, although
not everything has been worked in yet (see below).
Lars Ellenberg schrieb:
[..]
+ RX_PACKETS_NEW="`get_rx_packets`"
+ RX_PACKETS_OLD="$RX_PACKETS_NEW"
^^^ assign old here, only
ok, done.
+ for n in `seq $OCF_RESKEY_pktcnt_timeout`; do
+ sleep 1
maybe even sub second sleep?
likely we are busy anyways,
so we should see some packets very quickly
Sounds reasonable to me. Would 0.1s steps be okay? Although I would
think it doesn't really matter that much.
+ RX_PACKETS_OLD="$RX_PACKETS_NEW"
not here. because, if it changed, you return anyways.
if not, well, it has not changed right?
I like this :-) done.
+ RX_PACKETS_NEW="`get_rx_packets`"
+ ocf_log debug "RX_PACKETS_OLD: $RX_PACKETS_OLD RX_PACKETS_NEW:
$RX_PACKETS_NEW"
+ if [ "$RX_PACKETS_OLD" -ne "$RX_PACKETS_NEW" ]; then
+ ocf_log debug "we received some packages."
s/packages/packets/
ok.
+ return
explicitly return 0 here, or you will be returning the result of
ocf_log, and you don't want to cause recovery action because the logging
failed somewhere.
ok.
(Funny I that have never stumbled about "return" returning the result of
the last used function...)
+ fi
+ done
+ return 1
+}
I do like this packet counter monitoring.
Thanks! :)
+# returns list of chached ARP entries for $NIC
s/chached/cached
ok.
[..]
+ $IP2UTIL -s neighbour show dev $NIC \
+ | grep -v "^$BASEIP " \
my own ip is not included anyways,
at least on my box?
I just wanted to make sure and didn't check. But now I think you are
right - my own IP should never come up in the arp cache.
do you want to grep REACHABLE here, too?
not sure.
I don't see a problem with also trying stale, delay and probe entries,
but maybe we should filter out at least the "incomplete" and "failed"
state? Or others to?
Maybe we should somehow make sure that we are only trying to arping IPs
which are on the subnet for the IP we manage. There could be several IPs
with different subnets assigned to one device. What do you think?
+ | tr '/' ' ' | sort -n -k8 | cut -d' ' -f1 \
| sort -t/ -k2,2n | cut -d' ' -f1 \
thanks.
+ | head -n $OCF_RESKEY_arping_cache_entries | tr '\n' ' '
what for do you tr \n ' ' ?
no need.
right. I thought that it was needed, because piping the command into
less always gave me newlines.
[..]
+do_arping () {
+ if $DEBUG ; then arping -c $OCF_RESKEY_arping_count -I $NIC $1; return
$?; fi
+ arping -q -c $OCF_RESKEY_arping_count -I $NIC $1
I think you should add the source IP.
I will see into this.
Is this the only arping out there? I think there are at least two
variants, with slightly different options.
Ooops, I did not know that.
If your invokation is compatible, add a comment that it is.
If not, find a way to deal with both.
Will go and investigate this...
+ return $?
return $? can go, functions return with the exit code of the last command
anyways.
Glad I learned this now. :)
[..]
+ip_monitor_if_check () {
+ local CURRENT_CHECK_LEVEL
+ CURRENT_CHECK_LEVEL=$1
+ # always check link status first
+ link_status="`get_link_status`"
+ ocf_log debug "link_status: $link_status"
+ case $link_status in
+ UP)
+ if [ "$CURRENT_CHECK_LEVEL" -eq 0 ]; then
+ return $OCF_SUCCESS
+ fi
+ ;;
+ DOWN)
+ # remove address from NIC
+ return $OCF_NOT_RUNNING
+ ;;
+ *) # this should not happen.
+ return $OCF_ERR_GENERIC
+ ;;
+ esac
# $CURRENT_CHECK_LEVEL > 0 and link_status UP is implicit from now on
added.
[..]
+ # watch for packet counter changes
+ if [ "$CURRENT_CHECK_LEVEL" -eq 10 ]; then
Do you want -le here?
if not, add a comment why you want -eq.
doesn't make any difference, but -le definitely looks more consistent.
[..]
+ # check arping_ip_list
+ if [ $CURRENT_CHECK_LEVEL -le 19 ]; then
+ ocf_log debug "check arping_ip_list"
+ if [ "$OCF_RESKEY_arping_ip_list" != "" ]; then
No need to check for != "",
the for loop will just not execute if it is.
ok.
+ for ip in $OCF_RESKEY_arping_ip_list; do
+ do_arping $ip && return $OCF_SUCCESS
+ done
+ fi
+ fi
+
+ # check arping ARP cache entries
+ if [ $CURRENT_CHECK_LEVEL -le 20 ]; then
+ ocf_log debug "check arping ARP cache entries"
+ if [ "$OCF_RESKEY_arping_ip_list" != "" ]; then
no need here either, probably even wrong.
You want this even if there is no arping_ip_list explicitly set, right?
Right. Ooops, c&p error.
+ for ip in `get_arp_list`; do
+ do_arping $ip && return $OCF_SUCCESS
+ done
+ fi
+ fi
+
+ # watch for packet counter changes in promiscios mode
+ if [ $CURRENT_CHECK_LEVEL -le 30 ]; then
+ ocf_log debug "watch for packet counter changes in promiscios mode"
+ # be sure switch off promiscios mode in any case
+ trap "$IP2UTIL link set dev $NIC promisc off; exit" INT TERM
EXIT
I don't like this.
You assume that the dev does not operate in promisc mode.
This will break the (presumably minority of) cases that actually
run promisc intentionally during "normal operation".
Oh yes, you are right. I honestly just didn't think about this.
+ $IP2UTIL link set dev $NIC promisc on
+ watch_pkt_counter && return $OCF_SUCCESS
+ $IP2UTIL link set dev $NIC promisc off
+ trap - INT TERM EXIT
+ fi
How about you remember the packet stats first thing,
then do all the other checks,
finally you do a broadcast ping or something,
then go again waiting for packets,
comparing with the first remembered count?
This sounds like a good idea to me, although I'm not sure about that
"broadcast ping or something". Maybe as an option, but I wouldn't want
to do any broadcasts by default. (and I have no idea, what that other
something could be)
If you really want the promisc check,
maybe first check if it was on,
and if so, leave it on.
Or make that an advanced option: keep_promisc_on...
Maybe I'm just wrong here, and it's ok as you have it.
No, you are right. I think will add a check and only switch the mode if
necessary. But right now I do not yet see a way to omit a possible race
condition where someone switches to promisc mode just right after the
check, so that we switch it off afterwards. I will have to think a bit
about this.
About the advanced option: I would rather not alter the mode by default,
but maybe offer an option always_switch_promisc_off. Although I don't
know how useful this would be.
+ # looks like it's not working (for whatever reason)
+ # remove address from NIC
+ return $OCF_NOT_RUNNING
+}
+
Enough for today...
More than enough for me to do for now!
Thanks again for this valuable contribution.
Thank you very much for your comments.
Sorry for the slow replies -- I'm on part time right now and quite busy.
But you can be assured that I will keep working on this.
Robert.
Medium: IPaddr2: add real monitor capabilities
expand the monitor function to do some real monitoring
(check link status, arping, packat flow)
[first draft version]
diff -r 41cd73a9dedb heartbeat/IPaddr2
--- a/heartbeat/IPaddr2 Wed Feb 02 15:20:38 2011 +0100
+++ b/heartbeat/IPaddr2 Fri Feb 11 18:47:18 2011 +0100
@@ -56,6 +56,11 @@
# OCF_RESKEY_arp_count
# OCF_RESKEY_arp_bg
# OCF_RESKEY_arp_mac
+# OCF_RESKEY_startup_check_level
+# OCF_RESKEY_pktcnt_timeout
+# OCF_RESKEY_arping_ip_list
+# OCF_RESKEY_arping_count
+# OCF_RESKEY_arping_cache_entries
#
# OCF_RESKEY_CRM_meta_clone
# OCF_RESKEY_CRM_meta_clone_max
@@ -87,6 +92,42 @@
It can add an IP alias, or remove one.
In addition, it can implement Cluster Alias IP functionality
if invoked as a clone resource.
+
+The standard start monitor operation of depth 0 (also known as probe)
+checks if the link on the interface is up.
+If you want deeper tests, set startup_check_level (for start)
+resp. OCF_CHECK_LEVEL (for monitor) to one of the following values:
+
+09: check for nonempty ARP Cache
+
+This checks wether the ARP cache for the interface is not empty.
+ARP cache entries are usually kept for about 5-20 minutes
+depending on the setup.
+If unsuccessful the tests for higher OCF_CHECK_LEVELs are run.
+
+10: watch for packet counter changes
+
+This checks wether packets are received on the interface
+(in non promiscious mode, so STP traffic is not seen).
+
+19: check arping_ip_list
+
+This checks wether any of the IPs given in arping_ip_list answers
+an ARP REQUEST (arping).
+If unsuccessful the tests for higher OCF_CHECK_LEVELs are run.
+
+20: check arping ARP Cache entries
+
+This checks wether any of the IPs found in the lokal ARP cache answers
+an ARP REQUEST (arping).
+If unsuccessful the tests for higher OCF_CHECK_LEVELs are run.
+
+30: watch for packet counter changes in promiscios mode
+
+This checks wether packets are received on the interface
+in promiscious mode.
+Since eg. (R)STP packages are also counted in this mode this test
+should always succeed if a STP-using Switch is connected
</longdesc>
<shortdesc lang="en">Manages virtual IPv4 addresses (Linux specific
version)</shortdesc>
@@ -250,6 +291,51 @@
<content type="boolean" default="false"/>
</parameter>
+<parameter name="startup_check_level">
+<longdesc lang="en">
+Defines which checks to do on the interface at start of the resource.
+For more details look at the general description of this RA.
+</longdesc>
+<shortdesc lang="en">startup check level for the interface</shortdesc>
+<content type="integer" default="0"/>
+</parameter>
+
+<parameter name="pktcnt_timeout">
+<longdesc lang="en">
+Timeout for the packet counter. Stop listening for packet counter
+changes after the given number of seconds
+</longdesc>
+<shortdesc lang="en">packet counter timeout</shortdesc>
+<content type="integer" default="5"/>
+</parameter>
+
+<parameter name="arping_ip_list">
+<longdesc lang="en">
+List of IPs to check for ARP REQUEST (arping) answers.
+Only IP addresses are allowed, not hostnames.
+</longdesc>
+<shortdesc lang="en">arping IP list</shortdesc>
+<content type="string" default=""/>
+</parameter>
+
+<parameter name="arping_count">
+<longdesc lang="en">
+Number of ARP REQUEST packets to send for every IP.
+Usually one ARP REQUEST (arping) is send
+</longdesc>
+<shortdesc lang="en">Number of arpings per IP</shortdesc>
+<content type="integer" default="1"/>
+</parameter>
+
+<parameter name="arping_cache_entries">
+<longdesc lang="en">
+Maximum number of IPs from ARP cache list to check for
+ARP REQUEST (arping) answers. Newest entries are tried first.
+</longdesc>
+<shortdesc lang="en">Number of ARP cache entries to try</shortdesc>
+<content type="integer" default="5"/>
+</parameter>
+
</parameters>
<actions>
<action name="start" timeout="20s" />
@@ -391,6 +477,37 @@
fi
IP_CIP_FILE="/proc/net/ipt_CLUSTERIP/$BASEIP"
fi
+
+ : ${OCF_RESKEY_startup_check_level:="0"}
+ if ! ocf_is_decimal "$OCF_RESKEY_startup_check_level"; then
+ ocf_log err "Invalid OCF_RESKEY_startup_check_level
[$OCF_RESKEY_startup_check_level]"
+ exit $OCF_ERR_CONFIGURED
+ fi
+ : ${OCF_RESKEY_pktcnt_timeout:="5"}
+ if ! ocf_is_decimal "$OCF_RESKEY_pktcnt_timeout"; then
+ ocf_log err "Invalid OCF_RESKEY_pktcnt_timeout
[$OCF_RESKEY_pktcnt_timeout]"
+ exit $OCF_ERR_CONFIGURED
+ fi
+ : ${OCF_RESKEY_arping_count:="1"}
+ if ! ocf_is_decimal "$OCF_RESKEY_arping_count"; then
+ ocf_log err "Invalid OCF_RESKEY_arping_count
[$OCF_RESKEY_arping_count]"
+ exit $OCF_ERR_CONFIGURED
+ fi
+ : ${OCF_RESKEY_arping_cache_entries:="5"}
+ if ! ocf_is_decimal "$OCF_RESKEY_arping_cache_entries"; then
+ ocf_log err "Invalid OCF_RESKEY_arping_cache_entries
[$OCF_RESKEY_arping_cache_entries]"
+ exit $OCF_ERR_CONFIGURED
+ fi
+ : ${OCF_RESKEY_arping_ip_list:=""}
+ # OCF_RESKEY_arping_ip_list should only contain valid IP addresses
+ for ip in $OCF_RESKEY_arping_ip_list ; do
+ for i in `echo $ip | sed -e "s/\.\(.*\)\.\(.*\)\./ \1 \2 /g"`;do
+ if ! ocf_is_decimal "$i" || [ "$i" -lt "0" -o "255" -lt
"$i" ]; then
+ ocf_log err "Invalid OCF_RESKEY_arping_ip_list
[$OCF_RESKEY_arping_ip_list]"
+ exit $OCF_ERR_CONFIGURED
+ fi
+ done
+ done
}
#
@@ -610,6 +727,153 @@
exit $OCF_ERR_GENERIC
}
+# get the link status on $NIC
+# returns UP or DOWN or whatever ip reports (UNKNOWN?)
+get_link_status () {
+ $IP2UTIL -o link show dev "$NIC" \
+ | sed 's/.* state \([^ ]*\) .*/\1/'
+}
+
+# returns the number of received rx packets on $NIC
+get_rx_packets () {
+ ocf_log debug "$IP2UTIL -o -s link show dev $NIC"
+ $IP2UTIL -o -s link show dev "$NIC" \
+ | sed 's/.* RX: [^0-9]*[0-9]* *\([0-9]*\) .*/\1/'
+ # the first number after RX: ist the # of bytes ,
+ # the second is the # of packets received
+}
+
+# watch for packet counter changes for max. OCF_RESKEY_pktcnt_timeout seconds
+# returns immedeately with return code 0 if any packets were received
+# otherwise 1 is returned
+watch_pkt_counter () {
+ local RX_PACKETS_NEW
+ local RX_PACKETS_OLD
+ RX_PACKETS_OLD="`get_rx_packets`"
+ for n in `seq $OCF_RESKEY_pktcnt_timeout`; do
+ sleep 1
+ RX_PACKETS_NEW="`get_rx_packets`"
+ ocf_log debug "RX_PACKETS_OLD: $RX_PACKETS_OLD
RX_PACKETS_NEW: $RX_PACKETS_NEW"
+ if [ "$RX_PACKETS_OLD" -ne "$RX_PACKETS_NEW" ]; then
+ ocf_log debug "we received some packets."
+ return 0
+ fi
+ done
+ return 1
+}
+
+# returns list of cached ARP entries for $NIC
+# sorted by age ("last confirmed")
+# max. OCF_RESKEY_arping_cache_entries entries
+get_arp_list () {
+ $IP2UTIL -s neighbour show dev $NIC \
+ | sort -t/ -k2,2n | cut -d' ' -f1 \
+ | head -n $OCF_RESKEY_arping_cache_entries
+ # the "used" entries in `ip -s neighbour show` are:
+ # "last used"/"last confirmed"/"last updated"
+}
+
+# arping the IP given as argument $1 on $NIC
+# until OCF_RESKEY_arping_count answers are received
+do_arping () {
+ # TODO: add the source IP
+ # TODO: check for diffenrent arping versions out there
+ if $DEBUG ; then arping -c $OCF_RESKEY_arping_count -I $NIC $1; return
$?; fi
+ arping -q -c $OCF_RESKEY_arping_count -I $NIC $1
+ # return with the exit code of the arping command
+ return
+}
+
+#
+# Check the interface depending on the level given as parameter:
+# (this will be startup_check_level on start or OCF_CHECK_LEVEL on
monitor)
+#
+# 09: check for nonempty ARP cache
+# 10: watch for packet counter changes
+#
+# 19: check arping_ip_list
+# 20: check arping ARP cache entries
+#
+# 30: watch for packet counter changes in promiscios mode
+#
+# If unsuccessfull in levels 18 and above,
+# the tests for higher check levels are run.
+#
+ip_monitor_if_check () {
+ # TODO: maybe remember the packet stats first thing,
+ # and use this to compare later on
+ local CURRENT_CHECK_LEVEL
+ CURRENT_CHECK_LEVEL=$1
+ # always check link status first
+ link_status="`get_link_status`"
+ ocf_log debug "link_status: $link_status"
+ case $link_status in
+ UP)
+ if [ "$CURRENT_CHECK_LEVEL" -eq 0 ]; then
+ return $OCF_SUCCESS
+ fi
+ ;;
+ DOWN)
+ # remove address from NIC
+ return $OCF_NOT_RUNNING
+ ;;
+ *) # this should not happen.
+ return $OCF_ERR_GENERIC
+ ;;
+ esac
+
+ # $CURRENT_CHECK_LEVEL > 0 and link_status UP is implicit from now on
+
+ # check for nonempty ARP cache
+ if [ $CURRENT_CHECK_LEVEL -le 9 ]; then
+ ocf_log debug "check for nonempty ARP cache"
+ if [ "`get_arp_list`" != "" ]; then
+ return $OCF_SUCCESS
+ fi
+ fi
+
+ # watch for packet counter changes
+ if [ "$CURRENT_CHECK_LEVEL" -le 10 ]; then
+ ocf_log debug "watch for packet counter changes"
+ watch_pkt_counter && return $OCF_SUCCESS
+ fi
+
+ # check arping_ip_list
+ if [ $CURRENT_CHECK_LEVEL -le 19 ]; then
+ ocf_log debug "check arping_ip_list"
+ for ip in $OCF_RESKEY_arping_ip_list; do
+ do_arping $ip && return $OCF_SUCCESS
+ done
+ fi
+ fi
+
+ # check arping ARP cache entries
+ # TODO: Maybe we should somehow make sure that we are only trying to
+ # arping IPs which are on the subnet for the IP we manage.
+ if [ $CURRENT_CHECK_LEVEL -le 20 ]; then
+ ocf_log debug "check arping ARP cache entries"
+ for ip in `get_arp_list`; do
+ do_arping $ip && return $OCF_SUCCESS
+ done
+ fi
+
+ # watch for packet counter changes in promiscios mode
+ if [ $CURRENT_CHECK_LEVEL -le 30 ]; then
+ ocf_log debug "watch for packet counter changes in promiscios
mode"
+ # be sure switch off promiscios mode in any case
+ # TODO: check first, wether promisc is already on and leave it
untouched.
+ trap "$IP2UTIL link set dev $NIC promisc off; exit" INT TERM
EXIT
+ $IP2UTIL link set dev $NIC promisc on
+ watch_pkt_counter && return $OCF_SUCCESS
+ $IP2UTIL link set dev $NIC promisc off
+ trap - INT TERM EXIT
+ fi
+
+ # looks like it's not working (for whatever reason)
+ # remove address from NIC
+ return $OCF_NOT_RUNNING
+}
+
#######################################################################
ip_usage() {
@@ -639,7 +903,14 @@
local ip_status=`ip_served`
if [ "$ip_status" = "ok" ]; then
- exit $OCF_SUCCESS
+ ip_monitor_if_check $OCF_RESKEY_startup_check_level
+ local mon_rc=$?
+# if [ "$mon_rc" -eq "$OCF_NOT_RUNNING" ]; then
+# # make sure IP address is removed from interface
+# ip_stop
+# fi
+ ocf_log info "Monitoring return code: $mon_rc"
+ exit $mon_rc
fi
if [ -n "$IP_CIP" ] && [ $ip_status = "no" ] || [ $ip_status =
"partial2" ]; then
@@ -690,7 +961,14 @@
fi
;;
esac
- exit $OCF_SUCCESS
+ ip_monitor_if_check $OCF_RESKEY_startup_check_level
+ local mon_rc=$?
+# if [ "$mon_rc" -eq "$OCF_NOT_RUNNING" ]; then
+# # make sure IP address is removed from interface
+# ip_stop
+# fi
+ ocf_log info "Monitoring return code: $mon_rc"
+ exit $mon_rc
}
ip_stop() {
@@ -717,12 +995,12 @@
if [ $ip_status = "no" ]; then
: Requested interface not in use
- exit $OCF_SUCCESS
+ return $OCF_SUCCESS
fi
if [ -n "$IP_CIP" ] && [ $ip_status != "partial2" ]; then
if [ $ip_status = "partial" ]; then
- exit $OCF_SUCCESS
+ return $OCF_SUCCESS
fi
echo "-$IP_INC_NO" >$IP_CIP_FILE
if [ "x$(cat $IP_CIP_FILE)" = "x" ]; then
@@ -746,7 +1024,7 @@
if [ "$ip_del_if" = "yes" ]; then
delete_interface $BASEIP $NIC $NETMASK
if [ $? -ne 0 ]; then
- exit $OCF_ERR_GENERIC
+ return $OCF_ERR_GENERIC
fi
if [ "$LVS_SUPPORT" = 1 ]; then
@@ -754,23 +1032,33 @@
fi
fi
- exit $OCF_SUCCESS
+ return $OCF_SUCCESS
}
ip_monitor() {
- # TODO: Implement more elaborate monitoring like checking for
- # interface health maybe via a daemon like FailSafe etc...
+ # TODO: Look wether ip_monitor_if_check can also be used
+ # for partial and partial2 states
local ip_status=`ip_served`
case $ip_status in
ok)
- return $OCF_SUCCESS
+ ip_monitor_if_check $OCF_CHECK_LEVEL
+ local mon_rc=$?
+# if [ "$mon_rc" -eq "$OCF_NOT_RUNNING" ]; then
+# # make sure IP address is removed from interface
+# ip_stop
+# fi
+ ocf_log info "Monitoring return code: $mon_rc"
+ exit $mon_rc
+ return $mon_rc
;;
partial|no|partial2)
exit $OCF_NOT_RUNNING
+ return $OCF_NOT_RUNNING
;;
*)
# Errors on this interface?
+ exit $OCF_ERR_GENERIC
return $OCF_ERR_GENERIC
;;
esac
@@ -870,6 +1158,7 @@
start) ip_start
;;
stop) ip_stop
+ exit $?
;;
status) ip_status=`ip_served`
if [ $ip_status = "ok" ]; then
@@ -881,6 +1170,7 @@
fi
;;
monitor) ip_monitor
+ exit $?
;;
validate-all) ;;
*) ip_usage
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
