Hi Alexander, On Mon, Jun 06, 2011 at 05:42:30PM +0200, alexander.kra...@basf.com wrote: > Dejan Muhamedagic schrieb am 04.04.2011 14:35:34: > > On Fri, Mar 18, 2011 at 04:15:16PM +0100, alexander.kra...@basf.com > wrote: > > > Hi, > > > > > > Dejan Muhamedagic schrieb am 18.03.2011 14:31:08: > > > > Hi, > > > > > > > > On Wed, Mar 16, 2011 at 04:58:25PM +0100, Corvus Corax wrote: > > > > > > > > > > IPAddr2 puts the interface up on start and down on stop. > > > > > But its not able to detect an UP or DOWN change in status or > monitor. > > > > > > > > > > Therefore an "ifconfig <interface> down" from a thrird program or > a > > > > > careless administrator would drop the link without pacemaker > noticing! > > > > > > > > Hmm, careless administrator is somewhat of a paradox, right? > > > > > > > > Really, what was your motivation for this? It makes me wonder, > > > > since this RA has existed for many years and so far nobody > > > > bothered to test this. > > > > > > Hm, maybe the idea behind is not totally new. Remember this thread: > > > > http://lists.community.tummy.com/pipermail/linux-ha-dev/2011-February/018184.html > > > > > > > I would go with the remarks of LMB, that this is something closer to > > > the pingd than to Ipaddr2. Isn't the real intention of both post, that > you > > > want to know, if your network interface is vital ? > > > > Yes. > > > > > You may use pingd for that, but someone may be concerned to ping the > right > > > remote device (also a default-gateway might not be a very static thing > in > > > a modern network). > > > > > > My imagination is currently an agent (let's call it ethmonitor) that > > > monitors > > > a network interface with a combination of the fine methods that Robert > > > > Euhus > > > has posted in his patch. Than you could define some rules in CIB how > to > > > react on the event of a failed network interface. Sure this assumes > that > > > you > > > do your heartbeats over more than one interface. > > > > > > It would check: > > > 1. interface link up ? > > > 2. does the RX counter of the interface increase during a certain > amout > > > of time ? > > > 3. do I have some other nodes in my arp-cache which I could arping ? > > > 4. maybe retry all checks to overcome short outages > > > If all questions are answered with NO - the interface is dead. > > > > > > I would add my vote for such a feature. > > > > Just took a look at the thread you referenced above. > > Unfortunately, the author didn't get back with the new code > > after review and short discussion. > > > > Now I took the code from Robert in the above referenced thread and put it > into a complete new RA. > It is based very much on the existing pind agent, but implements the > monitoring like discussed above.
Great! > Please let me know, what you think about it. Does it work? :) See below for a few comments. Cheers, Dejan > > Cheers, > Alex > #!/bin/sh > # > # OCF Resource Agent compliant script. > # Monitor the vitality of a local network interface. > # > # Based on the work by Robert Euhus and Lars Marowsky-Brée. > # > # Transfered from Ipaddr2 into ethmonitor by Alexander Krauth > # > # Copyright (c) 2011 Robert Euhus, Alexander Krauth, Lars Marowsky-Brée > # All Rights Reserved. > # > # This program is free software; you can redistribute it and/or modify > # it under the terms of version 2 of the GNU General Public License as > # published by the Free Software Foundation. > # > # This program is distributed in the hope that it would be useful, but > # WITHOUT ANY WARRANTY; without even the implied warranty of > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > # > # Further, this software is distributed without any warranty that it is > # free of the rightful claim of any third person regarding infringement > # or the like. Any license provided herein, whether implied or > # otherwise, applies only to this software file. Patent licenses, if > # any, provided herein do not apply to combinations of this program with > # other software, or any other product whatsoever. > # > # You should have received a copy of the GNU General Public License > # along with this program; if not, write the Free Software Foundation, > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > # > # OCF parameters are as below > # > # OCF_RESKEY_interface > # OCF_RESKEY_multiplicator > # OCF_RESKEY_name > # OCF_RESKEY_repeat_count > # OCF_RESKEY_repeat_interval > # OCF_RESKEY_pktcnt_timeout > # OCF_RESKEY_arping_count > # OCF_RESKEY_arping_timeout > # OCF_RESKEY_arping_cache_entries > # > # TODO: Check against IPv6 > # > ####################################################################### > # Initialization: > > : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/resource.d/heartbeat} > . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs > > ####################################################################### > > meta_data() { > cat <<END > <?xml version="1.0"?> > <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd"> > <resource-agent name="ethmonitor"> > <version>1.2</version> > > <LONGdesc lang="en"> > Monitor the vitality of a local network interface. > > You may setup this RA as a clone resource to monitor the network interfaces > on different nodes, with the same interface name. > This is not related to the IP adress or the network on which a interface is > configured. > You may use this RA to move resources away from a node, which has a faulty > interface or prevent moving resources to such a node. > This gives you independend control of the resources, without involving > cluster intercommunication. But it requires your nodes to have more than one > network interface. > > The resource configuration requires a monitor operation, because the monitor > does the main part of the work. > In addition to the resource configuration, you need to configure some > location contraints, based on a CIB attribute value. > The name of the attribute value is configured in the 'name' option of this RA. > > Example constraint configuration: > location loc_connected_node my_resource_grp \ > rule $id="rule_loc_connected_node" -INF: ethmonitor eq 0 > > The ethmonitor works in 3 different modes to test the interface vitality. > 1. call ip to see if the link status is up (if link is down -> error) > 2. call ip an watch the RX counter (if packages come around in a certain time > -> success) > 3. call arping to check wether any of the IPs found in the lokal ARP cache > answers an ARP REQUEST (one answer -> success) > 4. return error I think that some parts of this long description should go to www.linux-ha.org/wiki/ethmonitor_(resource_agent) > </longdesc> > <shortdesc lang="en">Monitors network interfaces</shortdesc> > > <parameters> > <parameter name="interface" unique="0" required="1"> shouldn't this be unique? > <longdesc lang="en"> > The name of the network interface which should be monitored (e.g. eth0). > </longdesc> > <shortdesc lang="en">Network interface name</shortdesc> > <content type="string" default=""/> > </parameter> > > <parameter name="name" unique="0"> and this too? > <longdesc lang="en"> > The name of the CIB attribute to set. This is the name to be used in the > constraints. > </longdesc> > <shortdesc lang="en">Attribute name</shortdesc> > <content type="integer" default="ethmonitor"/> > </parameter> > > <parameter name="multiplier" unique="0" > > <longdesc lang="en"> > Multiplier for the value of the CIB attriobute specified in parameter name. > </longdesc> > <shortdesc lang="en">Multiplier for result variable</shortdesc> > <content type="integer" default="1"/> > </parameter> > > <parameter name="repeat_count"> > <longdesc lang="en"> > Specify how often the interface will be monitored, before the status is set > to failed. You need to set the timeout of the monitoring operation to at > least repeat_count * repeat_interval > </longdesc> > <shortdesc lang="en">Monitor repeat count</shortdesc> > <content type="integer" default="5"/> > </parameter> > > <parameter name="repeat_interval"> > <longdesc lang="en"> > Specify how long to wait in seconds between the repeat_counts. > </longdesc> > <shortdesc lang="en">Monitor repeat interval in seconds</shortdesc> > <content type="integer" default="10"/> > </parameter> > > <parameter name="pktcnt_timeout"> > <longdesc lang="en"> > Timeout for the RX packet counter. Stop listening for packet counter changes > after the given number of seconds. > </longdesc> > <shortdesc lang="en">packet counter timeout</shortdesc> > <content type="integer" default="5"/> > </parameter> > > <parameter name="arping_count"> > <longdesc lang="en"> > Number of ARP REQUEST packets to send for every IP. > Usually one ARP REQUEST (arping) is send > </longdesc> > <shortdesc lang="en">Number of arpings per IP</shortdesc> > <content type="integer" default="1"/> > </parameter> > > <parameter name="arping_timeout"> > <longdesc lang="en"> > Time in seconds to wait for ARP REQUESTs (all packets of arping_count). > This is to limit the time for arp requests, to be able to send requests to > more than one node, without running in the monitor operation timeout. > </longdesc> > <shortdesc lang="en">Timeout for arpings per IP</shortdesc> > <content type="integer" default="1"/> > </parameter> > > <parameter name="arping_cache_entries"> > <longdesc lang="en"> > Maximum number of IPs from ARP cache list to check for ARP REQUEST (arping) > answers. Newest entries are tried first. > </longdesc> > <shortdesc lang="en">Number of ARP cache entries to try</shortdesc> > <content type="integer" default="5"/> > </parameter> > > </parameters> > <actions> > <action name="start" timeout="20s" /> > <action name="stop" timeout="20s" /> > <action name="status" depth="0" timeout="20s" interval="10s" /> > <action name="monitor" depth="0" timeout="20s" interval="10s" /> > <action name="meta-data" timeout="5s" /> > <action name="validate-all" timeout="20s" /> > </actions> > </resource-agent> > END > > exit $OCF_SUCCESS > } > > # > # Return true, if the interface exists > # > is_interface() { > # > # List interfaces but exclude FreeS/WAN ipsecN virtual interfaces > # > local iface=`$IP2UTIL -o -f inet addr show | grep " $1 " \ > | cut -d ' ' -f2 | sort -u | grep -v '^ipsec[0-9][0-9]*$'` > if [ "$iface" != "" ]; then return 0; fi > return 1 [ "$iface" != "" ] is enough instead of the previous two lines > } > > if_init() { > local rc > > if [ X"$OCF_RESKEY_interface" = "X" ]; then > ocf_log err "Interface name (the interface parameter) is > mandatory" > exit $OCF_ERR_CONFIGURED > fi > > NIC="$OCF_RESKEY_interface" > > if is_interface $NIC > then > case "$NIC" in > *:*) ocf_log err "Do not specify a virtual interface : > $OCF_RESKEY_interface" > exit $OCF_ERR_CONFIGURED;; > *) ;; > esac > else > case $__OCF_ACTION in > validate-all) ocf_log err "Interface $OCF_RESKEY_interface does not > exist" > exit $OCF_ERR_CONFIGURED;; > *) ocf_log warn "Interface $OCF_RESKEY_interface does > not exist" > ## It might be a bond interface which is > temporarily not available, therefore we want to continue here > ;; Why not use NIC instead of OCF_RESKEY_interface when you already set that? > esac > fi > > : ${OCF_RESKEY_multiplier:="1"} > if ! ocf_is_decimal "$OCF_RESKEY_multiplier"; then > ocf_log err "Invalid OCF_RESKEY_multiplier > [$OCF_RESKEY_multiplier]" > exit $OCF_ERR_CONFIGURED > fi > > ATTRNAME=${OCF_RESKEY_name:-ethmonitor} > > REP_COUNT=${OCF_RESKEY_repeat_count:-5} > if ! ocf_is_decimal "$REP_COUNT" -o [ $REP_COUNT -lt 1 ]; then > ocf_log err "Invalid OCF_RESKEY_repeat_count [$REP_COUNT]" > exit $OCF_ERR_CONFIGURED > fi > REP_INTERVAL_S=${OCF_RESKEY_repeat_interval:-10} > if ! ocf_is_decimal "$REP_INTERVAL_S"; then > ocf_log err "Invalid OCF_RESKEY_repeat_interval > [$REP_INTERVAL_S]" > exit $OCF_ERR_CONFIGURED > fi > : ${OCF_RESKEY_pktcnt_timeout:="5"} > if ! ocf_is_decimal "$OCF_RESKEY_pktcnt_timeout"; then > ocf_log err "Invalid OCF_RESKEY_pktcnt_timeout > [$OCF_RESKEY_pktcnt_timeout]" > exit $OCF_ERR_CONFIGURED > fi > : ${OCF_RESKEY_arping_count:="1"} > if ! ocf_is_decimal "$OCF_RESKEY_arping_count"; then > ocf_log err "Invalid OCF_RESKEY_arping_count > [$OCF_RESKEY_arping_count]" > exit $OCF_ERR_CONFIGURED > fi > : ${OCF_RESKEY_arping_timeout:="1"} > if ! ocf_is_decimal "$OCF_RESKEY_arping_timeout"; then > ocf_log err "Invalid OCF_RESKEY_arping_timeout > [$OCF_RESKEY_arping_count]" > exit $OCF_ERR_CONFIGURED > fi > : ${OCF_RESKEY_arping_cache_entries:="5"} > if ! ocf_is_decimal "$OCF_RESKEY_arping_cache_entries"; then > ocf_log err "Invalid OCF_RESKEY_arping_cache_entries > [$OCF_RESKEY_arping_cache_entries]" > exit $OCF_ERR_CONFIGURED > fi > return $OCF_SUCCESS > } > > # get the link status on $NIC > # returns UP or DOWN or whatever ip reports (UNKNOWN?) > get_link_status () { > $IP2UTIL -o link show dev "$NIC" \ > | sed 's/.* state \([^ ]*\) .*/\1/' This prints "UNKNOWN" for my bridge (brn) interfaces which are up: [0]hex-12:~ > ip -o link show dev br0 6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \ link/ether 00:23:7d:a7:29:96 brd ff:ff:ff:ff:ff:ff > } > > # returns the number of received rx packets on $NIC > get_rx_packets () { > ocf_log debug "$IP2UTIL -o -s link show dev $NIC" > $IP2UTIL -o -s link show dev "$NIC" \ > | sed 's/.* RX: [^0-9]*[0-9]* *\([0-9]*\) .*/\1/' > # the first number after RX: ist the # of bytes , > # the second is the # of packets received > } > > # watch for packet counter changes for max. OCF_RESKEY_pktcnt_timeout seconds > # returns immedeately with return code 0 if any packets were received > # otherwise 1 is returned > watch_pkt_counter () { > local RX_PACKETS_NEW > local RX_PACKETS_OLD > RX_PACKETS_OLD="`get_rx_packets`" > for n in `seq $(( $OCF_RESKEY_pktcnt_timeout * 10 ))`; do > sleep 0.1 > RX_PACKETS_NEW="`get_rx_packets`" > ocf_log debug "RX_PACKETS_OLD: $RX_PACKETS_OLD > RX_PACKETS_NEW: $RX_PACKETS_NEW" > if [ "$RX_PACKETS_OLD" -ne "$RX_PACKETS_NEW" ]; then > ocf_log debug "we received some packets." > return 0 > fi > done > return 1 > } > > # returns list of cached ARP entries for $NIC > # sorted by age ("last confirmed") > # max. OCF_RESKEY_arping_cache_entries entries > get_arp_list () { > $IP2UTIL -s neighbour show dev $NIC \ > | sort -t/ -k2,2n | cut -d' ' -f1 \ > | head -n $OCF_RESKEY_arping_cache_entries > # the "used" entries in `ip -s neighbour show` are: > # "last used"/"last confirmed"/"last updated" > } > > # arping the IP given as argument $1 on $NIC > # until OCF_RESKEY_arping_count answers are received > do_arping () { > # TODO: add the source IP > # TODO: check for diffenrent arping versions out there > arping -q -c $OCF_RESKEY_arping_count -w $OCF_RESKEY_arping_timeout -I > $NIC $1 > # return with the exit code of the arping command > return $? > } > > # > # Check the interface depending on the level given as parameter: > $OCF_RESKEY_check_level > # > # 09: check for nonempty ARP cache > # 10: watch for packet counter changes > # > # 19: check arping_ip_list > # 20: check arping ARP cache entries > # > # 30: watch for packet counter changes in promiscios mode > # > # If unsuccessfull in levels 18 and above, > # the tests for higher check levels are run. > # > if_check () { > # always check link status first > link_status="`get_link_status`" > ocf_log debug "link_status: $link_status" > case $link_status in > UP) > ;; > DOWN) > # remove address from NIC > return $OCF_NOT_RUNNING > ;; > *) # this should not happen. > return $OCF_ERR_GENERIC > ;; > esac > > # watch for packet counter changes > ocf_log debug "watch for packet counter changes" > watch_pkt_counter && return $OCF_SUCCESS > > # check arping ARP cache entries > ocf_log debug "check arping ARP cache entries" > for ip in `get_arp_list`; do > do_arping $ip && return $OCF_SUCCESS > done > > # watch for packet counter changes in promiscios mode > # ocf_log debug "watch for packet counter changes in promiscios mode" > # be sure switch off promiscios mode in any case > # TODO: check first, wether promisc is already on and leave it > untouched. > # trap "$IP2UTIL link set dev $NIC promisc off; exit" INT TERM EXIT > # $IP2UTIL link set dev $NIC promisc on > # watch_pkt_counter && return $OCF_SUCCESS > # $IP2UTIL link set dev $NIC promisc off > # trap - INT TERM EXIT > > # looks like it's not working (for whatever reason) > return $OCF_NOT_RUNNING > } > > ####################################################################### > > if_usage() { > cat <<END > usage: $0 {start|stop|status|monitor|validate-all|meta-data} > > Expects to have a fully populated OCF RA-compliant environment set. > END > } > > set_cib_value() { > local score=`expr $1 \* $OCF_RESKEY_multiplier` > attrd_updater -n $ATTRNAME -v $score -q > local rc=$? > case $rc in > 0) ocf_log debug "attrd_updater: Updated $ATTRNAME = $score" ;; > *) ocf_log warn "attrd_updater: Could not update $ATTRNAME = $score: > rc=$rc";; > esac > return $rc > } > > if_monitor() { > ha_pseudo_resource $OCF_RESOURCE_INSTANCE monitor > local pseudo_status=$? > if [ $pseudo_status -ne $OCF_SUCCESS ]; then > exit $pseudo_status > fi > > local mon_rc=$OCF_NOT_RUNNING > local attr_rc=$OCF_NOT_RUNNING > local runs=0 > local start_time > local end_time > local sleep_time > while [ $mon_rc -ne $OCF_SUCCESS -a $REP_COUNT -gt 0 ] > do > start_time=`date +%s%N` > if_check > mon_rc=$? > REP_COUNT=$(( $REP_COUNT - 1 )) > if [ $mon_rc -ne $OCF_SUCCESS -a $REP_COUNT -gt 0 ]; then > ocf_log warn "Monitoring of $OCF_RESOURCE_INSTANCE failed, $REP_COUNT > retries left." > end_time=`date +%s%N` > sleep_time=`echo "scale=9; ( $start_time + ( $REP_INTERVAL_S * > 1000000000 ) - $end_time ) / 1000000000" | bc -q 2> /dev/null` > sleep $sleep_time 2> /dev/null > runs=$(($runs + 1)) > fi > > if [ $mon_rc -eq $OCF_SUCCESS -a $runs -ne 0 ]; then > ocf_log info "Monitoring of $OCF_RESOURCE_INSTANCE recovered from > error" > fi > done > > ocf_log debug "Monitoring return code: $mon_rc" > if [ $mon_rc -eq $OCF_SUCCESS ]; then > set_cib_value 1 > attr_rc=$? > else > ocf_log err "Monitoring of $OCF_RESOURCE_INSTANCE failed." > set_cib_value 0 > attr_rc=$? > fi > > ## The resource should not fail, if the interface is down. It should > fail, if the update of the CIB variable has errors. > ## To react on the interface failure you must use constraints based on > the CIB variable value, not on the recourse itself. recourse -> resource > exit $attr_rc > } > > if_validate() { > check_binary $IP2UTIL check_binary arping ? > if_init > return $? this line is superfluous > } > > case $__OCF_ACTION in > meta-data) meta_data > ;; > usage|help) if_usage > exit $OCF_SUCCESS > ;; > esac > > if_validate > > case $__OCF_ACTION in > start) ha_pseudo_resource $OCF_RESOURCE_INSTANCE start > exit $? > ;; > stop) attrd_updater -D -n $ATTRNAME > ha_pseudo_resource $OCF_RESOURCE_INSTANCE stop > exit $? > ;; > monitor|status) if_monitor > exit $? > ;; > validate-all) exit $? > ;; > *) if_usage > exit $OCF_ERR_UNIMPLEMENTED > ;; > esac _______________________________________________________ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/