On 2013-11-19 10:48, Lars Ellenberg wrote:
On Wed, Nov 13, 2013 at 09:02:47AM +0300, Vladislav Bogdanov wrote:
13.11.2013 04:46, Jefferson Ogata wrote:
...
In practice i ran into failover problems under load almost immediately.
Under load, when i would initiate a failover, there was a race
condition: the iSCSILogicalUnit RA will take down the LUNs one at a
time, waiting for each connection to terminate, and if the initiators
reconnect quickly enough, they get pissed off at finding that the target
still exists but the LUN they were using no longer does, which is often
the case during this transient takedown process. On the initiator, it
looks something like this, and it's fatal (here LUN 4 has gone away but
the target is still alive, maybe working on disconnecting LUN 3):
Nov 7 07:39:29 s01c kernel: sd 6:0:0:4: [sde] Sense Key : Illegal
Request [current]
Nov 7 07:39:29 s01c kernel: sd 6:0:0:4: [sde] Add. Sense: Logical unit
not supported
Nov 7 07:39:29 s01c kernel: Buffer I/O error on device sde, logical
block 16542656
One solution to this is using the portblock RA to block all initiator
In addition I force use of multipath on initiators with no_path_retry=queue
...
1. Lack of support for multiple targets using the same tgt account. This
is a problem because the iSCSITarget RA defines the user and the target
at the same time. If it allowed multiple targets to use the same user,
it wouldn't know when it is safe to delete the user in a stop operation,
because some other target might still be using it.
To solve this i did two things: first i wrote a new RA that manages a
Did I miss it, or did you post it somewhere?
Fork on Github and push there, so we can have a look?
Not set up with git right now; i've attached it here. It's short.
tgt user; this is instantiated as a clone so it runs along with the tgtd
clone. Second i tweaked the iSCSITarget RA so that on start, if
incoming_username is defined but incoming_password is not, the RA skips
the account creation step and simply binds the new target to
incoming_username. On stop, it similarly no longer deletes the account
if incoming_password is unset. I also had to relax the uniqueness
constraint on incoming_username in the RA metadata.
2. Disappearing LUNs during failover cause initiators to blow chunks.
For this i used portblock, but had to modify it because the TCP Send-Q
would never drain.
3. portblock preventing TCP Send-Q from draining, causing tgtd
connections to hang. I modified portblock to reverse the sense of the
iptables rules it was adding: instead of blocking traffic from the
initiator on the INPUT chain, it now blocks traffic from the target on
the OUTPUT chain with a tcp-reset response. With this setup, as soon as
portblock goes active, the next packet tgtd attempts to send to a given
initiator will get a TCP RST response, causing tgtd to hang up the
connection immediately. This configuration allows the connections to
terminate promptly under load.
I'm not totally satisfied with this workaround. It means
acknowledgements of operations tgtd has actually completed never make it
back to the initiator. I suspect this could cause problems in some
scenarios. I don't think it causes a problem the way i'm using it, with
each LUN as backing store for a distinct VM--when the LUN is back up on
the other node, the outstanding operations are re-sent by the initiator.
Maybe with a clustered filesystem this would cause problems; it
certainly would cause problems if the target device were, for example, a
tape drive.
Maybe only block "new" incoming connection attempts?
That's a good idea. Theoretically that should allow the existing
connections to drain. I'm worried it can lead to pacemaker timeouts
firing if there's a lot of queued data in the send queues, but i'll test it.
Thanks for the suggestion.
#!/bin/sh
#
# Resource script for managing tgt users
#
# Description: Manages a tgt user as an OCF resource in
# an High Availability setup.
#
# Author: Jefferson Ogata <jefferson.og...@noaa.gov>
# License: GNU General Public License (GPL)
#
#
# usage: $0 {start|stop|status|monitor|validate-all|meta-data}
#
# The "start" arg adds the user.
#
# The "stop" arg deletes it.
#
# OCF parameters:
# OCF_RESKEY_username
# OCF_RESKEY_password
#
# Example crm configuration:
#
# primitive tgtd lsb:tgtd op monitor interval="10s"
# clone clone.tgtd tgtd
# primitive user.foo ocf:heartbeat:tgtUser params username="foo"
password="secret"
# clone clone.user.foo user.foo
# order clone.tgtd_before_clone.user.foo inf: clone.tgtd:start
clone.user.foo:start
#
##########################################################################
# Initialization:
: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}
. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs
USAGE="Usage: $0 {start|stop|status|monitor|validate-all|meta-data}";
##########################################################################
usage()
{
echo $USAGE >&2
}
meta_data()
{
cat <<END
<?xml version="1.0"?>
<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
<resource-agent name="tgtUser">
<version>1.0</version>
<longdesc lang="en">
This script manages a user definition in tgtd, which can be bound
to one or more targets in the iSCSITarget RA.
</longdesc>
<shortdesc lang="en">Manages a tgt user</shortdesc>
<parameters>
<parameter name="username">
<longdesc lang="en">
The username.
</longdesc>
<shortdesc lang="en">Username</shortdesc>
<content type="string" default=""/>
</parameter>
<parameter name="password">
<longdesc lang="en">
The password.
</longdesc>
<shortdesc lang="en">Password</shortdesc>
<content type="string" default="" />
</parameter>
</parameters>
<actions>
<action name="start" timeout="10s"/>
<action name="stop" timeout="10s"/>
<action name="monitor" depth="0" timeout="10s" interval="60s" />
<action name="validate-all" timeout="10s"/>
<action name="meta-data" timeout="5s"/>
</actions>
</resource-agent>
END
exit $OCF_SUCCESS
}
add_user()
{
ocf_run tgtadm --lld iscsi --mode account --op delete --user
"$OCF_RESKEY_username" 2>/dev/null
ocf_run tgtadm --lld iscsi --mode account --op new \
--user "$OCF_RESKEY_username" \
--password "$OCF_RESKEY_password" || exit $OCF_ERR_GENERIC
exit $OCF_SUCCESS
}
delete_user()
{
ocf_run tgtadm --lld iscsi --mode account --op delete \
--user "$OCF_RESKEY_username" || exit $OCF_ERR_GENERIC
exit $OCF_SUCCESS
}
check_user()
{
ocf_run tgtadm --lld iscsi --mode account --op show | egrep -q
"^[[:space:]]+$OCF_RESKEY_username$"
if [ $? -ne 0 ]; then
ocf_log err "tgt user $OCF_RESKEY_username not present."
exit $OCF_NOT_RUNNING
fi
ocf_log info "tgt user $OCF_RESKEY_username okay."
exit $OCF_SUCCESS
}
validate_all()
{
if [ -z "$OCF_RESKEY_username" ]; then
ocf_log err "tgt username not specified"
exit $OCF_ERR_ARGS
fi
if [ -z "$OCF_RESKEY_password" ]; then
ocf_log err "tgt password not specified"
exit $OCF_ERR_ARGS
fi
return $OCF_SUCCESS
}
#
# Main
#
if [ $# -ne 1 ]; then
usage
exit $OCF_ERR_ARGS
fi
case $1 in
start) add_user
;;
stop) delete_user
;;
status) check_user
;;
monitor) check_user
;;
validate-all) validate_all
;;
meta-data) meta_data
;;
usage) usage
exit $OCF_SUCCESS
;;
*) usage
exit $OCF_ERR_UNIMPLEMENTED
;;
esac
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
