On 5/28/07, Horms <[EMAIL PROTECTED]> wrote:
On Sat, May 26, 2007 at 11:22:40AM +0900, Horms wrote:
> On Fri, May 25, 2007 at 09:30:52AM +0000, Sebastien Estienne wrote:
> >
> > I didn't try 2.6.21 yet, but using ubuntu dapper kernel (2.6.15) i
> > can't reproduce the bug.
> > When i was using feisty kernel (2.6.20), i can reproduce in less than 5
> > minutes.
> >
> > I'm using lvs to loadbalance some mysql servers, i wrote a deamon that
> > check the synchro of the mysql replication on each slave and adjust
> > the wieght on the lvs every 500ms
>
> It does look a lot like there is some sort of locking problem in there.
> Would it be possible to send your kernel config, as the locking
> deatails to change a little with different configs.
About the kernel .config, i'm using the vanilla kernel "-server" from
ubuntu feisty
If you also have some details of you ipvs configuration,
that might help narrow down which code-paths to investigate.
i attached the output of ipvsadm-save
i'm adjusting the weight every 500ms by generating lines like this:
-e -t 10.33.1.231:3306 -r 10.33.1.1 -w 100
and piping all the needed changes in ipvadm -R
it can represent something like 20 to 40 updates in one time.
i also noticed that sometimes when i execute "ipvsadm" the display get
locked in the middle for a second and then finish.
I spent some time this afternoon looking into this probem,
and what I think is happening is:
1. Due to your weight-update operations, one processor
is sitting in ip_vs_edit_dest() called by do_ip_vs_set_ctl(),
holding write_lock_bh(&__ip_vs_svc_lock) and waiting
for svc->usecnt to go down to 1.
2. Another process is trying to grab
read_lock(&__ip_vs_svc_lock) in ip_vs_service_get(),
called from tcp_conn_schedule() and in turn ip_vs_in().
I guess that for some reason svc->usecnt isn't going down to 0.
Though I haven't been able to isolate anything particularly
interesting.
That said, the locking isn't that simple, IMHO, so there seems
to be quite a lot of scope for errors.
Some things that are of minor insterst are:
I.
ip_vs_edit_dest() loops with the following construct:
while (atomic_read(&svc->usecnt) > 1) {};
whereas similar code in the same file uses
IP_VS_WAIT_WHILE(atomic_read(&svc->usecnt) > 1);
which expands to
while (atomic_read(&svc->usecnt) > 1) { cpu_relax(); }
But I dount this is a problem, except for burning the cpu a bit harder
than it needs to.
II.
ip_vs_set_ctl() does seem to leak svc->usecnt in one corner case,
but I doubt that is what you are seeing - if it was your ipvsadm
command(s) would hang. The problem is a bit wordy to describe,
but this fix should illustrate the problem.
--- linux-2.6.orig/net/ipv4/ipvs/ip_vs_ctl.c
+++ linux-2.6/net/ipv4/ipvs/ip_vs_ctl.c
@@ -2000,7 +2000,7 @@ do_ip_vs_set_ctl(struct sock *sk, int cm
if (cmd != IP_VS_SO_SET_ADD
&& (svc == NULL || svc->protocol != usvc->protocol)) {
ret = -ESRCH;
- goto out_unlock;
+ goto out_svc;
}
switch (cmd) {
@@ -2034,9 +2034,9 @@ do_ip_vs_set_ctl(struct sock *sk, int cm
ret = -EINVAL;
}
+ out_svc:
if (svc)
ip_vs_service_put(svc);
-
out_unlock:
mutex_unlock(&__ip_vs_mutex);
out_dec:
III.
Perhaps if you are calling ipvsadm a lot then there is a remote
possibility that write_lock_bh() could starve read_lock(). This
seems ludicrous, but I'm just mentioning it as it crossed my mind.
--
Horms
H: http://www.vergenet.net/~horms/
W: http://www.valinux.co.jp/en/
--
Sebastien Estienne
-A -t 10.33.1.231:3306 -s wlc
-a -t 10.33.1.231:3306 -r 10.33.1.59:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.58:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.57:3306 -g -w 100
-a -t 10.33.1.231:3306 -r 10.33.1.56:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.55:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.54:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.53:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.52:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.51:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.50:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.49:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.48:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.47:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.46:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.45:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.44:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.43:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.42:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.41:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.40:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.39:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.38:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.37:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.36:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.34:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.33:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.32:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.31:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.30:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.29:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.28:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.27:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.26:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.25:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.24:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.23:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.22:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.21:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.20:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.19:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.18:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.17:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.16:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.14:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.13:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.12:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.9:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.4:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.3:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.2:3306 -g -w 0
-a -t 10.33.1.231:3306 -r 10.33.1.1:3306 -g -w 0
-A -t 10.33.1.232:3306 -s wlc
-a -t 10.33.1.232:3306 -r 10.33.1.59:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.58:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.57:3306 -g -w 100
-a -t 10.33.1.232:3306 -r 10.33.1.56:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.55:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.54:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.53:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.52:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.51:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.50:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.49:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.48:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.47:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.46:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.45:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.44:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.43:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.42:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.41:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.40:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.39:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.38:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.37:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.36:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.34:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.33:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.32:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.31:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.30:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.29:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.28:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.27:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.26:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.25:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.24:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.23:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.22:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.21:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.20:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.19:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.18:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.17:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.16:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.14:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.13:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.12:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.9:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.4:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.3:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.2:3306 -g -w 0
-a -t 10.33.1.232:3306 -r 10.33.1.1:3306 -g -w 0
-A -t 10.33.1.221:3306 -s wlc
-a -t 10.33.1.221:3306 -r 10.33.1.59:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.58:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.57:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.56:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.55:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.54:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.53:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.52:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.51:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.50:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.49:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.48:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.47:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.46:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.45:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.44:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.43:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.42:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.41:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.40:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.39:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.38:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.37:3306 -g -w 0
-a -t 10.33.1.221:3306 -r 10.33.1.36:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.34:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.33:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.32:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.31:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.30:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.29:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.28:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.27:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.26:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.25:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.24:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.23:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.22:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.21:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.20:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.19:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.18:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.17:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.16:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.14:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.13:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.12:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.9:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.4:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.3:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.2:3306 -g -w 100
-a -t 10.33.1.221:3306 -r 10.33.1.1:3306 -g -w 100
-A -t 10.33.1.222:3306 -s wlc
-a -t 10.33.1.222:3306 -r 10.33.1.59:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.58:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.57:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.56:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.55:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.54:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.53:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.52:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.51:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.50:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.49:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.48:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.47:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.46:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.45:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.44:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.43:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.42:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.41:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.40:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.39:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.38:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.37:3306 -g -w 0
-a -t 10.33.1.222:3306 -r 10.33.1.36:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.34:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.33:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.32:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.31:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.30:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.29:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.28:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.27:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.26:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.25:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.24:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.23:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.22:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.21:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.20:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.19:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.18:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.17:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.16:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.14:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.13:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.12:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.9:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.4:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.3:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.2:3306 -g -w 100
-a -t 10.33.1.222:3306 -r 10.33.1.1:3306 -g -w 100
