On Sat, 14 Mar 2009, Nick Withers wrote:
Right, here we go!
...
Turns out that the problem is a lock cycle triggered by the syncache calling,
indirectly, the firewall during output, and the firewall trying to look up the
connection for the packet. Thread one:
Tracing PID 31 tid 100030 td 0xffffff00012016e0
sched_switch() at sched_switch+0xdf
mi_switch() at mi_switch+0x18b
turnstile_wait() at turnstile_wait+0x1c4
_mtx_lock_sleep() at _mtx_lock_sleep+0x76
_mtx_lock_flags() at _mtx_lock_flags+0x95
syncache_lookup() at syncache_lookup+0xee
syncache_expand() at syncache_expand+0x38
tcp_input() at tcp_input+0x99b
ip_input() at ip_input+0xaf
ether_demux() at ether_demux+0x1b9
ether_input() at ether_input+0x1bb
fxp_intr() at fxp_intr+0x224
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffffffe80174d30, rbp = 0 ---
This thread holds TCP locks and is trying to acquire the syncache lock.
Thread two:
sched_switch() at sched_switch+0xdf
mi_switch() at mi_switch+0x18b
turnstile_wait() at turnstile_wait+0x1c4
_rw_rlock() at _rw_rlock+0x9c
ipfw_chk() at ipfw_chk+0x3ac1
ipfw_check_out() at ipfw_check_out+0xb1
pfil_run_hooks() at pfil_run_hooks+0xac
ip_output() at ip_output+0x357
syncache_respond() at syncache_respond+0x2fd
syncache_timer() at syncache_timer+0x15a
softclock() at softclock+0x270
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
This is the syncache timer holding syncache locks, calling IP output, and IPFW
trying to acquire TCP locks.
Am I right in thinking that you are using uid/gid/jail firewall rules? They
suffer from a fundamental architectural problem in that they require reaching
"up" to a higher level of the stack at times when it's not always a good idea
to do so. In general we solve the problem by passing "down" the inpcb for a
connection in the output path so that TCP doesn't have to look it up --
however, in the case of the syncache we actually don't have the inpcb easily
in hand (or at least, we have it, but we can't just lock it because syncache
locks are after TCP locks in the lock order...). It transpires that what the
firewall really wants is not the inpcb, but the credential, but those are
interfaces we can't change right now.
I'll need to think a bit about a proper fix for this, but you'll find the
problem likely goes away if you eliminate all uid/gid/jail rules from your
firewall. You could also tweak the syncache logic not to use a retransmit
timer, which might slightly extend the time it takes for systems to connect to
your host in the presence of packet loss, but would eliminate this
transmission path entirely. We'll need a real and more general fix, however,
to commit, and I'll look and see what I can come up with.
Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
