Re: NICs locking up, "*tcp_sc_h"
On Wed, 2009-03-18 at 11:52 +, Robert Watson wrote: > On Sun, 15 Mar 2009, Nick Withers wrote: > > >> I'll need to think a bit about a proper fix for this, but you'll find the > >> problem likely goes away if you eliminate all uid/gid/jail rules from your > >> firewall. You could also tweak the syncache logic not to use a retransmit > >> timer, which might slightly extend the time it takes for systems to > >> connect > >> to your host in the presence of packet loss, but would eliminate this > >> transmission path entirely. We'll need a real and more general fix, > >> however, to commit, and I'll look and see what I can come up with. > > > > Brilliant, thanks very much. I'll work without uid rules for the time > > being, > > then. > > Could I ask you to file a PR on this problem, btw, with the two traces I > singled out as interesting included, then forward me the PR receipt? That > will make the problem easier to keep track of. Done - Please see newborn kern/132774! > We're currently pondering ways to fix the problem that don't disturb the > stability of the ABI, and may have a workaround patch available shortly > that's > appropriate for MFC. Wow! Must admit I'd assumed this one was going to take a while to sort out, so I'm certainly happy to hear this. Cheers very much! > Robert N M Watson > Computer Laboratory > University of Cambridge -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 signature.asc Description: This is a digitally signed message part
Re: NICs locking up, "*tcp_sc_h"
On Sun, 15 Mar 2009, Nick Withers wrote: I'll need to think a bit about a proper fix for this, but you'll find the problem likely goes away if you eliminate all uid/gid/jail rules from your firewall. You could also tweak the syncache logic not to use a retransmit timer, which might slightly extend the time it takes for systems to connect to your host in the presence of packet loss, but would eliminate this transmission path entirely. We'll need a real and more general fix, however, to commit, and I'll look and see what I can come up with. Brilliant, thanks very much. I'll work without uid rules for the time being, then. Could I ask you to file a PR on this problem, btw, with the two traces I singled out as interesting included, then forward me the PR receipt? That will make the problem easier to keep track of. We're currently pondering ways to fix the problem that don't disturb the stability of the ABI, and may have a workaround patch available shortly that's appropriate for MFC. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Sat, 2009-03-14 at 18:01 +, Robert Watson wrote: > On Sat, 14 Mar 2009, Nick Withers wrote: > > > Right, here we go! > ... > > Turns out that the problem is a lock cycle triggered by the syncache calling, > indirectly, the firewall during output, and the firewall trying to look up > the > connection for the packet. Thread one: > > > Tracing PID 31 tid 100030 td 0xff00012016e0 > > sched_switch() at sched_switch+0xdf > > mi_switch() at mi_switch+0x18b > > turnstile_wait() at turnstile_wait+0x1c4 > > _mtx_lock_sleep() at _mtx_lock_sleep+0x76 > > _mtx_lock_flags() at _mtx_lock_flags+0x95 > > syncache_lookup() at syncache_lookup+0xee > > syncache_expand() at syncache_expand+0x38 > > tcp_input() at tcp_input+0x99b > > ip_input() at ip_input+0xaf > > ether_demux() at ether_demux+0x1b9 > > ether_input() at ether_input+0x1bb > > fxp_intr() at fxp_intr+0x224 > > ithread_loop() at ithread_loop+0xe9 > > fork_exit() at fork_exit+0x112 > > fork_trampoline() at fork_trampoline+0xe > > --- trap 0, rip = 0, rsp = 0xfffe80174d30, rbp = 0 --- > > This thread holds TCP locks and is trying to acquire the syncache lock. > Thread two: > > > sched_switch() at sched_switch+0xdf > > mi_switch() at mi_switch+0x18b > > turnstile_wait() at turnstile_wait+0x1c4 > > _rw_rlock() at _rw_rlock+0x9c > > ipfw_chk() at ipfw_chk+0x3ac1 > > ipfw_check_out() at ipfw_check_out+0xb1 > > pfil_run_hooks() at pfil_run_hooks+0xac > > ip_output() at ip_output+0x357 > > syncache_respond() at syncache_respond+0x2fd > > syncache_timer() at syncache_timer+0x15a > > softclock() at softclock+0x270 > > ithread_loop() at ithread_loop+0xe9 > > fork_exit() at fork_exit+0x112 > > fork_trampoline() at fork_trampoline+0xe > > This is the syncache timer holding syncache locks, calling IP output, and > IPFW > trying to acquire TCP locks. > > Am I right in thinking that you are using uid/gid/jail firewall rules? You are indeed. > They > suffer from a fundamental architectural problem in that they require reaching > "up" to a higher level of the stack at times when it's not always a good idea > to do so. In general we solve the problem by passing "down" the inpcb for a > connection in the output path so that TCP doesn't have to look it up -- > however, in the case of the syncache we actually don't have the inpcb easily > in hand (or at least, we have it, but we can't just lock it because syncache > locks are after TCP locks in the lock order...). It transpires that what the > firewall really wants is not the inpcb, but the credential, but those are > interfaces we can't change right now. Thanks for the explanation! > I'll need to think a bit about a proper fix for this, but you'll find the > problem likely goes away if you eliminate all uid/gid/jail rules from your > firewall. You could also tweak the syncache logic not to use a retransmit > timer, which might slightly extend the time it takes for systems to connect > to > your host in the presence of packet loss, but would eliminate this > transmission path entirely. We'll need a real and more general fix, however, > to commit, and I'll look and see what I can come up with. Brilliant, thanks very much. I'll work without uid rules for the time being, then. Ta for your time and help on this! > Robert N M Watson > Computer Laboratory > University of Cambridge -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 signature.asc Description: This is a digitally signed message part
Re: NICs locking up, "*tcp_sc_h"
On Sat, 14 Mar 2009, Nick Withers wrote: Right, here we go! ... Turns out that the problem is a lock cycle triggered by the syncache calling, indirectly, the firewall during output, and the firewall trying to look up the connection for the packet. Thread one: Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xdf mi_switch() at mi_switch+0x18b turnstile_wait() at turnstile_wait+0x1c4 _mtx_lock_sleep() at _mtx_lock_sleep+0x76 _mtx_lock_flags() at _mtx_lock_flags+0x95 syncache_lookup() at syncache_lookup+0xee syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0x99b ip_input() at ip_input+0xaf ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x224 ithread_loop() at ithread_loop+0xe9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffe80174d30, rbp = 0 --- This thread holds TCP locks and is trying to acquire the syncache lock. Thread two: sched_switch() at sched_switch+0xdf mi_switch() at mi_switch+0x18b turnstile_wait() at turnstile_wait+0x1c4 _rw_rlock() at _rw_rlock+0x9c ipfw_chk() at ipfw_chk+0x3ac1 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0xac ip_output() at ip_output+0x357 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a softclock() at softclock+0x270 ithread_loop() at ithread_loop+0xe9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe This is the syncache timer holding syncache locks, calling IP output, and IPFW trying to acquire TCP locks. Am I right in thinking that you are using uid/gid/jail firewall rules? They suffer from a fundamental architectural problem in that they require reaching "up" to a higher level of the stack at times when it's not always a good idea to do so. In general we solve the problem by passing "down" the inpcb for a connection in the output path so that TCP doesn't have to look it up -- however, in the case of the syncache we actually don't have the inpcb easily in hand (or at least, we have it, but we can't just lock it because syncache locks are after TCP locks in the lock order...). It transpires that what the firewall really wants is not the inpcb, but the credential, but those are interfaces we can't change right now. I'll need to think a bit about a proper fix for this, but you'll find the problem likely goes away if you eliminate all uid/gid/jail rules from your firewall. You could also tweak the syncache logic not to use a retransmit timer, which might slightly extend the time it takes for systems to connect to your host in the presence of packet loss, but would eliminate this transmission path entirely. We'll need a real and more general fix, however, to commit, and I'll look and see what I can come up with. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 2009-03-13 at 09:49 +, Robert Watson wrote:
> On Fri, 13 Mar 2009, Robert Watson wrote:
>
> > Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks"
> should read WITNESS
> > and "show allchains" would be useful. I've had a report of a TCP lock leak
> > possibly in tcp_input(), but haven't managed to track it down yet -- this
> > could well be it as well.
Right, here we go!
Here's "show alllocks"' output:
Process 31 (irp20: fxp0+) thread 0xff00012016e0 (100030)
exclusive rw tcpinp r = 0 (0xff000392d570) locked
@ /usr/src/sys/netinet/tcp_input.c:480
exclusive rw tcp r = 0 (0x806dcbe8) locked
@ /usr/src/sys/netinet/tcp_input.c:400
Process 17 (swi6: Giant taskq) thread 0xff0001173000 (100016)
exclusive sleep mutex Giant r = 0 (0x80652520) locked
@ /usr/src/sys/kern/kern_intr.c:1087
Process 12 (swi4: clock) thread 0xff00010c6370 (13)
shared rw IPFW static rules r = 0 (0x806db2b8) locked
@ /usr/src/sys/netinet/ip_fw2.c:2460
shared rw PFil hoow read/write mutex r = 0 (0x806dba28) locked
@ /usr/src/sys/net/pfil.c:73
exclusive sleep mutex tcp_sc_head r = 0 (0xfffe8036c8f8) locked
@ /usr/src/sys/kern/kern_timeout.c:241
...and here's "show allchains"':
chain 1:
thread 100031 (pid 32, irp22: rl0) blocked on lock 0x806dcbe8
(rw) "tcp"
thread 100030 (pid 31, irq20: fxp0+) blocked on lock 0xfffe8036c8f8
(sleep mutex) "tcp_sc_head"
thread 13 (pid 12, swi4: clock) blocked on lock 0x806dcbe8
(rw) "tcp"
thread 100030 (pid 31, irq20: fxp0+) blocked on lock 0xfffe8036c8f8
(sleep mutex) "tcp_sc_head"
thread 13 (pid 12, swi4: clock) blocked on lock 0x806dcbe8
(rw) "tcp"
thread 100030 (pid 31, irq20: fxp0+) blocked on lock 0xfffe8036c8f8
(sleep mutex) "tcp_sc_head"
thread 13 (pid 12, swi4: clock) blocked on lock 0x806dcbe8
(rw) "tcp"
(...and so on, these last two seeming to go on forever.)
This is with fxp0 and rl0 lagg(4)-ed
For completeness...
- "ps" in DDB shows that:
- PID 32 ("[irp22: rl0]") is in state "LL" on "*tcp"
- PID 31 ("[irq20: fxp0+]") is in state "LL" on "*tcp_sc_h"
- PID 12 ("[swi4: clock]") is in state "LL" on "*tcp"
- "where 32" gives:
sched_switch() at sched_switch+0xdf
mi_switch() at mi_switch+0x18b
turnstile_wait() at turnstile_wait+0x1c4
_rw_lock_hard() at _rw_lock_hard+0xa3
_rw_wlock() at _rw_wlock+0x54
tcp_input() at tcp_input+0x318
ip_input() at ip_input+0xaf
ether_demux() at ether_demux+0x1b9
ether_input() at ether_input+0x1bb
rl_rxeof() at rl_rxeof+0x1c8
rl_intr() at rl_intr+0x138
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffe91e0cd30, rbp = 0 ---
- "where 31" gives:
Tracing PID 31 tid 100030 td 0xff00012016e0
sched_switch() at sched_switch+0xdf
mi_switch() at mi_switch+0x18b
turnstile_wait() at turnstile_wait+0x1c4
_mtx_lock_sleep() at _mtx_lock_sleep+0x76
_mtx_lock_flags() at _mtx_lock_flags+0x95
syncache_lookup() at syncache_lookup+0xee
syncache_expand() at syncache_expand+0x38
tcp_input() at tcp_input+0x99b
ip_input() at ip_input+0xaf
ether_demux() at ether_demux+0x1b9
ether_input() at ether_input+0x1bb
fxp_intr() at fxp_intr+0x224
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffe80174d30, rbp = 0 ---
- "where 12" gives:
sched_switch() at sched_switch+0xdf
mi_switch() at mi_switch+0x18b
turnstile_wait() at turnstile_wait+0x1c4
_rw_rlock() at _rw_rlock+0x9c
ipfw_chk() at ipfw_chk+0x3ac1
ipfw_check_out() at ipfw_check_out+0xb1
pfil_run_hooks() at pfil_run_hooks+0xac
ip_output() at ip_output+0x357
syncache_respond() at syncache_respond+0x2fd
syncache_timer() at syncache_timer+0x15a
softclock() at softclock+0x270
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffe80017d30, rbp = 0 ---
- Before having entered the debugger, the following were logged:
lock order reversal:
1st 0xff00032947b0 tcpinp (tcpinp)
@ /usr/src/sys/netinet/tcp_timer.c:169
2nd 0x806dba28 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x565
_rw_rlock() at _rw_rlock+0x25
pfil_run_hooks() at pfil_run_hooks+0x44
ip_output() at ip_output+0x357
tcp_output() at tcp_output+0xa1d
tcp_timer_delack() at tcp_timer_delack+0xc7
softclock() at softclock+0x270
ithread_loop() at ithread_loop+0xe9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffe80017d30, rbp = 0 ---
...and:
lock order reversal:
1st 0xfffe80365df0 tcp_sc_h
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009 20:56:24 +1100 Nick Withers wrote: > I'm sorry to ask what is probably a very simple question, but is there > somewhere I should look to get clues on debugging from a manually > generated dump? I tried "panic" after manually envoking the kernel > debugger but proved highly inept at getting from the dump the same > information "ps" / "where" gave me within the debugger live. You can capture ddb session in capture buffer and then extract it from the dump. In ddb run capture on do your debugging then run "panic" or "call doadump" and after reboot: ddb capture -M /var/crash/vmcore.X print > out I would recommend to increase debug.ddb.capture.bufsize sysctl variable to be sure all the ddb session will be captured. -- Mikolaj Golub ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Nick Withers wrote: Sorry for the original double-post, by the way, not quite sure how that happened... I can reproduce this problem relatively easily, by the way (every 3 days, on average). I meant to say this before, too, but it seems to happen a lot more often on the fxp than on rl. I'm sorry to ask what is probably a very simple question, but is there somewhere I should look to get clues on debugging from a manually generated dump? I tried "panic" after manually envoking the kernel debugger but proved highly inept at getting from the dump the same information "ps" / "where" gave me within the debugger live. If this is, in fact, a TCP input lock leak of some sort, then most likely some particular property of a host your system talks to, or a network it runs over, triggers this (presumably) unusual edge case -- perhaps a firewall that mucks with TCP in a funny way, etc. Of course, it might be something completely different -- the fact that everything is blocked on *tcp_sc_h and *tcp, simply means that something holding TCP locks hasn't released them, and this could happen for a number of reasons. Once you've acquired a crashdump, you can run crashinfo(8), which will produce a summary of useful debugging information. There are some things that are a bit easier to do in the run-time debugger, such as lock analysis, as the run-time debugger is more up-close and personal with in-kernel data structures; other things are easier in kgdb, which has complete source code and C type access. I find kgdb works pretty well for everything but "show much what locks are held". Many of our system monitoring tools, including ps and portions of netstat, can actually be run on crashdumps to report the state of the system at the time it crashed -- take a look at the -M and -N command line arguments, which respectively allow you to point those tools at the crashdump and at a kernel with debugging symbols (typically kernel.debug or kernel.symbols) matching the kernel that was booted at the time of the crash. Robert N M Watson Computer Laboratory University of Cambridge Ta for your help! Robert N M Watson Computer Laboratory University of Cambridge Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _mtx_lock_sleep() at _mtx_lock_sleep+0x76 syncache_lookup() at syncache_lookup+0x176 syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0xa7d ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x233 ithread_loop() at ithread_loop+0x17f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe A "where" on a process stuck in "*tcp", in this case "[swi4: clock]", gave the somewhat similar: sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _rw_rlock() at _rw_rlock+0x8c ipfw_chk() at ipfw_chk+0x3ab2 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0x9c ip_output() at ip_output+0x367 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a (...) In this particular case, the fxp0 card is in a lagg with rl0, but this problem can be triggered with either card on their own... The scheduler is SCHED_ULE. I'm not too sure how to give more useful information that this, I'm afraid. It's a custom kernel, too... Do I need to supply information on what code actually exists at the relevant addresses (I'm not at all clued in on how to do this... Sorry!)? Should I chuck WITNESS, INVARIANTS et al. in? I *think* every time this has been triggered there's been a "python2.5" process in the "*tcp" state. This machine runs net-p2p/deluge and generally has at least 100 TCP connections on the go at any given time. Can anyone give me a clue as to what I might do to track this down? Appreciate any pointers. -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 2009-03-13 at 09:37 +, Robert Watson wrote: > On Fri, 13 Mar 2009, Nick Withers wrote: > > > I recently installed my first amd64 system (currently running RELENG_7 from > > 2009-03-11) to replace an aged ppc box and have been having dramas with the > > network locking up. > > > > Breaking into the debugger manually and ps-ing shows the network card > > (e.g., > > "[irq20: fxp0+]") in state "LL" in "*tcp_sc_h". It seems the process(es) > > trying to access the card at the time is / are in state "L" in "*tcp". > > > > I thought this may have been something-or-other in the fxp driver, so > > installed an rl card and sadly ran into the issue again. > > > > The console appears unresponsive, but I can get into the debugger (and as > > soon as I have, input I'd sent seems to "go through", e.g., if I hit > > "Enter" > > a couple o' times, nothing happens; when I ++ into the > > debugger a few login prompts pop up before the debugger output). > > > > A "where" on the fxp / rl process (thread?) gives (transcribed from the > > console): > > Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks" > and "show allchains" would be useful. I've had a report of a TCP lock leak > possibly in tcp_input(), but haven't managed to track it down yet -- this > could well be it as well. Righto, I'll recompile the kernel with INVARIANTS (hell, I'll go bananas and include everything listed in http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-deadlocks.html - anything else I might include?). Sorry for the original double-post, by the way, not quite sure how that happened... I can reproduce this problem relatively easily, by the way (every 3 days, on average). I meant to say this before, too, but it seems to happen a lot more often on the fxp than on rl. I'm sorry to ask what is probably a very simple question, but is there somewhere I should look to get clues on debugging from a manually generated dump? I tried "panic" after manually envoking the kernel debugger but proved highly inept at getting from the dump the same information "ps" / "where" gave me within the debugger live. Ta for your help! > Robert N M Watson > Computer Laboratory > University of Cambridge > > > > > > Tracing PID 31 tid 100030 td 0xff00012016e0 > > sched_switch() at sched_switch+0xf1 > > mi_switch() at mi_switch+0x18f > > turnstile_wait() at turnstile_wait+0x1cf > > _mtx_lock_sleep() at _mtx_lock_sleep+0x76 > > syncache_lookup() at syncache_lookup+0x176 > > syncache_expand() at syncache_expand+0x38 > > tcp_input() at tcp_input+0xa7d > > ip_input() at ip_input+0xa8 > > ether_demux() at ether_demux+0x1b9 > > ether_input() at ether_input+0x1bb > > fxp_intr() at fxp_intr+0x233 > > ithread_loop() at ithread_loop+0x17f > > fork_exit() at fork_exit+0x11f > > fork_trampoline() at fork_trampoline+0xe > > > > > > A "where" on a process stuck in "*tcp", in this case "[swi4: clock]", > > gave the somewhat similar: > > > > > > sched_switch() at sched_switch+0xf1 > > mi_switch() at mi_switch+0x18f > > turnstile_wait() at turnstile_wait+0x1cf > > _rw_rlock() at _rw_rlock+0x8c > > ipfw_chk() at ipfw_chk+0x3ab2 > > ipfw_check_out() at ipfw_check_out+0xb1 > > pfil_run_hooks() at pfil_run_hooks+0x9c > > ip_output() at ip_output+0x367 > > syncache_respond() at syncache_respond+0x2fd > > syncache_timer() at syncache_timer+0x15a > > (...) > > > > > > In this particular case, the fxp0 card is in a lagg with rl0, but this > > problem can be triggered with either card on their own... > > > > The scheduler is SCHED_ULE. > > > > I'm not too sure how to give more useful information that this, I'm > > afraid. It's a custom kernel, too... Do I need to supply information on > > what code actually exists at the relevant addresses (I'm not at all > > clued in on how to do this... Sorry!)? Should I chuck WITNESS, > > INVARIANTS et al. in? > > > > I *think* every time this has been triggered there's been a "python2.5" > > process in the "*tcp" state. This machine runs net-p2p/deluge and > > generally has at least 100 TCP connections on the go at any given time. > > > > Can anyone give me a clue as to what I might do to track this down? > > Appreciate any pointers. > > -- > > Nick Withers > > email: n...@nickwithers.com > > Web: http://www.nickwithers.com > > Mobile: +61 414 397 446 > > -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 signature.asc Description: This is a digitally signed message part
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Robert Watson wrote: Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks" should read WITNESS and "show allchains" would be useful. I've had a report of a TCP lock leak possibly in tcp_input(), but haven't managed to track it down yet -- this could well be it as well. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Nick Withers wrote: I recently installed my first amd64 system (currently running RELENG_7 from 2009-03-11) to replace an aged ppc box and have been having dramas with the network locking up. Breaking into the debugger manually and ps-ing shows the network card (e.g., "[irq20: fxp0+]") in state "LL" in "*tcp_sc_h". It seems the process(es) trying to access the card at the time is / are in state "L" in "*tcp". I thought this may have been something-or-other in the fxp driver, so installed an rl card and sadly ran into the issue again. The console appears unresponsive, but I can get into the debugger (and as soon as I have, input I'd sent seems to "go through", e.g., if I hit "Enter" a couple o' times, nothing happens; when I ++ into the debugger a few login prompts pop up before the debugger output). A "where" on the fxp / rl process (thread?) gives (transcribed from the console): Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks" and "show allchains" would be useful. I've had a report of a TCP lock leak possibly in tcp_input(), but haven't managed to track it down yet -- this could well be it as well. Robert N M Watson Computer Laboratory University of Cambridge Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _mtx_lock_sleep() at _mtx_lock_sleep+0x76 syncache_lookup() at syncache_lookup+0x176 syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0xa7d ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x233 ithread_loop() at ithread_loop+0x17f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe A "where" on a process stuck in "*tcp", in this case "[swi4: clock]", gave the somewhat similar: sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _rw_rlock() at _rw_rlock+0x8c ipfw_chk() at ipfw_chk+0x3ab2 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0x9c ip_output() at ip_output+0x367 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a (...) In this particular case, the fxp0 card is in a lagg with rl0, but this problem can be triggered with either card on their own... The scheduler is SCHED_ULE. I'm not too sure how to give more useful information that this, I'm afraid. It's a custom kernel, too... Do I need to supply information on what code actually exists at the relevant addresses (I'm not at all clued in on how to do this... Sorry!)? Should I chuck WITNESS, INVARIANTS et al. in? I *think* every time this has been triggered there's been a "python2.5" process in the "*tcp" state. This machine runs net-p2p/deluge and generally has at least 100 TCP connections on the go at any given time. Can anyone give me a clue as to what I might do to track this down? Appreciate any pointers. -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
