reproducible 2.2.1x nethangs
kernel versions 2.2.17 and 2.2.18-pre23 (same behavior) monolithic kernel i21143 tulip card (may or may not be significant, stock kernel driver) egcs-1.1.2, glibc-2.1.3, binutils-2.9.1.0.25 I can reliably hang either 2.2.17 or 2.2.18-pre23 (same way, same circumstances) with httpd over eth0. It is not a particularly exotic kernel config (ethernet, tulip, dummy, ppp, aha154x, scsi hd/cd/tape, pio ide, i486, generic pci driving an sis496 pci 2.0 bus, no pci bridge optimization, firewall enabled, no masq, no proxy, no adv routing). All of this hardware and the network are stable on 2.0.38 (ie the tcp/ip over ethernet hang never happens there). It happens without any ipchains rules installed (the support is there, but it's not configured). It doesn't seem to do it on ftp (although that may simply be not having pushed it hard enough). It can handle 100s of mbs in a single ftp session without falling over, but a rapid sequence of httpd requests will knock it over every time. Minor points of evidence: * on one test, "strace -ff ..." showed the second argument to accept() scribbled over (6-7 lines of "^@^@..." in the child) about three forks before it deadlocked. I saw the same thing at the bottom of the httpd server's log after an earlier hang. * It doesn't simply stop, it suddenly gets really slow on the connect where it is going to hang. The last html page downloaded on one test ended up with a partial document, so it sometimes starts the data transfer, it simply can't complete it (the kernel/network_stack is already on it's way to the twilight zone when the download starts, it simply manages to squeeze out a few packets before it gets there). * When it happens, it takes the keyboard with it, and you can't ping it. * It's not the hd filesystems. I can html browse the same files that it hangs on over eth0 via lynx on the same host where the httpd server is running for hours without reproducing the kernel hang. I can move gbs of data around on those filesystems without errors and without filesystem corruption. Since the error can be reproduced so reliably, it should be possible to debug it, if I know where/how to enable verbose logging. Suggestions? Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: reproducible 2.2.1x nethangs
(from layer above device driver, imho) The fact that the http hang does not happen when connecting to the httpd server from an http client running on the same host as the server implicates the ethernet interface, but I would be shocked to find that the cause is a bug in specifically the tulip driver driving a real tulip card. You can hang it with http with a tiny fraction of the packets that are transferred during an ftp session that doesn't bother it at all, so the device driver streams packets just fine. The http hang has many more individual connects and forks and connection shutdowns, however, so I would guess that somewhere in the interface between tcp/ip stack and device driver bottom half calls is where the bug hits. I doubt that it matters at all which ethernet device driver it is exactly, other than perhaps different latencies affecting the timing on interrupt races (ie any card with the same average latency as an i21143 tulip card will probably see the same problem in the same kernel versions). So, what code is different between a socket connection from a listening daemon to a pci ethernet device driver and a socket connection from the same listening daemon to a client connected via localhost? There is a race or other bug in the first that isn't in the second, and it is a race/bug that is not in 2.0.38. I can't knock 2.0.38 over at all with http over the same ethernet lan from the same client (but 2.0.38 doesn't work with ipchains and doesn't have the dentry cache, vm improvements, etc, so this is worth fixing). (Note: i486, no SMP) -- Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: reproducible 2.2.1x nethangs
I retract the comment about "accept() 2nd argument scribbled over in the child". That was a misinterpretation of the strace log. strace shows the struct sockaddr * scribble in the parent after a restart of the accept() call. Also, the firewalling code is eliminated from consideration. I compiled it and ppp out, and the only difference was that the hang happened sooner, in about a dozen http connects instead of around 30. What happens, according to strace, is that accept() gets interrupted when the SIGCHLD signal is delivered after the child that handled the previous connect exits, and accept() sets ERESTARTSYS. For the first N connects, this is not problematic. accept() is called again, it returns normally, the fork() happens, everything is copascetic. But eventually the restarted accept() call shows nul bytes in the struct sockaddr *addr arg value in the strace output after restarting accept, and a few connects later the kernel hangs. If the socket itself is truly ozoned when strace says the accept() struct sockaddr * argument has changed in between interrupt and restart, the subsequent forked child shouldn't be able to send at all, but that is not what happens. That html document gets there. Yet the kernel always hangs within a few connects of seeing that, so I'd be suspicious of the internal kernel data associated with that socket when it comes time to deallocate it. The accept() code in the context of interrupts that cause it to be stopped and restarted is at least worth a look for possible races. Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: reproducible 2.2.1x nethangs (dma?)
Hmmm. Memory corruption (shows up in the args of a traced httpd process sitting on it's stack), concurrent with pci busmastering ethernet yes, concurrent with localhost connect to same server no (just tried it with 20 times the number of connects from local client that reliably hang it over ethernet, didn't flinch; same SIGCHLD during accept() events showed in the log, but never with the httpd parent's stack corrupted after accept() restarted). Dma error? On connect only? (Doesn't happen during 100mb ftp transfers.) (The ethernet card does support dma buffer management.) Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
tulip log (2.2.17-2.2.18pre23 http-induced kernel deadlock)
(I'm posting this to the kernel list because I'm still not convinced that the deadlock results from a problem with a specific ethernet driver.) I enabled debuggin in tulip.c with #define TULIP_DEBUG 7 and directed klogd to save all kernel messages to a file. After initialization (seems to work, since subsequent large ftp transfers have no problem), the verbose debugging log from the tulip driver starts out like this: # nothing happening on the network Nov 29 12:44:26 turpin kernel: eth0: 21143 negotiation status 00c6, MII. Nov 29 12:44:26 turpin kernel: eth0: MII status 782d, Link partner report 4481. Nov 29 12:46:26 turpin kernel: klogd 1.4-0, log source = /proc/kmsg started. Nov 29 12:46:26 turpin kernel: eth0: 21143 negotiation status 00c6, MII. Nov 29 12:46:26 turpin kernel: eth0: MII status 782d, Link partner report 4481. Nov 29 12:47:26 turpin kernel: eth0: 21143 negotiation status 00c6, MII. Nov 29 12:47:26 turpin kernel: eth0: MII status 782d, Link partner report 4481. Nov 29 12:48:26 turpin kernel: eth0: 21143 negotiation status 00c6, MII. Nov 29 12:48:26 turpin kernel: eth0: MII status 782d, Link partner report 4481. Nov 29 12:49:26 turpin kernel: eth0: 21143 negotiation status 00c6, MII. Nov 29 12:49:26 turpin kernel: eth0: MII status 7829, Link partner report 4481. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf0670040 new csr5=0xf066. # a connect request Nov 29 12:49:53 turpin kernel: In tulip_rx(), entry 0 00400720. Nov 29 12:49:53 turpin kernel: eth0: In tulip_rx(), entry 0 00400720. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf066 new csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: exiting interrupt, csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf0670004 new csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf0670040 new csr5=0xf066. Nov 29 12:49:53 turpin kernel: In tulip_rx(), entry 1 00420320. Nov 29 12:49:53 turpin kernel: eth0: In tulip_rx(), entry 1 00420320. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf066 new csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: exiting interrupt, csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf0670004 new csr5=0xf066. Nov 29 12:49:53 turpin kernel: eth0: interrupt csr5=0xf0670040 new csr5=0xf066. So I get 500k of that, and the log ends like this after the kernel deadlock: Nov 29 12:54:29 turpin kernel: In tulip_rx(), entry 27 00400320. Nov 29 12:54:29 turpin kernel: eth0: In tulip_rx(), entry 27 00400320. Nov 29 12:54:29 turpin kernel: eth0: interrupt csr5=0xf066 new csr5=0xf066. Nov 29 12:54:29 turpin kernel: eth0: exiting interrupt, csr5=0xf066. Nov 29 12:54:29 turpin kernel: eth0: interrupt csr5=0xf0670040 new csr5=0xf066. Nov 29 12:54:29 turpin kernel: In tulip_rx(), entry 28 00400320. It simply stops (because the kernel stopped). In that 500k of reporting interrupts and anything else of interest, there are no "... error ...", no "... warning ...", no "... stopped ...", no ring buffer address mismatches, no reconfigurations, no "interrupt called from interrupt handler", nothing beyond routine interrupt service reports. Whatever the problem is, tulip.c doesn't notice it in full debug mode. (Think about it: the changes between the working .90 in 2.0.38 and provisionally suspect .91g in 2.2.15+ with respect to a DS21143 are miniscule, while the changes to the rest of the device, networking, and network-related vm layer between 2.0.38 and 2.2.17-2.2.18pre23 are gigantic. What would you suspect? An overlooked lock that needs to be there? You bet. It could be a lock that the 2.2.x tulip driver needs to set and does not, but that is not a given, and if so it is odd that the maintainer and the people that have hacked the tulip driver to do experimental hardware flow control, etc, haven't noticed it by now.) imho, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: tulip log (additional detail: single-threaded httpd)
Note: if anyone else is wondering what may be deadlocking 2.2.17+ in the context of http connects over ethernet (assuming that it is not the ethernet driver itself): it is also not the httpd server's use of linux kernel threads (wn is single threaded). And there isn't much going on in the background when this is tested. A bunch of daemons are running, but only the k*d (kupdated, kflushd, kpiod, kswapd, klogd) and utmpd besides the httpd server itself ever run during the test. Tests were conducted both with inetd handling the http connect and with wnsd listening on the http port directly, no difference. Unless one of those k*d daemons uses kernel threads, kernel threads exported to user space are not possibly the source of the deadlock. (Might have looked that way due to the stack corruption in the httpd parent that strace seems to see a few connects before the kernel deadlock.) Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
tulip-ethernet 2.2.17-2.2.18preN http nethangs fixed
This problem is solved. Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers (just kidding) It was the cache alignment setting in tulip.c (saved me sifting through the bowels of the pci code to find it). I choose "486" for the cpu in "make config" for the kernel for a Cyrix 5x86 running on an Asus sp3 with sis496 pci 2.0 bus (it has been wonderfully stable for 5 years this way, why rock the boat). The .90 driver in 2.0.38 detects the "486" and sets the cache alignment to 0x01a0480. The .91g-ppc driver in 2.2.15+ doesn't check to see if we're running a 486 or lower, it simply sets CSR0 to 0x01A08000 if __i386__ is defined. Using "static int csr0 = 0x01A0 | 0x4800;" and recompiling fixed the http-induced kernel deadlocks that I was seeing. I could send a patch, but I don't know if the variable that the code in .90 was using is defined in 2.2.1x, etc, whether it is deprecated, various stuff that anyone maintaining 2.2.x pci ethernet drivers probably already knows. But that fixed it. -- make config for 5x86 note: Both the AMD and Cyrix, different as they are, should be 486 IIRC. The AMD certainly does not implement the p5 instruction set. The Cyrix might benefit from 586 instruction scheduling, but I'd have to see it to believe it, and it certainly won't help the AMD5x86, which is merely a fast 486dx with a bigger cache. If your gcc specs file considers "-m586" to mean both -mcpu=i586 and -march=i586, you are doomed if you compile with that flag for sure for the AMD, and maybe for the Cyrix 5x86 too. Who knows whether the p5 instructions all worked in the m1sc, even if that logic was on the chip before they hacked the m1sc to use the narrow 486 memory bus? IIRC Cyrix never claimed that you could run p5 binaries on it. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: problems with Tulip driver in 2.2 and 2.4 (true 21143 in 2.2.x, too)
It is not only the "almost standard" tulip clones that have problems in 2.2.1x. Stock Debian potato (2.2.17-pre6, IIRC) i386 kernel, Kingston KNE100TX w/i21143: works fine in 2.0.38 (.90 driver), hung the kernel solid during an ftp running the potato kernel (100/half-duplex). It was not quite a standard ftp exchange (win98 client sending a typo, i.e. invalid ftp command, rather than a file transfer, when it hung), but regardless, deadlocking the kernel without so much as a squeak of an error message probably is not the intended result even in the presence of invalid input to a userspace server. (Assuming that the deadlock was not the product of some untirely unrelated problem that happened just at that moment. ext2fs is a little squirrelly with that 2.2.16+ kernel, too, fixable reference count anomalies turning up more-or-less randomly about every 10th e2fsck, different partitions and drives that never show such symptoms running 2.0.38. It's not as if the ext2fs metadata write code in 2.2.17 has big revisions from 2.0.38. pci bus in question is 2.0, no apic.) Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
update: i21143 (tulip) and 2.2.17 (upgrade to .91-g and egcs-1.1.2 seems to have fixed it)
Update on the hang during ftp on stock Debian potato (2.2.17-pre6 IIRC) with i21143 tulip pci ethernet: I changed to gcc-2.91.66 (egcs-1.1.2; Debian afaik used gcc 2.7.2.3 or 2.95.2 to compile the pre-compiled vanilla distribution kernel in potato) and upgraded the 2.2.17-pre_something kernel to the final 2.2.17 (with the .91-g tulip driver), using binutils-2.9.1.0.25, glibc-2.1.3, and compiled a monolithic kernel while running 2.0.38. Everything seems ok (simple needs). Too many changes to pin down the cause of the earlier deadlock that I reported, and it's too soon to see if there will still be sporadic wrong ext2fs block counts at e2fsck time, but no deadlocks so far, no oopses, ftp works, ping works, http works, etc. So the kernel upgrade and kernel compiler change in combination is at least much more stable, without being to say why exactly (ie "for anyone that had doubts, there are kernels after 2.0.38 where a real tulip card is reliable, if you have the lucky compiler"). :-), Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
question about tulip patch to set CSR0 for pci 2.0 bus
Shouldn't the setting of the CSR0 value for x86 switch between normal (0x01A08000) and cautious (0x01A04800) based on some notion of what generation of pci bus is installed rather than what cpu the kernel is compiled for? That's one thing that bothered me about the method that the .90 driver used. It worked for me, of course, cool, but when I thought about putting a real general purpose patch into later versions of tulip.c to solve the same problem, it bothered me that the old method assumes an association between pci bus and cpu that may not be valid. I don't know that there are any 386/486/5x86 systems that can use the 0x01A08000 setting (that apparently works on most x86 pci 2.1 busses), but then again I don't know that there aren't, either. If the pci bus level is 2.0, it makes sense to use the cautious CSR0 setting, for the same reasons that the .90 tulip.c in 2.0.38 does, and if the pci level is 2.1, you aren't taking any chances with 0x01A08000 that the driver doesn't take now. The pci driver, initialized before any pci devices, appears to know whether you have a pci 2.0 or pci 2.1 bus, so why not use that information instead of cpu generation? (Merely a suggestion) Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "" - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Signal 11
This is unrelated to the signal 11 problem, but something to consider for "random crashes and segfaults", ie are you using this compiler and glibc version combination. There has a been a thread on the teTeX mailing list the last few days about a (RedHat, but probably more general than just their rpms) gcc-2.9.6 w/glibc-2.2.x bug. At -O2, it can miscompile unsigned varname; /* "unsigned int varname;" is ok */ (no problem at -O or no optimization at all, and doesn't happen if teTeX is compiled with kgcc). Showed up in the kpathsea library (which began to split paths on '-' as well as '/' after a user upgraded compiler and libc and recompiled teTeX). Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: question about tulip patch to set CSR0 for pci 2.0 bus
SIS 85C496 on Asus sp3 appears to have an equivalent bug (same CSR0 setting fixes kernel deadlocks that occur when using the non-burst-challenged 0x01A08000 setting), so if you're going to catch it with a special case in the pci code, you need to catch that as well as the Saturn II chipset. Note: the pci probe (cat /proc/pci) on this machine says "No bursts" for the host bridge. Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
klogd -f logname "append to logname" patch
(For anyone else who shares the opinion that
klogd -f /var/log/kmsg -x
should append to its kmsg log file instead of overwriting it.)
-f insulates klogd messages from syslogd bugs (although you can't split
them up into multiple files the way that /etc/syslog.conf enables). You
can always solve any klogd-to-syslogd problem *after* you find out what
is otherwise wrong with your kernel (and lose the -f switch then if that's
what you prefer or require). But when you are debugging a kernel deadlock
with no oops, for example, and actually have local persistent storage
to write to, it helps if the next hard reset doesn't truncate the -f
pathname when klogd opens it.
--- klogd.c.origMon Sep 18 00:34:11 2000
+++ klogd.c Wed Nov 29 14:06:40 2000
@@ -1109,7 +1109,7 @@
{
if ( strcmp(output, "-") == 0 )
output_file = stdout;
- else if ( (output_file = fopen(output, "w")) == (FILE *) 0 )
+ else if ( (output_file = fopen(output, "a")) == (FILE *) 0 )
{
fprintf(stderr, "klogd: Cannot open output file " \
"%s - %s\n", output, strerror(errno));
Regards,
Clayton Weaver
<mailto:[EMAIL PROTECTED]>
(Seattle)
"Everybody's ignorant, just in different subjects." Will Rogers
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
makefile bash2 typo (Re: We interrupt you regularly scheduled catfight for.. Linux 2.2.18pre13)
> [] > Another problem in Makefile. I guess this change between pre 12 and 13 > is a typo: > CONFIG_SHELL := $(shell if [ -x "$$BASH" ]; then echo $$BASH; \ > - else if [ -x /bin/bash ]; then echo /bin/bash; \ > + else if [ -x /bin/bash ]; then echo /bin/bash2; \ >else echo sh; fi ; fi) > [] What is the second "fi" for? Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
PID bit-width
Has everyone forgotten the old coda_fs-security discussion and the question of how to be sure that you are still talking to the original process on a system fast enough to wrap the pid counter in < 1 sec? (That question doesn't have to be solved with the pid, you can use a wide cookie, but if coda can potentially have this problem, so can the kernel and network in other contexts.) Sheesh, these things may need to become 64-bit as well as "random select next pid", just for security reasons, not because anyone is really running that many concurrent processes/kernel_threads. More convenient user display formatting is far down the priority list here compared to security and uniquely identifying processes even in the context of an intelligent attack on fast hardware that may not even have been invented yet. Convenience is good. Security is mission-critical. Regards, Clayton Weaver <mailto:[EMAIL PROTECTED]> (Seattle) "Everybody's ignorant, just in different subjects." Will Rogers - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: xor as a lazy comparison
Is not xor (^) typically compiled to a one cycle instruction regardless of requested optimization level? (May not always have been the case on every target architecture for != equality tests.) Clayton Weaver cgweav at fastmail dot fm PS: Anyone know where I can get a waterproof, battery powered gps that will fit inside the handle of a Fenwick? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
