Oops in 2.6.23-rc5
Hi, today I switched from 2.6.22.3 to 2.6.23-rc5 (skipped quite a few -rc versions due to lack of time), and the box keeps panicking under certain circumstances. I suspected disk related problems, because: when the box is up, I usually resume ~10 bittorrent files. When doing this, each file (~200MB...1GB) is checked and disk activity is pretty high (20MB/s or so), and after 1 minute of doing so the box panicks. Every time. However, I could not reproduce it while generating disk-io with say tar or rsync to the same fs. It always panicked when the torrent client(s) start up. As the box would not log anything via remote-syslog before halting, I connected a vga display. As I don't have a digital camera, I tried to write down some stuff: http://ww.nerdbynature.de/bits/2.6.23-rc5/ (I'll try to write down the full oops to this place, or what was still visible from it, because the first few(?) lines where lost, display scrollback was not working, only sysrq was). The backtrace mentions do_page_fault, error_code, tcp_rtt_estimator, tcp_ack_saw_timestamp, tcp_ack, tcp_rcv_established, tcp_v4_do_rcv, tcp_v4_rcv, ip_local_delimiter, netif_receive_skb, process_backlog, net_rcv_activate, __do_softirq, do_softirq - in that order. As said, the correct addresses will be put on above's url (Q: do I really need *all* the numbers? Or just a few?). These snippets made me suspect network related issues, because: aside from disk-io, the bittorrent clients will establish quite a few (~50 in total) connections to all the peers. The box is a amd-k7, 2 NICs (forcedeth, 3c59x), 2 GB RAM, ACPI disabled, gcc-4.1 Thanks for looking into this, Christian. -- BOFH excuse #335: the AA battery in the wallclock sends magnetic interference - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Oops in 2.6.23-rc5
On Sun, 2 Sep 2007, Herbert Xu wrote: You want this patch (by davem). I applied the patch and the box is up for 1hr now. Since I was able to reproduce the oops pretty reliable with this bittorrent thingy, I did the same a few times now, but the box did NOT crash :) Unfortunately people are travelling so I'm not sure when it'll get picked up by Linus. I've seen this patch only in: http://article.gmane.org/gmane.linux.network/70781 And, for the archives, a simliar looking error report: http://article.gmane.org/gmane.linux.network/70777 Thanks for the quick reply, Herbert! Christian. -- BOFH excuse #297: Too many interrupts - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
2.6.23-rc5: possible irq lock inversion dependency detected
Hi, after upgrading to 2.6.23-rc5 (and applying davem's fix [0]), lockdep was quite noisy when I tried to shape my external (wireless) interface: [ 6400.534545] FahCore_78.exe/3552 just changed the state of lock: [ 6400.534713] (&dev->ingress_lock){-+..}, at: [] netif_receive_skb+0x2d5/0x3c0 [ 6400.534941] but this lock took another, soft-read-irq-unsafe lock in the past: [ 6400.535145] (police_lock){-.--} This happened when I executed: http://nerdbynature.de/bits/2.6.23-rc5/qos.sh.txt (using iproute2-ss070313). The is still running, I just noticed a short hickup, probably when it was busy writing the warning to the disk. More details and .config: http://nerdbynature.de/bits/2.6.23-rc5/ I'm not really sure what the application mentioned in the message above has to do with this: the application[1] has been running since bootup as a non-privileged user and did so for earlier kernel versions too. Christian. [0] http://lkml.org/lkml/2007/9/2/6 [1] http://folding.stanford.edu/linux.html -- BOFH excuse #294: PCMCIA slave driver - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RTNL: assertion failed at net/core/dev.c
Wow, I should really update more often. Skipping the last -rc versions AND adding a new device (zd1211rw) to the box turns out to be quite interesting ([0],[1]). However, this time loading of a (proprietary) module is involved. Knowing that lkml cannot really help here (and I should contact vmware), I just wanted to let you netdev guys know, because the only occurences I found on the net were from 1999, but given the amount of changes currently going into net/ I thought this might be interesting: [15604.137408] RTNL: assertion failed at net/core/dev.c (2595) [15604.137772] [] show_trace_log_lvl+0x1a/0x30 [15604.138121] [] show_trace+0x12/0x20 [15604.138449] [] dump_stack+0x15/0x20 [15604.138807] [] __dev_set_promiscuity+0xc2/0xd0 [15604.139163] [] dev_set_promiscuity+0x1b/0x40 [15604.139515] [] VNetBridgeStartPromisc+0x2b/0x50 [vmnet] [15604.139896] [] VNetBridgePortsChanged+0x2a/0x40 [vmnet] [15604.140276] [] VNetHubPortsChanged+0x65/0xc0 [vmnet] [15604.140648] [] VNetConnect+0x7a/0xb0 [vmnet] [15604.141000] [] VNetFileOpOpen+0xbd/0x170 [vmnet] [15604.141362] [] chrdev_open+0x83/0x180 [15604.141696] [] __dentry_open+0xa1/0x1a0 [15604.142036] [] nameidata_to_filp+0x35/0x40 [15604.142383] [] do_filp_open+0x49/0x60 [15604.142717] [] do_sys_open+0x45/0x80 [15604.142957] [] sys_open+0x1c/0x20 [15604.143087] [] sysenter_past_esp+0x6b/0xb5 [15604.143227] === details: http://nerdbynature.de/bits/2.6.23-rc5/ Christian. [0] http://www.spinics.net/lists/netdev/msg40247.html [1] http://www.spinics.net/lists/netdev/msg40281.html -- BOFH excuse #31: cellular telephone interference - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RTNL: assertion failed at net/core/dev.c
On Sun, 2 Sep 2007, Stephen Hemminger wrote: Vendor module calls kernel api incorrectly. dev_set_promiscuity requires that the calling thread hold rtnl mutex (ie call rtnl_lock). It's their bug, netdev doesn't want to hear about it. OK, that's all I needed to know. Thank you both for your comments! Christian. -- BOFH excuse #435: Internet shut down due to maintenance - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
2.6.20.4: NETDEV WATCHDOG and lockups
Hi there, we have serious problems with 2 of our servers: both shiny new amd64 dual core, with both 2GB RAM, 32bit kernel+userland (Debian/testing). Both servers have 2 NICs, RTL8139 (eth0, irq10) and RTL8169s (eth1, irq11). Both boxes are running fine but after "a while" they lock up and eventually restart all of a sudden. The last messages in the logfile are: 14:15:11 db2 kernel: NETDEV WATCHDOG: eth0: transmit timed out 14:15:14 db2 kernel: eth0: link up, 100Mbps, full-duplex, lpa 0x45E1 Then the box reboots, nothing else in the log. As the servers have been set up recently, we only know that it happend with Debian's 2.6.17-? kernel. When we upgraded the installation, we went to 2.6.18-4-k7 and the problem persistent. We're using now vanilla 2.6.20.4 and while the problem persists, it takes longer to lockup (~20h as opposed to 4-5h). While this is a good thing for us, it's now harder to reproduce (we have to wait longer). Searching the archives turned up quite a few results but no real fix and lots of old postings too. We then disabled ACPI completely and booted with 'noapic'. Now both boxes are running for > 20h and we're curious how long they make it. However, booting with 'noapic' slowed down both servers *a lot*. From /proc/interrupts we can see that only CPU0 (core 0) is handling interrupts while CPU1 does not. We compiled with CONFIG_IRQBALANCE=n so that irqbalance(1) would work - but to no avail. Please see http://nerdbynature.de/bits/2.6.20.4/ for details for both hosts and feel free to ask for more details. Although both boxes are in production we'll be happy test more bootoptions/patches and the like. TIA, Christian. -- BOFH excuse #266: All of the packets are empty. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Mon, 2 Apr 2007, Chuck Ebbert wrote: Please see http://nerdbynature.de/bits/2.6.20.4/ for details for both hosts and feel free to ask for more details. Although both boxes are in production we'll be happy test more bootoptions/patches and the like. Where is the info from before you changed to "noapic"? Or were the machines always using XT-PIC for all the interrupts??? Who, didn't notice the XT-PIC in /proc/interrupts. From the count of your questionmarks I figure it's a bad thing then? @Malte, do we still have this information around? Thanks, Christian. -- BOFH excuse #63: not properly grounded, please bury computer - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Mon, 2 Apr 2007, Chuck Ebbert wrote: Where is the info from before you changed to "noapic"? Or were the machines always using XT-PIC for all the interrupts??? XT-PIC is only used since we switched to noapic, before there was IO-APIC-fasteoi on both ethernet cards and interrupts were balanced well. Thanks, Christian. -- BOFH excuse #340: Well fix that in the next (upgrade, update, patch release, service pack). - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Tue, 3 Apr 2007, Len Brown wrote: Which increased stability, disabling ACPI, or disabling the IOAPIC? To be honest, we're not sure. See below. Your box has MPS, so you should be able to use the IOAPIC in either mode. MPS - Multiprocessor Specification? SMP? Yes, it'd be good to use the IOAPIC again. Note that you can do these both independently at boot-time with "acpi=off" and "noapic", respectively. eg. 4 combos 1. 2. noapic 3. acpi=off 4. acpi=off noapic you started with #1, and are running hard-coded #4 now, but skipped #2 and #3 Indeed, we skipped quite a few options. As mentioned before, the boxes are in production already so we don't have much time to play around and we were just happy when they survived a few hours :( But yes, we'll try booting with "acpi=off" and enabled IOAPIC again. @Malte: when will we be able to do so? Len et al., do you even suggest to use ACPI on a server system at all? I myself always thought of ACPI being evil and to avoid when possible (thus switching it off completely on a serversystem). Since these NETDEV WATCHDOG issues seems to be a "known issue" (kinda, since the many postings on the lists in the past), is there something else we should look into? Would more debug .config options help to find out why they lock up? Thanks for your comments, Christian. -- BOFH excuse #340: Well fix that in the next (upgrade, update, patch release, service pack). - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Tue, 3 Apr 2007, Jarek Poplawski wrote: Did you try with 8139cp instead of 8139too? I forgot about that, thanks. (Maybe even try some other card to narrow the problem?) We're try to convince our hosting provider to replace the NIC with a e1000. You could also try to test without ehci, if it's possible. Sure, I think USB is not needed. thanks for your input! Christian. -- BOFH excuse #387: Your computer's union contract is set to expire at midnight. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Mon, 2 Apr 2007, Chuck Ebbert wrote: Where is the info from before you changed to "noapic"? Or were the machines always using XT-PIC for all the interrupts??? We booted with 'acpi=off lapic' (with ACPI options compiled in, to be able to boot with acpi=on later on) and the box locked up again. I'll try to boot with a slightly different .config later on. With 'lapic' /proc/interrupts looks like: CPU0 CPU1 0: 63656 63396 IO-APIC-edge timer 1: 0 8 IO-APIC-edge i8042 2: 0 0XT-PIC-XTcascade 4: 6 1 IO-APIC-edge serial 6: 0 3 IO-APIC-edge floppy 8: 0 1 IO-APIC-edge rtc 17: 17050 1 IO-APIC-fasteoi eth1 18:102 80615 IO-APIC-fasteoi eth0 20: 195817 77721 IO-APIC-fasteoi libata NMI: 0 0 LOC: 126969 126970 ERR: 0 MIS: 0 Christian. -- BOFH excuse #146: Communications satellite used by the military for star wars. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Tue, 3 Apr 2007, Jarek Poplawski wrote: Did you try with 8139cp instead of 8139too? Tried that, 8139cp could not be loaded :( (Maybe even try some other card to narrow the problem?) You could also try to test without ehci, if it's possible. USB has been disabled completely. After booting with 'acpi=off lapic' the box survived ~30min then locked up again and rebooted. Christian. -- BOFH excuse #305: IRQ-problems with the Un-Interruptible-Power-Supply - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Tue, 3 Apr 2007, Francois Romieu wrote: Christian Kujau <[EMAIL PROTECTED]> : If the apic voodoo makes no difference, you can: 1 - leave it enabled Well, we tried to boot with ACPI compiled in again, but disabled during boot: - acpi=off lapic, crashed after 1h (almost exactly) of service - acpi=off lapic, crashed again, this time after 4h (almost exactly) - acpi=off noapic, still running, now 21h. The 2nd node has been booted with 'noapic' and ACPI *not* compiled in and is now running for 2,5 days. However, interrupts are not shared between cores. This means we still have to test booting with 'lapic' and ACPI enabled. Unfortnately there are a few more sub-options to choose from: - acpi=force -- enable ACPI if default was off - acpi=noirq -- do not use ACPI for IRQ routing - acpi=ht -- run only enough ACPI to enable Hyper Threading - acpi=strict -- Be less tolerant of platforms that are not strictly ACPI specification compliant. 2 - check that netconsole is not used with the 8139 (it is busted) 3 - check that netconsole is not used at all Actually I was thinking about *using* netconsole, since even setting up remote (userspace-)syslog left nothing on the syslog-server, when the machine crashed. But if it's b0rked in 8139, I will refrain from doing so. 4 - try: http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.21-rc5/r8169-20070402 Are they in -rc5 yet or 'not in -rc5 but should be applied to -rc5'? Thanks for your time, Christian. -- BOFH excuse #235: The new frame relay network hasn't bedded down the software loop transmitter yet. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Wed, 4 Apr 2007, Jarek Poplawski wrote: So, it's a lot sooner than before. (BTW, isn't there anything in debug log?) No, nothing. I've set up remote-syslgging to the other node (node1 logging to node2 and vice versa) - nothing :( I see both CPUs did interrupt handling again. Yes, when booting with 'lapic' both CPUs/cores are handling interrupts again. However, since 'lapic' seems to lead to crashes here, we would be more than happy to just boot with 'noapic' but have 'irqbalance' working. Unfortunately, irqbalance is unable to write to /proc/irq/*/smp_affinity (did not help to disable CONFIG_IRQBALANCE). Maybe it's a real locking problem. Here are some more suggestions for testing (if you don't find anything better): - try without SMP, so: 'acpi=off lapic nosmp' Yeah, we'll see if we still have time for trying this. But I figure this will not be a real (long term) option for us :( - lock debugging turned on as much as possible plus maybe for curiosity: - different CONFIG_HZ > - 8139TOO_PIO = y Indeed, that's what I too wanted to do. @Malte: any plans for another downtime? Thanks for your comments! Christian. -- BOFH excuse #265: The mouse escaped. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Wed, 4 Apr 2007, Christian Kujau wrote: Maybe it's a real locking problem. Here are some more suggestions for testing (if you don't find anything better): - try without SMP, so: 'acpi=off lapic nosmp' We were able to have our hosting provider to replace the 8139too with a E100, the onboard r8169 stayed of course. After this, the box came back fine...only to lock up again shortly after :( So again we spoke to our hosting provider and they just took out the 2 SATA disks and put them in a completely new system: amd64 dualcore again, 2 GB ram, r8169 onboard NIC, e100 pci-slot NIC. Now booting 2.6.20.4 and even 2.6.18-4-k7 (the debian kernel) with IOAPIC eabled seems to work, meaning the box is up since yesterday evening and interrupts are shared. Not equally, but still: # cat /proc/interrupts CPU0 CPU1 0:111 0 IO-APIC-edge timer 1: 7 9 IO-APIC-edge i8042 4:260 1 IO-APIC-edge serial 6: 0 3 IO-APIC-edge floppy 8: 0 1 IO-APIC-edge rtc 9: 0 0 IO-APIC-fasteoi acpi 16:157 575579 IO-APIC-fasteoi eth0 17:3812553 1 IO-APIC-fasteoi eth1 19: 1006518262484 IO-APIC-fasteoi libata NMI: 0 0 LOC: 17272991 17266237 ERR: 0 MIS: 0 While this is a good thing, we now have different problems: our 2nd sata drive is not usable any more, but we again we doubt hardware problems, because this disk has been replaced already back in the old box... but yes, this seem to be different problems, for the curious among you I've put details here: http://nerdbynature.de/bits/2.6.20.4/db2/ Thanks to all who have replied, Christian. -- BOFH excuse #209: Only people with names beginning with 'A' are getting mail this week (a la Microsoft) - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.20.4: NETDEV WATCHDOG and lockups
On Fri, 6 Apr 2007, Christian Kujau wrote: but yes, this seem to be different problems, for the curious among you I've put details here: http://nerdbynature.de/bits/2.6.20.4/db2/ that's http://nerdbynature.de/bits/2.6.20.4/db1/2/ sorry. -- BOFH excuse #270: Someone has messed up the kernel pointers - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
networking/ip-sysctl.txt: SRR or SSRR
In Documentation/networking/ip-sysctl.txt, the accept_source_route parameter is described with: Accept packets with SRR option. I could not find anything describing an "SRR option". The closest thing was something called "SSR - Strict Source Route"[0] - is that what is meant here? If so, RFC791 then refers to the same as "SSRR", as in "strict source and record route". Does the patch below clears that up or did I confuse it with something else? Thanks, Christian. Signed-off-by: Christian Kujau diff --git a/Documentation/networking/ip-sysctl.txt b/Documentation/networking/ip-sysctl.txt index b183e2b..db6b538 100644 --- a/Documentation/networking/ip-sysctl.txt +++ b/Documentation/networking/ip-sysctl.txt @@ -1057,9 +1057,9 @@ bootp_relay - BOOLEAN Not Implemented Yet. accept_source_route - BOOLEAN - Accept packets with SRR option. + Accept packets with SSRR option (RFC791 3.1). conf/all/accept_source_route must also be set to TRUE to accept packets - with SRR option on the interface + with SSRR option on the interface default TRUE (router) FALSE (host) [0] https://www.iana.org/assignments/ip-parameters/ip-parameters.xhtml -- BOFH excuse #224: Jan 9 16:41:27 huber su: 'su root' succeeded for on /dev/pts/1
Re: networking/ip-sysctl.txt: SRR or SSRR
On Tue, 10 May 2016, David Miller wrote: > Every single variable name and function name dealing with this IP > option uses "srr", therefore I don't think we should adjust the > documentation either. > > "srr" covers both the loose source route option and the strict source > route option. Now that I know where the 2nd "r" comes from, the "srr" makes sense. So yeah, SRR it is. Maybe a hint to the correct RFC would help, but if nobody else complained yet, I guess it's not really worth the commit. Thanks, Christian. -- BOFH excuse #255: Standing room only on the bus.
Re: [PATCH v3 1/3] siphash: add cryptographically secure hashtable function
On Thu, 15 Dec 2016, Jason A. Donenfeld wrote: > > I'd still drop the "24" unless you really think we're going to have > > multiple variants coming into the kernel. > > Okay. I don't have a problem with this, unless anybody has some reason > to the contrary. What if the 2/4-round version falls and we need more rounds to withstand future cryptoanalysis? We'd then have siphash_ and siphash48_ functions, no? My amateurish bike-shedding argument would be "let's keep the 24 then" :-) C. -- BOFH excuse #354: Chewing gum on /dev/sd3c