Re: kern/133902: [tun] Killing tun0 iface ssh tunnel causes Panic String: page fault
Synopsis: [tun] Killing tun0 iface ssh tunnel causes Panic String: page fault State-Changed-From-To: open-closed State-Changed-By: bz State-Changed-When: Fri Oct 15 09:45:00 UTC 2010 State-Changed-Why: Duplicate of PR kern/116837. Responsible-Changed-From-To: freebsd-net-bz Responsible-Changed-By: bz Responsible-Changed-When: Fri Oct 15 09:45:00 UTC 2010 Responsible-Changed-Why: Take in case of follow-ups. http://www.freebsd.org/cgi/query-pr.cgi?pr=133902 ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: NFE adapter 'hangs'
On 4 Sep 2010, at 01:53, Pyun YongHyeon wrote: On Fri, Sep 03, 2010 at 07:59:26AM +0100, Melissa Jenkins wrote: Thank you for your very quick response :) [...] Also I'd like to know whether both RX and TX are dead or only one RX/TX path is hung. Can you see incoming traffic with tcpdump when you think the controller is in stuck? Yes, though not very much. The traffic to 4800 is every second so you can see in the following trace when it stops 07:10:42.287163 IP 192.168.1.203 224.0.0.240: pfsync 108 07:10:42.911995 07:10:43.112073 STP 802.1d, Config, Flags [Topology change], bridge-id 8000.c4:7d:4f:a9:ac:30.8008, length 43 07:10:43.148659 IP 192.168.1.203.57026 192.168.1.255.4800: UDP, length 60 07:10:43.148684 IP 172.31.1.203 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 192.168.1.129.4800: UDP, length 60 07:10:43.148689 IP 172.31.1.203 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 192.168.1.1.4800: UDP, length 60 07:10:43.148918 IP 192.168.1.213.40677 192.168.1.255.4800: UDP, length 48 [...] a bit later on, still broken, a slight odd message: 07:11:43.079720 IP 172.31.1.129 172.31.1.213: GREv0, length 52: IP 192.168.1.129.60446 192.168.1.213.179: tcp 12 [bad hdr length 16 - too short, 20] 07:11:44.210794 IP 172.31.1.129 172.31.1.203: GREv0, length 84: IP 192.168.1.129.64744 192.168.1.203.4800: UDP, length 52 07:11:44.210831 IP 172.31.1.129 172.31.1.213: GREv0, length 84: IP 192.168.1.129.64744 192.168.1.213.4800: UDP, length 52 Now this really is odd, I don't recognise either of those MAC addresses, though the SQL shown is used on this machine ( 07:12:13.054393 45:43:54:20:41:63 00:00:03:53:45:4c, ethertype Unknown (0x6374), length 60: 0x: 556e 6971 7565 4964 2046 524f 4d20 7261 UniqueId.FROM.ra 0x0010: 6461 6363 7420 2057 4845 5245 2043 616c dacct..WHERE.Cal 0x0020: 6c69 6e67 5374 6174 696f 6e49 6420 lingStationId. Hmm, it seems you're using really complex setup. It's very hard to narrow down guilty ones under these environments. Could you setup simple network configuration that reproduces the issue? One of possible cause would be wrong(garbled) data might be passed up to upper stack. But I have no idea why you see GRE packets with truncated TCP header(172.31.1.129 172.31.1.213). How about disabling TX/RX checksum offloading as well as TSO? [...] I then restarted the interface (nfe down/up, route restart) From dmesg at the time (slight obfuscated) Sep 3 07:10:19 manch2 bgpd[89612]: neighbor XX: received notification: HoldTimer expired, unknown subcode 0 Sep 3 07:10:49 manch2 bgpd[89612]: neighbor XX connect: Host is down # at this point I took the interface down up and reloaded the routing tables Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: nfe0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:11 manch2 kernel: nfe0: link state changed to UP Sep 3 07:12:11 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:14 manch2 kernel: carp0: link state changed to UP Hmm, it does not look right, carp0 showed link DOWN message four times in a row. By the way, are you using IPMI on MCP55? nfe(4) is not ready to handle MAC operation with IPMI. Turning off tx rc checksum offloading seems to have resolved the problem: ifconfig nfe0 -txcsum -rxcsum Seems to have stopped both the corruption and the interface hanging. I ran it for about 16 hours on the FreeBSD 8 box. It also appears to have fixed the problem on my FreeBSD 7 machine as well. I didn't try turning off TSO. Thank you for your suggestion help! Mel ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: [PATCH] Netdump for review and testing -- preliminary version
On Thu, 14 Oct 2010, Attilio Rao wrote: No, what I'm saying is: UMA needs to not call its drain handlers, and ideally not call into VM to fill slabs, from the dumping context. That's easy to implement and will cause the dump to fail rather than causing the system to hang. My point is, however, still the same: that should not happen just for the netdump specific case but for all the dumping/KDB/panic cases (I know it is unlikely current code !netdump calls into UMA but it is not an established pre-requisite and may still happen that some added code does). I still see this as a weakness on the infrastructure, independently from netdump. I can see that your point is that it is vital to netdump correct behaviour though, so I'd wonder if it worths fixing it now or later. Quite a bit of our kernel and dumping infrastructure special cases debugging and dumping behavior to avoid sources of non-robustness. For example, serial drivers avoid locking, and for disk dumps we bypass GEOM to avoid the memory allocation, freeing, and threading that it depends on. The goal here is to be robust when handling dumps: hanging is worse than not dumping, since you won't get the dump either way, and if you don't reboot then the system requires manual intervention to recover. Example of things that are critical to avoid include: - The dumping thread tripping over locks held by the panicked thread, or by another now-suspended thread, leading to deadlock against a suspended thread. - Corrupting dumps by increasing concurrency in the panic case. We ran into a case a year or two ago where changing VM state during the dump on amd64 caused file system corruption as the dump code assumed that the space required for a dump didn't change while dumping took place. Any code dependency we add in the panic / KDB / dump path is one more risk that we don't successfully dump and reboot, so we need to minimize that code. Robert ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: [PATCH] Netdump for review and testing -- preliminary version
On 15 Oct 2010, at 20:39, Garrett Cooper wrote: But there are already some cases that aren't properly handled today in the ddb area dealing with dumping that aren't handled properly. Take for instance the following two scenarios: 1. Call doadump twice from the debugger. 2. Call doadump, exit the debugger, reenter the debugger, and call doadump again. Both of these scenarios hang reliably for me. I'm not saying that we should regress things further, but I'm just noting that there are most likely a chunk of edgecases that aren't being handled properly when doing dumps that could be handled better / fixed. Right: one of the points I've made to Attilio is that we need to move to a more principled model as to what sorts of things we allow in various kernel environments. The early boot is a special environment -- so is the debugger, but the debugger on panic is not the same as the debugger when you can continue. Likewise, the crash dumping code is special, but also not the same as the debugger. Right now, exceptional behaviour to limit hangs/etc is done inconsistently. We need to develop a set of principles that tell us what is permitted in what contexts, and then use that to drive design decisions, normalizing what's there already. This is not dissimilar to what we do with locking already, BTW: we define a set of kernel environments (fast interrupt handlers, non-sleepable threads, sleepable thread holding non-sleepable locks, etc), and based on those principles prevent significant sources of instability that might otherwise arise in a complex, concurrent kernel. We need to apply the same sort of approach to handling kernel debugging and crashing. BTW, my view is that except in very exceptional cases, it should not be possible to continue after generating a dump. Dumps often cause disk controllers to get reset, which may leave outstanding I/O in nasty situations. Unless the dump device and model is known not to interfere with operation, we should set state indicating that the system is non-continuable once a dump has occurred. Robert ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: [PATCH] Netdump for review and testing -- preliminary version
On Fri, Oct 15, 2010 at 12:25 PM, Robert Watson rwat...@freebsd.org wrote: On Thu, 14 Oct 2010, Attilio Rao wrote: No, what I'm saying is: UMA needs to not call its drain handlers, and ideally not call into VM to fill slabs, from the dumping context. That's easy to implement and will cause the dump to fail rather than causing the system to hang. My point is, however, still the same: that should not happen just for the netdump specific case but for all the dumping/KDB/panic cases (I know it is unlikely current code !netdump calls into UMA but it is not an established pre-requisite and may still happen that some added code does). I still see this as a weakness on the infrastructure, independently from netdump. I can see that your point is that it is vital to netdump correct behaviour though, so I'd wonder if it worths fixing it now or later. Quite a bit of our kernel and dumping infrastructure special cases debugging and dumping behavior to avoid sources of non-robustness. For example, serial drivers avoid locking, and for disk dumps we bypass GEOM to avoid the memory allocation, freeing, and threading that it depends on. The goal here is to be robust when handling dumps: hanging is worse than not dumping, since you won't get the dump either way, and if you don't reboot then the system requires manual intervention to recover. Example of things that are critical to avoid include: - The dumping thread tripping over locks held by the panicked thread, or by another now-suspended thread, leading to deadlock against a suspended thread. - Corrupting dumps by increasing concurrency in the panic case. We ran into a case a year or two ago where changing VM state during the dump on amd64 caused file system corruption as the dump code assumed that the space required for a dump didn't change while dumping took place. Any code dependency we add in the panic / KDB / dump path is one more risk that we don't successfully dump and reboot, so we need to minimize that code. But there are already some cases that aren't properly handled today in the ddb area dealing with dumping that aren't handled properly. Take for instance the following two scenarios: 1. Call doadump twice from the debugger. 2. Call doadump, exit the debugger, reenter the debugger, and call doadump again. Both of these scenarios hang reliably for me. I'm not saying that we should regress things further, but I'm just noting that there are most likely a chunk of edgecases that aren't being handled properly when doing dumps that could be handled better / fixed. Thanks, -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
Re: NFE adapter 'hangs'
On Fri, Oct 15, 2010 at 01:25:08PM +0100, Melissa Jenkins wrote: On 4 Sep 2010, at 01:53, Pyun YongHyeon wrote: On Fri, Sep 03, 2010 at 07:59:26AM +0100, Melissa Jenkins wrote: Thank you for your very quick response :) [...] Also I'd like to know whether both RX and TX are dead or only one RX/TX path is hung. Can you see incoming traffic with tcpdump when you think the controller is in stuck? Yes, though not very much. The traffic to 4800 is every second so you can see in the following trace when it stops 07:10:42.287163 IP 192.168.1.203 224.0.0.240: pfsync 108 07:10:42.911995 07:10:43.112073 STP 802.1d, Config, Flags [Topology change], bridge-id 8000.c4:7d:4f:a9:ac:30.8008, length 43 07:10:43.148659 IP 192.168.1.203.57026 192.168.1.255.4800: UDP, length 60 07:10:43.148684 IP 172.31.1.203 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 192.168.1.129.4800: UDP, length 60 07:10:43.148689 IP 172.31.1.203 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 192.168.1.1.4800: UDP, length 60 07:10:43.148918 IP 192.168.1.213.40677 192.168.1.255.4800: UDP, length 48 [...] a bit later on, still broken, a slight odd message: 07:11:43.079720 IP 172.31.1.129 172.31.1.213: GREv0, length 52: IP 192.168.1.129.60446 192.168.1.213.179: tcp 12 [bad hdr length 16 - too short, 20] 07:11:44.210794 IP 172.31.1.129 172.31.1.203: GREv0, length 84: IP 192.168.1.129.64744 192.168.1.203.4800: UDP, length 52 07:11:44.210831 IP 172.31.1.129 172.31.1.213: GREv0, length 84: IP 192.168.1.129.64744 192.168.1.213.4800: UDP, length 52 Now this really is odd, I don't recognise either of those MAC addresses, though the SQL shown is used on this machine ( 07:12:13.054393 45:43:54:20:41:63 00:00:03:53:45:4c, ethertype Unknown (0x6374), length 60: 0x: 556e 6971 7565 4964 2046 524f 4d20 7261 UniqueId.FROM.ra 0x0010: 6461 6363 7420 2057 4845 5245 2043 616c dacct..WHERE.Cal 0x0020: 6c69 6e67 5374 6174 696f 6e49 6420 lingStationId. Hmm, it seems you're using really complex setup. It's very hard to narrow down guilty ones under these environments. Could you setup simple network configuration that reproduces the issue? One of possible cause would be wrong(garbled) data might be passed up to upper stack. But I have no idea why you see GRE packets with truncated TCP header(172.31.1.129 172.31.1.213). How about disabling TX/RX checksum offloading as well as TSO? [...] I then restarted the interface (nfe down/up, route restart) From dmesg at the time (slight obfuscated) Sep 3 07:10:19 manch2 bgpd[89612]: neighbor XX: received notification: HoldTimer expired, unknown subcode 0 Sep 3 07:10:49 manch2 bgpd[89612]: neighbor XX connect: Host is down # at this point I took the interface down up and reloaded the routing tables Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: nfe0: link state changed to DOWN Sep 3 07:12:07 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:11 manch2 kernel: nfe0: link state changed to UP Sep 3 07:12:11 manch2 kernel: carp0: link state changed to DOWN Sep 3 07:12:14 manch2 kernel: carp0: link state changed to UP Hmm, it does not look right, carp0 showed link DOWN message four times in a row. By the way, are you using IPMI on MCP55? nfe(4) is not ready to handle MAC operation with IPMI. Turning off tx rc checksum offloading seems to have resolved the problem: ifconfig nfe0 -txcsum -rxcsum Seems to have stopped both the corruption and the interface hanging. I ran it for about 16 hours on the FreeBSD 8 box. It also appears to have fixed the problem on my FreeBSD 7 machine as well. Hmm, could you try the patch at the following URL? http://people.freebsd.org/~yongari/nfe/nfe.mcp55.txcsum.patch The patch ensures that the first fragment of mbuf holds ip/tcp/udp header including options. If this patch fix the issue then it means there is an issue in TX checksum offloading on MCP55. But I'm still not sure whether it makes any difference because there was no report on broken TX checksum offloading on nfe(4). At least I don't remember that kind of report so far. Note, the patch was not tested at all, I have no longer access to nfe(4) controllers so please make sure to test it first before applying the patch. I didn't try turning off TSO. Ok, your tcpdump shows garbled packets for non-TSO frames so the patch above does no special handling for TSO case. Thank you for your suggestion help! Mel ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
