[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
*** This bug is a duplicate of bug 1317811 *** https://bugs.launchpad.net/bugs/1317811 We finally have a way to reproduce this at will. It turns out that any kernel newer than 3.7 (which has the change to handle compound pages that I mentioned in comment #7) will suffer from this problem. The work-around that Brian found in the previous comment will indeed work as it prevents the use of of fragments (A ethtool -K eth0 sg off should be enough). I will mark this bug as a duplicate of the newer one since that already has more detail info. ** This bug has been marked a duplicate of bug 1317811 Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
FWIW, we ran into the same messages and in researching, I came across this page https://silenteh.com/sysadmin/2013/08/08/amazon-ec2-xennet-skb-rides-the-rocket.html which suggests disabling TCP offload functionality using ethtool -K eth0 rx off tx off sg off tso off ufo off gso off gro off lro off to work around the problem. Disabling offload does in fact silence the messages (but also has the unfortunate side effect of dropping the MTU on boxes using jumbo frames) Given that that happens to quiet things down, figured it might help narrow down what patch or patches are responsible for a fix... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
Not to my knowledge. Also depends on the exact issue. Reading through the report there might be two separate issues. 1. Packet loss when this message appears. This is the the expected behaviour when trying to transmit a packet that would require more than 16+1 slots or 64kB. Compared to the Amazon kernel mentioned all kernels later than 3.7 have the following change: * xen/netfront: handle compound page fragments on transmit If there is something wrong with that, I have not yet seen a patch for it. There might be a related bug (bug #1275879) but hitting that would cause a BUG stacktrace in the guest. 2. That invalid socket state mentioned in comment #3. I wonder whether that could hint the netback driver running on the host side has closed the connection. There could be some messages to help in the host's dmesg but we cannot get to those (only Amazon can). If that is the case we should ensure those two patches are included in the kernel running in dom0 on the host: * xen-netback: don't disconnect frontend when seeing oversize packet * xen-netback: coalesce slots in TX path and fix regressions -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
Having the same issue. Log: 2014-03-25T15:39:21.750+00:00 kern/alert(1) kernel[]: [330160.244029] xen_netfront: xennet: skb rides the rocket: 19 slots running kernel version: 3.11.0-18-generic Is their any updates to this bug on a fix? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
The kernel version 3.8.0-19 sounds like the initially released kernel. There has been an inquiry about some xen related network patches by Matt Wilson on the xen-devel mailing list this morning. I looked for the patches and those have been applied and released with the 3.8.0-28.41 kernel (or higher). Btw, the Raring release recently went out of support (the first of the non-LTS releases with reduced duration of maintenance). The kernel currently gets a bit longer support but only as a hardware enablement option under Precise / 12.04). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
Oh, and for what it's worth, the same test runs fine on Amazon Linux, which is running kernel 3.4.57-48.42.amzn1.x86_64. There are no skb rides the rocket messages when the test is run on Amazon Linux, only when it's run on Ubuntu. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
I'm seeing this on 3.8.0-19-generic at Amazon EC2 as well. It's very reproducible (happens every time), and it causes a bizarre (theoretically impossible) state for my sockets. I have a client and a server on separate machines, both running Ubuntu 13.04 on EC2, and shortly into a stress test, all traffic ceases. I can watch the output from netstat on the two machines, and there is no change in the sockets' queue lengths. The client's socket has zero bytes in both its send queue and its receive queue. The server's socket has zero bytes in its receive queue and a few hundred thousand bytes in its send queue. Theoretically this state should be impossible unless the client has reduced its receive window size to zero. (I don't know any reason why it would be doing that, so I'm ruling that out.) Given that there is available space in the client's receive queue and pending bytes in the server's send queue, bytes should be moving from the server to the client, but this is not happening. The entire test hangs at this point until eventually the connection times out (after several minutes). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-lts-raring (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1195474] Re: xennet driver reports skb rides the rocket under moderate load
I am seeing this as well. I'm wondering if you've worked passed the issue. I'm thinking a different kernel more suitable for AWS? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1195474 Title: xennet driver reports skb rides the rocket under moderate load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1195474/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
