Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Fri, 31 Oct 2014 16:36:09 + Ben Hutchings wrote: We're going to document the live migration issue rather than fixing it. At work we are using libvirt guest suspend rather than shutdown across host reboots. It appears this feature uses migration because now we cannot start any of our VMs (error below). I am going to do a downgrade and upgrade dance to work around this bug but I really think that this issue should be have been fixed rather than documented :( kvm: Features 0x300fffe3 unsupported. Allowed features: 0x711fbbe3 qemu: warning: error while loading state for instance 0x0 of device ':00:03.0/virtio-net' -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On 30.10.2014 20:00, Ben Hutchings wrote: [...] Yes, I can see how this (disabling virtio features) would prevent live migration from old to new host kernels. You will probably need a one-time reboot of the guest when migrating to the new host kernel. We could either mention this in the DSA text, or keep UFO enabled even though it doesn't work correctly (in practice, we sort of have to do that as the tap device's feature flags aren't respected by guests - which I think is a QEMU bug). Please let us know whether live migration between two hosts running the new kernel version does work (I think it will). I confirm that. Although I did not exactly test this because of some constraints. I tested with virsh save/restore which uses the same codepaths as virsh migrate I think. This cold migration spewed out the same error messages as written earlier when migrating from old to new. And it went fine from new to new. BTW I use a patched qemu-kvm because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724758. So maybe there are only a few wheezy users doing live migrations. Cheers Harry -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Fri, 2014-10-31 at 09:13 +0100, Harald Staub wrote: On 30.10.2014 20:00, Ben Hutchings wrote: [...] Yes, I can see how this (disabling virtio features) would prevent live migration from old to new host kernels. You will probably need a one-time reboot of the guest when migrating to the new host kernel. We could either mention this in the DSA text, or keep UFO enabled even though it doesn't work correctly (in practice, we sort of have to do that as the tap device's feature flags aren't respected by guests - which I think is a QEMU bug). Please let us know whether live migration between two hosts running the new kernel version does work (I think it will). I confirm that. [...] Thanks. We're going to document the live migration issue rather than fixing it. Ben. -- Ben Hutchings The world is coming to an end. Please log off. signature.asc Description: This is a digitally signed message part
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
I can confirm that linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb fixes this regression for me. Cheers Harry -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On 30.10.2014 11:17, Harald Staub wrote: I can confirm that linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb fixes this regression for me. But it breaks live migration, probably when the source host runs an older kernel. When libvirt tries to complete the migration, it aborts with: error: Domain not found: no domain with matching name 'VMNAME' On the destination host, in /var/log/libvirt/qemu/VMNAME.log: kvm: Features 0x300fffe3 unsupported. Allowed features: 0x711fbbe3 qemu: warning: error while loading state for instance 0x0 of device ':00:03.0/virtio-net' load of migration failed I tried again with the older 3.2.60-1+deb7u3 on the destination host, this works fine. Cheers Harry -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Thu, 2014-10-30 at 17:28 +0100, Harald Staub wrote: On 30.10.2014 11:17, Harald Staub wrote: I can confirm that linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb fixes this regression for me. But it breaks live migration, probably when the source host runs an older kernel. When libvirt tries to complete the migration, it aborts with: error: Domain not found: no domain with matching name 'VMNAME' On the destination host, in /var/log/libvirt/qemu/VMNAME.log: kvm: Features 0x300fffe3 unsupported. Allowed features: 0x711fbbe3 qemu: warning: error while loading state for instance 0x0 of device ':00:03.0/virtio-net' load of migration failed I tried again with the older 3.2.60-1+deb7u3 on the destination host, this works fine. Yes, I can see how this (disabling virtio features) would prevent live migration from old to new host kernels. You will probably need a one-time reboot of the guest when migrating to the new host kernel. We could either mention this in the DSA text, or keep UFO enabled even though it doesn't work correctly (in practice, we sort of have to do that as the tap device's feature flags aren't respected by guests - which I think is a QEMU bug). Please let us know whether live migration between two hosts running the new kernel version does work (I think it will). Ben. -- Ben Hutchings The program is absolutely right; therefore, the computer must be wrong. signature.asc Description: This is a digitally signed message part
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Wed, 2014-10-22 at 01:02 +0100, Ben Hutchings wrote: I think this is the upstream change which would fix this. Please test this on top of the previous one, in case there are any more cases where ipv6_select_ident() may be called with a rt == NULL. However, it seems that with this change, when a VM offloads UDP/IPv6 fragmentation to us, we will always set the fragmentation ID to 0. I'm not sure whether that's a regression from 3.2.62, but I think it is. We should not be choosing fragment IDs for VMs, but currently they aren't telling us what to use! I've queried this upstream. I've now written and tested patches for that remaining regression. Source at: https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.dsc https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.debian.tar.xz Binary at: https://people.debian.org/~benh/packages/linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb There's a signed .changes file there as well, for verification (distribution=UNRELEASED in case you are wondering). Additional testing would be appreciated. Ben. -- Ben Hutchings Theory and practice are closer in theory than in practice. - John Levine, moderator of comp.compilers signature.asc Description: This is a digitally signed message part
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Mon, Oct 27, 2014 at 13:40:44 +, Ben Hutchings wrote: I've now written and tested patches for that remaining regression. Source at: https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.dsc https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.debian.tar.xz Binary at: https://people.debian.org/~benh/packages/linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb There's a signed .changes file there as well, for verification (distribution=UNRELEASED in case you are wondering). Additional testing would be appreciated. Installed your package on salieri.debian.org now. Anything we should look out for? Cheers, Julien signature.asc Description: Digital signature
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
On Mon, 2014-10-27 at 21:10 +0100, Julien Cristau wrote: On Mon, Oct 27, 2014 at 13:40:44 +, Ben Hutchings wrote: I've now written and tested patches for that remaining regression. Source at: https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.dsc https://people.debian.org/~benh/packages/linux_3.2.63-2+deb7u1.debian.tar.xz Binary at: https://people.debian.org/~benh/packages/linux-image-3.2.0-4-amd64_3.2.63-2+deb7u1_amd64.deb There's a signed .changes file there as well, for verification (distribution=UNRELEASED in case you are wondering). Additional testing would be appreciated. Installed your package on salieri.debian.org now. Anything we should look out for? 1. It shouldn't crash, obviously. 2. You may see a one-time warning about using UFO when we don't claim to support it, until you also upgrade the guest kernels. I found that libvirt+QEMU tells guests they can use UFO even if the host tap device refuses to support it. 3. UDP packets from the guest that require fragmentation should each get a different fragmentation ID, and if they are sent at a low rate, the IDs should not be consecutive (that was the purpose of the change in 3.2.63). You can check this by capturing and dumping IPv6 fragments on a *remote* machine: tcpdump -i eth0 -v 'ip6[6]==44' Example output: 21:33:12.57 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:0|1448) 32794 discard: UDP, length 10240 21:33:12.522271 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:1448|1448) 21:33:12.522282 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:2896|1448) 21:33:12.522293 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:4344|1448) 21:33:12.522301 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:5792|1448) 21:33:12.522312 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:7240|1448) 21:33:12.522322 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:8688|1448) 21:33:12.522334 IP6 (hlim 64, next-header Fragment (44) payload length: 120) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276214b:10136|112) 21:33:13.231373 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:0|1448) 59424 discard: UDP, length 10240 21:33:13.231414 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:1448|1448) 21:33:13.231424 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:2896|1448) 21:33:13.231432 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:4344|1448) 21:33:13.231441 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:5792|1448) 21:33:13.231450 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:7240|1448) 21:33:13.231460 IP6 (hlim 64, next-header Fragment (44) payload length: 1456) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:8688|1448) 21:33:13.231472 IP6 (hlim 64, next-header Fragment (44) payload length: 120) fe80::5604:a6ff:fec4:8ddd fe80::2a92:4aff:fe2f:b2d: frag (0x0276218f:10136|112) The first number after 'frag' is the fragmentation ID; here there are two sets of fragments from two original 10K packets. This should work whether or not the guest kernel is upgraded. Ben. -- Ben Hutchings Theory and practice are closer in theory than in practice. - John Levine, moderator of comp.compilers signature.asc Description: This is a digitally signed message part
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
Hi, On Wed Oct 22, 2014 at 01:02:07 +0100, Ben Hutchings wrote: I think this is the upstream change which would fix this. Please test this on top of the previous one, in case there are any more cases where ipv6_select_ident() may be called with a rt == NULL. However, it seems that with this change, when a VM offloads UDP/IPv6 fragmentation to us, we will always set the fragmentation ID to 0. I'm not sure whether that's a regression from 3.2.62, but I think it is. We should not be choosing fragment IDs for VMs, but currently they aren't telling us what to use! I've queried this upstream. i have build a new kernel with the above mentioned patches applied. This package is now available on https://people.debian.org/~zobel/linux-image-3.2.0-4-amd64_3.2.63-2a~test_amd64.deb Both machines that i installed the kernel on run for quite some time without any issue now. Before those patches they stayed up only for a few minutes. Cheers, Martin -- Martin Zobel-Helas zo...@debian.orgDebian System Administrator Debian GNU/Linux Developer Debian Listmaster http://about.me/zobel Debian Webmaster GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#766195: [PATCH] ipv6: reuse ip6_frag_id from ip6_ufo_append_data
I think this is the upstream change which would fix this. Please test this on top of the previous one, in case there are any more cases where ipv6_select_ident() may be called with a rt == NULL. However, it seems that with this change, when a VM offloads UDP/IPv6 fragmentation to us, we will always set the fragmentation ID to 0. I'm not sure whether that's a regression from 3.2.62, but I think it is. We should not be choosing fragment IDs for VMs, but currently they aren't telling us what to use! I've queried this upstream. Ben. -- Ben Hutchings For every action, there is an equal and opposite criticism. - Harrison From: Hannes Frederic Sowa han...@stressinduktion.org Date: Fri, 21 Feb 2014 02:55:35 +0100 Subject: ipv6: reuse ip6_frag_id from ip6_ufo_append_data Origin: https://git.kernel.org/linus/916e4cf46d0204806c062c8c6c4d1f633852c5b6 Currently we generate a new fragmentation id on UFO segmentation. It is pretty hairy to identify the correct net namespace and dst there. Especially tunnels use IFF_XMIT_DST_RELEASE and thus have no skb_dst available at all. This causes unreliable or very predictable ipv6 fragmentation id generation while segmentation. Luckily we already have pregenerated the ip6_frag_id in ip6_ufo_append_data and can use it here. Signed-off-by: Hannes Frederic Sowa han...@stressinduktion.org Signed-off-by: David S. Miller da...@davemloft.net [bwh: Backported to 3.2: adjust filename, indentation] --- net/ipv6/udp_offload.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1362,7 +1362,7 @@ static struct sk_buff *udp6_ufo_fragment fptr = (struct frag_hdr *)(skb_network_header(skb) + unfrag_ip6hlen); fptr-nexthdr = nexthdr; fptr-reserved = 0; - ipv6_select_ident(fptr, (struct rt6_info *)skb_dst(skb)); + fptr-identification = skb_shinfo(skb)-ip6_frag_id; /* Fragment the skb. ipv6 header and the remaining fields of the * fragment header are updated in ipv6_gso_segment() signature.asc Description: This is a digitally signed message part
