[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
** Also affects: linux-azure-edge (Ubuntu) Importance: Undecided Status: New ** Changed in: linux-azure-edge (Ubuntu) Assignee: (unassigned) => Marcelo Cerri (mhcerri) ** Changed in: linux-azure-edge (Ubuntu Xenial) Assignee: (unassigned) => Marcelo Cerri (mhcerri) ** Changed in: linux-azure-edge (Ubuntu) Status: New => Fix Committed ** Changed in: linux-azure-edge (Ubuntu Xenial) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
** Also affects: linux-azure (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
** Changed in: linux-azure (Ubuntu) Status: Confirmed => Fix Committed ** Changed in: linux-azure (Ubuntu Xenial) Status: New => Fix Committed ** Changed in: linux-azure (Ubuntu) Assignee: (unassigned) => Marcelo Cerri (mhcerri) ** Changed in: linux-azure (Ubuntu Xenial) Assignee: (unassigned) => Marcelo Cerri (mhcerri) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
https://lists.ubuntu.com/archives/kernel-team/2017-December/088827.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@mhcerri Thanks for the explanation! I built the kernels directly from the git repo and I didn't use the binary packages (I thought they should be the same). What's your repro-rate? Yesterday as Chris tested linux-azure-edge 4.13.0-1004.4 + my pull request, the repro rate was only ~1%, meaning we have to do "unload/reload netvsc" several hundred times to repro the hang or the calltrace. Previously without the pull request, Chris mentioned it was much easier to repro the issue. So it looks to me the bug is timing-sensitive, and somehow my pull request made it difficult to repro it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Ok. I've tested it again using gen1 VMs and I got the same results. Both linux-azure and linux-azure-edge show the problem, but for both cases I need to unload, load and unload the driver again to trigger the issue. I also tested the test kernel with the pull request and it shows the same problem. I will double check that build to make sure everything was correctly applied. I'm using Hyper-V on a Win10Pro machine. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@decui The repos are right. I'm asking because some of the kernel under http://kernel.ubuntu.com/~mhcerri/azure/ are testing kernels. linux-azure and linux-azure-edge version numbers are not equivalent. Thus, linux-azure 4.13.0-1001.1 is not equivalent to linux-azure-edge 4.13.0-1001.1. In fact, linux-azure 4.13.0-1001.1 should be equivalent to linux-azure-edge 4.13.0-1004.4. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@decui Are you building the kernels directly from the git repo or are you using binary packages from somewhere? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@mhcerri FYI: I'm not using any kernel from http://kernel.ubuntu.com/~mhcerri/azure/. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@mhcerri I'm using https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/, e.g. for 4.13.0-1004-azure-edge, I mean: https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/commit/?h=azure-edge-next=21d8a99f88af972684618521cf19adafe24dc566. Please let me know in case I'm using the incorrect repo? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@mhcerri I only tested Gen1 VM (4.13.0-1004-azure-edge) on my local Hyper-V host (WS 2016), and couldn't repro the issue, i.e. reloading hv_netvsc and changing MTU worked fine. As I understand, I think the bug was originally reported against 4.13.0-1001.1 (I did not test this version). That's why I think somehow the bug was fixed in 4.13.0-1004-azure-edge. Ubuntu-azure-edge-4.13.0-1001.1 is 567ef14ee13c5c4e336121106cc19733800d618e, and Ubuntu-azure-edge-4.13.0-1004.4 is 21d8a99f88af972684618521cf19adafe24dc566. There are non-trivial changes between them: root@decui-1604:/opt/linux-azure# git diff 567ef14ee13c5c4e336121106cc19733800d618e 21d8a99f88af972684618521cf19adafe24dc566 -- drivers/hv/ drivers/net/hyperv/| wc -l 2353 root@decui-1604:/opt/linux-azure# git diff 567ef14ee13c5c4e336121106cc19733800d618e 21d8a99f88af972684618521cf19adafe24dc566 -- drivers/hv/ drivers/net/hyperv/| grep ^diff diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c diff --git a/drivers/hv/hv_fcopy.c b/drivers/hv/hv_fcopy.c diff --git a/drivers/hv/hv_kvp.c b/drivers/hv/hv_kvp.c diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c root@decui-1604:/opt/linux-azure# git diff 567ef14ee13c5c4e336121106cc19733800d618e 21d8a99f88af972684618521cf19adafe24dc566 -- drivers/hv/ drivers/net/hyperv/| wc -l 2353 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
I couldn't properly verify the problem on azure. If I execute: sudo sh -c 'modprobe -r hv_netvsc; modprobe hv_netvsc' on an azure instance I lost SSH connection, but I don't see the hung task messages on the boot diagnostics windows. The problem is probably happening and maybe I'm not getting the messages. When I run on a local hyper-v gen2 VM, I can trigger the problem when I do the following sequence: modprobe -r hv_netvsc modprobe hv_netvsc modprobe -r hv_netvsc And I can reproduce the issue with linux-azure 4.13.0-1001.1, with linux-azure-edge 4.13.0-1004.4 and even with linux-azure 4.13.0-1001.1 with the pull request changes. I will re-test it using a gen1 VM. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Right now only Generation 1 VMs are used in Azure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Also, which specific builds of linux-azure and linux-azure-edge are you using? Are you using any versions from http://kernel.ubuntu.com/~mhcerri/azure/? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Dexuan, Are you using gen1 or gen2 VMs? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
This is indeed very odd. Examining both code bases, we don't see any significant difference. The source code is the same, as well as the configs and the module inclusion lists. We're checking if we can reproduce bug 1736283. @jpoulson probably knows the answer, but were both linux-azure and linux-azure-edge tested in the same manner? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Hrm, interesting. Thanks for the note, I'll circle around with the team here to see what the discrepancy might be. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
@leann Actually I'm not sure if the pull request in bug 1736283 can fix this bug. It looks 4.13.0-1004-azure-edge has already fixed this bug somehow. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
After having had a chat with Josh Poulson @Microsoft, it is believed that the pull request in bu https://bugs.launchpad.net/ubuntu/+source /linux-azure/+bug/1736283 should likely resolve the issue seen here. We will post a test kernel in bug 1736283. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
BTW, I tested "modprobe -r hv_netvsc; modprobe hv_netvsc" and "ip link set dev eth0 mtu 3000; ip link set dev eth0 mtu 4500" with 4.13.0-1004 -azure-edge on WS 2016 (Version 1607, OS build 14393:1943). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
I can't reproduce the issue with 4.13.0-1004-azure-edge (https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux- azure/commit/?h=azure-edge- next=21d8a99f88af972684618521cf19adafe24dc566) It looks this bug has been fixed by some patch between linux-azure 4.13.0-1001.1 and 4.13.0-1004-azure-edge. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-azure (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1735546] Re: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1
Adding this issue here, as in the past we saw linked issues between netvsc reload and mtu change. The same hung task failure occurs when changing the MTU on the interface. Repro steps: ip link set dev eth0 mtu 3000 ip link set dev eth0 mtu 4500 At this point the command will hang and then run into the hung messages after 120 seconds. ** Description changed: On proposed 4.13.0-1001.1 linux-azure kernel reloading the netvsc module will cause the VM network to stop working on WS2016. Manual repro is easy, start a VM with that kernel and run: modprobe -r hv_netvsc modprobe hv_netvsc At this point the command will hang and after 2 minutes the hung task messages will appear. Test env info: - ubuntu 16.04.3 - - Affected platforms: WS2016 and WS2016 fall update - 1709 + - Affected platforms: + a) WS2016 and WS2016 fall update - 1709 + b) WS2012R2 4.11 series linux-azure are not showing this behavior. - This seems to affect only WS2016, and it is functional on WS2012R2. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1735546 Title: [hyper-v] reloading netvsc issue on linux-azure 4.13.0-1001.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1735546/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
