[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
@Dexuan, the error 'WARNING ExtHandler CGroup walinuxagent.service: Crossed the Memory Threshold' started when the vm was first created on May 24, persisted until I stopped/dealocated the VM, and has not returned in syslog. Nor for what it's worth has this error been seen in over six months on another Ubuntu 18.04 VM we have. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
I rebooted our VM over 60 times today without any further issues. Since the VM network went down before I restarted the VM or lost access to the serial console login, I'm also thinking now it was a network issue to begin with. It could always be coincidence that I was able to get the VM back by force stopping and then starting the vm from a stopped state, when neither restarting nor redeploying helped, but there is one thing, correct me if this is wrong, that is different about force stopping and that is it deallocates the VM. If the network issue were internal to the VM I wonder if both restarting and redeploying could have with the best of intentions persisted the faulty network state to the point where the operations failed, while dealocating ended the issue and the VM was able to start over clean. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
I see now 5.0.0-1007.7 is also available for 19.04, but I'll wait and see what you advise @Dexuan after looking at my serial log. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
I got the VM back by force stopping, then starting. It looks like 5.0.0-1007.7 only comes with 19.10, which is not available yet on Azure? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
How do I upgrade the kernel to 5.0.0-1007.7 if that is what is recommended to avoid this issue? apt-get dist-upgrade on another vm with 4.18.0-1018-azure did not upgrade the kernel. Thank God it restarted! Or is 5.0.0-1007.7 not an Azure kernel so I would need to switch to stock Ubuntu kernel? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
@Dexuan I emailed you my serial log. I restarted via portal a number of times without success. I tried to redeploy twice and it failed. Trying to restart via cli now. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
kernel is 4.18.0-1018-azure -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1822133] Re: Azure Instance never recovered during series of instance reboots.
This bit me tonight on a few days old 18.04 VM with stock kernel. So far I can not get back in with reboot or redeploy to upgrade the kernel. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1822133 Title: Azure Instance never recovered during series of instance reboots. Status in linux-azure package in Ubuntu: In Progress Bug description: Description: During SRU Testing of various Azure Instances, there will be some cases where the instance will not respond following a system reboot. SRU Testing only restarts a giving instance once, after it preps all of the necessary files to-be-tested. Series: Disco Instance Size: Basic_A3 Region: (Default) US-WEST-2 Kernel Version: 4.18.0-1013-azure #13-Ubuntu SMP Thu Feb 28 22:54:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I initiated a series of tests which rebooted Azure Cloud instances 50 times. During the 49th Reboot, an Instance failed to return from a reboot.. Upon grabbing the console output the following was seen scrolling endlessly. I have seen this failure in cases where the instance only restarted a handful of times >5 [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus [84.247704]hyperv_fb: unable to send packet via vmbus In another test attempt I saw the following failure: ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes ERROR ExtHandler /proc/net/route contains no routes Both of these failures broke networking, Both of these failures were seen at least twice to three times, thus may explain why in some cases we never recover from an instance reboot. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1822133/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
