[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Expired Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
Joseph, thanks, we can talk to the customer about options to upgrade kernels. Unfortunately, we can't use ksplice as this kernel is not seen as LTS in that product. Again, if there is any specific fix we think is likely to help, that would be good to know about before trying to arrange downtime. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Incomplete Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
The test requested in comment #3 was only to see if the bug is fixes in the current mainline kernel. If this is a production machine, we would not want to put a development kernel on it. The current linux-lts-wily kernel is a few versions ahead of what you posted in comment #4. It might be good to apply the latest updates to see if the latest hwe kernel exhibits the bug. The current linux-ltx- wily kernel is as follows: linux-lts-wily | 4.2.0-27.32~14.04.1 | trusty-security | source linux-lts-wily | 4.2.0-27.32~14.04.1 | trusty-updates | source linux-lts-wily | 4.2.0-29.34~14.04.1 | trusty-proposed | source -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Incomplete Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
Sorry. no, this didn't happen after an upgrade. These machines are not using ksplice, so we have no auto-updating, and the kernel dates from the last full redeployment: Linux 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux These are trusty nodes using the above kernel to avoid bug 1497812. This is a production customer cloud, so we don't have scope to upgrade a machine just for testing, unfortunately. Is there a likely fix in the upgrade you recommend? As I say, so far we have not seen the issue since setting TSO to off. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Incomplete Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.5 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. Once testing of the upstream kernel is complete, please mark this bug as "Confirmed". Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-rc3-wily/ ** Tags added: kernel-da-key ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Incomplete Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
** Changed in: linux (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Confirmed Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
This is a cloud of customer machines. How intrusive is apport-collect, what load will it add to the machines, and what customer information will it collect and send, please? ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: Confirmed Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1540405] Re: i40e fails virtual networking
** Package changed: linux-meta (Ubuntu) => linux (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1540405 Title: i40e fails virtual networking Status in linux package in Ubuntu: New Bug description: With linux-hwe-generic-trusty we see kern.log entries for "TX driver issue detected, PF reset issued" on a NIC in a bond, followed by the same interface going down and up again. After this time networking to virtual machines bridged to that bond fails. We see either asymmetric traffic, with ARP replies not reaching the VMs, or no traffic at all to or from VMs. In this context, these are nova-compute nodes using OpenvSwitch and KVM. Reloading the i40e module corrects the networking. Setting TCP Segmentation Offload off on the NICs seems to prevent the PF reset, and we don't believe we have seen VM networking problems since doing so. # ethtool -i eth5 driver: i40e version: 1.3.4-k firmware-version: f4.33.31377 a1.2 n4.41 e1866 bus-info: :04:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1540405/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
