[Bug 1909062] Re: qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP tx csum offload
Hi Matthew, We have verified the fix with proposed kernel. I hope that I have corrected the "tags" appropriately. Thanks, Manish ** Tags removed: verification-needed-groovy ** Tags added: verification-done-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP tx csum offload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
** Description changed: - Customer is reporting a problem with QL41xxx and Ubuntu internal DNS - server. The issue appeared when the customer updated to the latest - Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not - observed with 4.5 ubuntu-linux. + With QL41xxx and Ubuntu DNS server DNS failures are seen when updated to + the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was + not observed with 4.5 ubuntu-linux. Problem Definition: - - Product: PowerEdge R740xd - Serial: C7J90W2 - Hostname: xkubmin1r12 OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) - Firmware: 15.15.11 Inbox driver qede v8.37.0.20 - - Completed Detailed Problem Description: + Complete Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. - Customer is experiencing the same issue described in this article. + Similar issue is described in this article. https://github.com/kubernetes/kubernetes/issues/95365 - - Customer Impact: Production site - The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment. Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 - qede: fix offload for IPIP tunnel packets + qede: fix offload for IPIP tunnel packets - IPIP tunnels packets are unknown to device, - hence these packets are incorrectly parsed and - caused the packet corruption, so disable offlods - for such packets at run time. + IPIP tunnels packets are unknown to device, + hence these packets are incorrectly parsed and + caused the packet corruption, so disable offlods + for such packets at run time. - Signed-off-by: Manish Chopra - Signed-off-by: Sudarsana Kalluru - Signed-off-by: Igor Russkikh - Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com - Signed-off-by: Jakub Kicinski - + Signed-off-by: Manish Chopra + Signed-off-by: Sudarsana Kalluru + Signed-off-by: Igor Russkikh + Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com + Signed-off-by: Jakub Kicinski Thanks, Manish -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
We don't have any logs to post, attached wire traces which we got from customer as we were only interested in that. Having said that, this bug is resolved with the above fix posted upstream, I guess we don't need any further logs. ** Attachment added: "wire_traces.zip" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+attachment/5446099/+files/wire_traces.zip ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1909062] [NEW] Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
Public bug reported: Customer is reporting a problem with QL41xxx and Ubuntu internal DNS server. The issue appeared when the customer updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: Product: PowerEdge R740xd Serial: C7J90W2 Hostname: xkubmin1r12 OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Firmware: 15.15.11 Inbox driver qede v8.37.0.20 Completed Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Customer is experiencing the same issue described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Customer Impact: Production site The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment. Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish ** Affects: ubuntu Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+bug/1909062/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1667790] Re: PXE Installation Failure - Ubuntu14.04.5
Hi Brian, In case of non working interface we observe these logs - related to netcfg segfault. This is what we see main difference between working [where we don't see these failure logs] and non working interface. eb 24 06:31:49 dhclient: DHCPREQUEST of 192.168.10.91 on p4p1 to 255.255.255.255 port 67 (xid=0x419a4dac) Feb 24 06:31:49 dhclient: DHCPACK of 192.168.10.91 from 192.168.10.100 Feb 24 06:31:49 dhclient: bound to 192.168.10.91 -- renewal in 2494 seconds. Feb 24 06:31:51 netcfg[3724]: DEBUG: Reading domain name returned via DHCP Feb 24 06:31:51 netcfg[3724]: DEBUG: DHCP domain name is 'redhatguides.local' Feb 24 06:31:51 netcfg[3724]: DEBUG: Reading nameservers from /etc/resolv.conf Feb 24 06:31:51 netcfg[3724]: DEBUG: Read nameserver 192.168.10.1 Feb 24 06:31:51 netcfg[3724]: DEBUG: State is now 1 Feb 24 06:31:51 netcfg[3724]: DEBUG: State is now 2 Feb 24 06:31:51 netcfg[3724]: DEBUG: State is now 5 Feb 24 06:31:51 netcfg[3724]: INFO: DHCP hostname: "kickseed" Feb 24 06:31:51 netcfg[3724]: DEBUG: kickseed is a valid FQDN Feb 24 06:31:51 netcfg[3724]: DEBUG: Preseeding domain from global: redhatguides.local Feb 24 06:31:51 netcfg[3724]: DEBUG: State is now 6 Feb 24 06:31:51 netcfg[3724]: DEBUG: Network config complete Feb 24 06:31:51 netcfg[3724]: DEBUG: No interface given; clearing /etc/network/interfaces Feb 24 06:31:51 netcfg[3724]: DEBUG: Writing informative header Feb 24 06:31:51 netcfg[3724]: DEBUG: Success! Feb 24 06:31:51 netcfg[3724]: DEBUG: Writing loopback interface Feb 24 06:31:51 netcfg[3724]: DEBUG: Success! Feb 24 06:31:51 netcfg[3724]: DEBUG: Writing DHCP stanza for p4p1 Feb 24 06:31:51 netcfg[3724]: DEBUG: Writing wireless options for p4p1 Feb 24 06:31:51 netcfg[3724]: DEBUG: Success! Feb 24 06:31:52 main-menu[2545]: WARNING **: Configuring 'netcfg' failed with error code 139 Feb 24 06:31:52 main-menu[2545]: WARNING **: Menu item 'netcfg' failed. Feb 24 06:31:52 kernel: [ 62.915859] netcfg[3724]: segfault at 0 ip 7f1ab154e34f sp 7ffc781aec78 error 4 in libc.so.6[7f1ab14b1000+1ba000] but we don't know why this can be caused - Any idea how to debug this further for the cause of netcfg segfault ? Appreciate your help in regards to this. Thanks, Manish -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1667790 Title: PXE Installation Failure - Ubuntu14.04.5 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1667790/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
