** Description changed: - If you have two hvs where hv1 is the gateway chassis and you have an - instance running on hv2. + [Impact] + Users of Open vSwitch on Focal will not be able to upgrade to v2.16.0 or newer until this long standing kernel bug has been fixed. + + Users of Open vSwitch on Jammy will be affected by this bug and + have no user space fix available. This bug currently blocks the + OpenStack Engineering team's charm product gate. + + [Test Plan] + Execute the OVN system testsuite utilizing the kernel data path with the test synthesis patch in comment #7 applied. + + In addition to that validating that the OpenStack charm test gate is + unblocked would be valuable. + + [Regression Potential] + The regression potential can be considered as low because: + - The calls added in the openvswitch kernel datapath code would + prior to Open vSwitch 2.16.0 have been initiated from the + userspace code and by chance concealed this bug. + - After an optimization done in 2.16.0 the kernel bug was + revealed and these calls now must be made from the kernel + datapath to retain functionality in use in the wild. + + [Original Bug Description] + If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2. On instance on hv2 hairpin traffic works for the first session, but not for the next: $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7 - $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. --- 10.78.95.89 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7 We made an attempt at using OVN built with [0], but that did unfortunately not help. If we however revert [1] it works again: $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1 $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1 - 0: https://patchwork.ozlabs.org/project/ovn/patch/20220401175516.2139179-1-mmich...@redhat.com/ 1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a - --- + --- ProblemType: Bug AlsaDevices: - total 0 - crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq - crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer + total 0 + crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq + crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 22.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: - Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub - Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub + Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Lsusb-t: - /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M - /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M + /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M + /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M MachineType: QEMU Standard PC (Q35 + ICH9, 2009) Package: linux (not installed) PciMultimedia: - + ProcEnviron: - TERM=xterm-256color - PATH=(custom, no user) - LANG=C.UTF-8 - SHELL=/bin/bash + TERM=xterm-256color + PATH=(custom, no user) + LANG=C.UTF-8 + SHELL=/bin/bash ProcFB: 0 virtio_gpudrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35 RelatedPackageVersions: - linux-restricted-modules-5.15.0-37-generic N/A - linux-backports-modules-5.15.0-37-generic N/A - linux-firmware 20220329.git681281e4-0ubuntu1 + linux-restricted-modules-5.15.0-37-generic N/A + linux-backports-modules-5.15.0-37-generic N/A + linux-firmware 20220329.git681281e4-0ubuntu1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: jammy uec-images Uname: Linux 5.15.0-37-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 02/06/2015 dmi.bios.release: 0.0 dmi.bios.vendor: EFI Development Kit II / OVMF dmi.bios.version: 0.0.0 dmi.board.name: LXD dmi.board.vendor: Canonical Ltd. dmi.board.version: pc-q35-7.0 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-q35-7.0 dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku: dmi.product.name: Standard PC (Q35 + ICH9, 2009) dmi.product.version: pc-q35-7.0 dmi.sys.vendor: QEMU
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw Status in linux package in Ubuntu: Invalid Status in openvswitch package in Ubuntu: Invalid Status in ovn package in Ubuntu: Invalid Status in linux source package in Focal: Fix Committed Status in openvswitch source package in Focal: Invalid Status in ovn source package in Focal: Invalid Status in linux source package in Impish: Won't Fix Status in openvswitch source package in Impish: New Status in ovn source package in Impish: New Status in linux source package in Jammy: Fix Committed Status in openvswitch source package in Jammy: Invalid Status in ovn source package in Jammy: Invalid Status in linux source package in Kinetic: Invalid Status in openvswitch source package in Kinetic: Invalid Status in ovn source package in Kinetic: Invalid Bug description: [Impact] Users of Open vSwitch on Focal will not be able to upgrade to v2.16.0 or newer until this long standing kernel bug has been fixed. Users of Open vSwitch on Jammy will be affected by this bug and have no user space fix available. This bug currently blocks the OpenStack Engineering team's charm product gate. [Test Plan] Execute the OVN system testsuite utilizing the kernel data path with the test synthesis patch in comment #7 applied. In addition to that validating that the OpenStack charm test gate is unblocked would be valuable. [Regression Potential] The regression potential can be considered as low because: - The calls added in the openvswitch kernel datapath code would prior to Open vSwitch 2.16.0 have been initiated from the userspace code and by chance concealed this bug. - After an optimization done in 2.16.0 the kernel bug was revealed and these calls now must be made from the kernel datapath to retain functionality in use in the wild. [Original Bug Description] If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2. On instance on hv2 hairpin traffic works for the first session, but not for the next: $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7 $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. --- 10.78.95.89 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7 We made an attempt at using OVN built with [0], but that did unfortunately not help. If we however revert [1] it works again: $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1 $ ping -c1 10.78.95.89 PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data. 64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms --- 10.78.95.89 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms $ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7 icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7 icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1 0: https://patchwork.ozlabs.org/project/ovn/patch/20220401175516.2139179-1-mmich...@redhat.com/ 1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 22.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Lsusb-t: /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M MachineType: QEMU Standard PC (Q35 + ICH9, 2009) Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: 0 virtio_gpudrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35 RelatedPackageVersions: linux-restricted-modules-5.15.0-37-generic N/A linux-backports-modules-5.15.0-37-generic N/A linux-firmware 20220329.git681281e4-0ubuntu1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: jammy uec-images Uname: Linux 5.15.0-37-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 02/06/2015 dmi.bios.release: 0.0 dmi.bios.vendor: EFI Development Kit II / OVMF dmi.bios.version: 0.0.0 dmi.board.name: LXD dmi.board.vendor: Canonical Ltd. dmi.board.version: pc-q35-7.0 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-q35-7.0 dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku: dmi.product.name: Standard PC (Q35 + ICH9, 2009) dmi.product.version: pc-q35-7.0 dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1967856/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
