On 10/20/25 11:28 AM, Ilya Maximets wrote: > On 10/20/25 10:56 AM, Frode Nordahl wrote: >> Hello all, >> >> On 10/20/25 10:17, Dumitru Ceara via dev wrote: >>> Hi Frode, >>> >>> Sorry for reaching out directly but I figured it might be easier for you >>> to report this to the Ubuntu kernel development team (if not, I can also >>> try to open a launchpad bug myself but my knowledge in that area is >>> limited). >>> >>> Our OVN CI (in GitHub actions) is broken since Friday, e.g.: >>> https://github.com/ovn-org/ovn/actions/runs/18622640352/job/53147265121 >>> >>> These tests fail: >>> 252: system-ovn-kmod.at:1006 Load Balancer LS hairpin IPv6 UDP - larger >>> than MTU -- parallelization=yes -- ovn_monitor_all=yes >>> lb >>> 253: system-ovn-kmod.at:1006 Load Balancer LS hairpin IPv6 UDP - larger >>> than MTU -- parallelization=yes -- ovn_monitor_all=no >>> lb >>> >>> They fail with: >>> >>> (cat datafile; sleep 3) | nc -6 -u 8800::0088 4040 -p 20000 -o >>> udp_frag_test_c1.recvd >>> NS_EXEC_HEREDOC >>> Ncat: Version 7.92 ( https://nmap.org/ncat ) >>> Ncat: Listening on 4200::1:2021 >>> stderr: >>> Ncat: Message too long. >>> stdout: >>> >>> As there were no OVN (or OVS) user space changes that could've caused >>> this (the last good run was on Thursday) we had a look at other >>> components that might have changed. >>> >>> It seems there was a GitHub ubuntu:24.04 runner image change that >>> happened since. The new version is: >>> >>> Image: ubuntu-24.04 >>> Version: 20251014.76.1 >>> Included Software: >>> https://github.com/actions/runner-images/blob/ubuntu24/20251014.76/images/ubuntu/Ubuntu2404-Readme.md >>> Image Release: >>> https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20251014.76 >>> >>> which uses kernel version 6.14.0-1012-azure. >>> >>> Our last known good CI runs were using kernel >>> version 6.11.0-1018-azure. >>> >>> I had a look at the linux-image-unsigned-6.14.0-1012-azure Ubuntu >>> kernel sources and it seems that we there might be a patch missing >>> there. I think we might be hitting the same issue as in: >>> >>> https://lore.kernel.org/stable/[email protected]/ >>> >>> Checking the unpacked Ubuntu kernel sources it seems the 6.11 kernel >>> didn't have the buggy patch: >>> a18dfa9925b9ef6107ea3aa5814ca3c704d34a8a "ipv6: save dontfrag in cork" >>> >>> While kernel 6.14.0-1012-azure includes the code from the buggy patch >>> but only has the first of the followup fixes: >>> - 54580ccdd8a9c6821fd6f72171d435480867e4c3 "ipv6: remove leftover ip6 >>> cookie initializer" >>> - 096208592b09c2f5fc0c1a174694efa41c04209d "ipv6: replace ipcm6_init calls >>> with ipcm6_init_sk" <<< the code doesn't have this commit. >>> >>> Would you happen to have some time to double check my findings and maybe >>> report this to the Ubuntu kernel team? >>> >>> Also, it seems ovn-kubernetes CI is affected by this too: >>> https://github.com/ovn-kubernetes/ovn-kubernetes/actions/runs/18638480933/job/53134392523#step:16:4399 >>> >>> [FAIL] [sig-network] Networking Granular Checks: Services [It] should be >>> able to handle large requests: udp [sig-network] >> >> Thank you for your detailed report, Dumitru. This appears to affect the >> regular generic kernel in Ubuntu 25.04 (Plucky Puffin) too [0]. >> >> The good news is that I don't see this in more recent kernels, so "all" we >> have to do is to find the missing commit ;) >> >> 0: >> https://autopkgtest.ubuntu.com/results/autopkgtest-plucky/plucky/amd64/o/ovn/20251002_082225_66b13@/log.gz >> > > AFAICT, 6.14.0-1014-azure should have the missing commit: > "ipv6: replace ipcm6_init calls with ipcm6_init_sk" > > I opened this ticket for GHA for now: > https://github.com/actions/runner-images/issues/13190 > > Best regards, Ilya Maximets. >
Hi Frode, Ilya, Thanks for the quick replies! I guess indeed all we have to do is wait until the newer kernel is picked up by the GHA runners. Thanks, Dumitru _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
