Hi Ilya! Thanks for your answer) I spent some more time on this issue, and my assumption from the previous email turned out to be incorrect.
You asked for a link to the build - here it is. This is exactly two red tests I'm talking about: https://github.com/Sashhkaa/ovn/actions/runs/17321708600 Here is the current picture I have. I have two Ubuntu 22.04 machines: * Completely identical build flags for OVS and OVN * Identical versions of the compiler (gcc), assembler, and linker (libc version 2.35-0ubuntu3.8, gcc 4:11.2.0-1ubuntu1) * Absolutely identical versions of all userspace packages * Kernel and its configs are identical (5.15.0-113-generic) * Accordingly, identical kernel modules * Just in case, I made all sysctl parameters identical The only thing that differs is the architecture - the first virtual machine has 2 CPU cores, and the second has only 1 CPU core. This is the only difference I found. On the machine where the test fails, there is 1 core; on the other with two cores, the test passes. Honestly, I have absolutely no ideas. I also built OVS and OVN on Ubuntu 24.04 using the same libc version that I had on Ubuntu 22 - I did not observe the same behavior there. So most likely, I am making a mistake somewhere in my current investigation. I would be glad if you have time to look into what the issue might be. Or maybe you have ideas on where else to look for differences between the builds on these two machines? Thanks a lot ! On 27.08.2025 23:40, Ilya Maximets wrote: On 8/27/25 10:24 PM, Ilya Maximets wrote: On 8/25/25 10:47 AM, Rukomoinikova Aleksandra wrote: Ilya, hi! Last time I didnt figured out the issue I wrote about in this thread, but I really want to put end to it. I will remind you what I wrote in this thread: I applied my patches to version 24.03, these patches shift table numbers in northd. After that, ovn userspace test started failing (79-82 userspace tests (ECMP symmetric reply). The test failed because it expects to see the conntrack state ct_state(+new-est-rpl+trk), but after my patches, I see ct_state(+new-rpl+trk) in the datapath flow. I understand why the userspace test was failing - you said it was all due to the classifier's behavior. I figured it out back then and it all made sense. I changed the test to expect ct_state(+new-rpl+trk). But after that, the test started failing in the kernelspace case. In your message above, you said it was likely due to supported features, but I, firstly, compared the supported features in the controller logs, and secondly, in OpenFlow. They are identical. I did some unsuccessful investigation: I set up several machines with different distributions. Initially, I expected that the test failure might depend on the kernel version. I tried different kernels and ended up with the following result: The test only fails on Ubuntu 24 - the same is used in ci tests on github (which has a 6+ kernel installed). With the same kernel version, the test does not fail on CentOS 9, Fedora, or Ubuntu 22.I checked that the issue is not related to the GCC version. I compared the kernel configs and didn't see anything significant there. I also compared the kernel module support. I understand that the issue is likely environmental, possibly related to the version of some libraries or system packages. But I'm completely out of ideas on where to look next. Maybe you could suggest where else I might investigate to get to the bottom of this test failure? This is very strange... Usually I would suggest to look at hashing. e.g. if you somehow end up with multiple rules of the same priority and a random one gets chosen. But it's hard to tell why this can be if you're using the same compiler flags. Are you using -march=native or something similar by any chance? Did you try building with -O0 I meant -march=x86-64, but -O0 may also be worth trying just in case. to avoid any potential instruction set differences? It's hard to tell what else can it be. Do you have a link to the code that you're running? Maybe I can find some time to try it and see if I can reproduce this behavior. Best regards, Ilya Maximets. On 18.06.2025 21:41, Ilya Maximets wrote: On 5/26/25 3:45 PM, Rukomoinikova Aleksandra wrote: Ilya, hi! Thank you for your help and sorry for the long reply. Generally, I figured out the classifier code and understood what you meant, but I noticed that when testing with a kernel module, there's a difference in the way the classifier works, or something similar. In the test I was having trouble with, the expected conntrack state is: +new-est-rpl+trk. After my changes, it turns out to be: +new-rpl+trk. I corrected the test to expect +new-rpl+trk, but in tests with the OVS kernel datapath(check kernel), the old сonntrack state is expected: +new-est-rpl+trk. I couldn't figure out the reason for this behavior, can you suggest where it would be better to look and what it could be connected with? Thanks! Sorry, lost track of this discussion for some time. If you run the same test with 'make check-kernel' and the 'make check-system-userspace', do you see different conntrack states? There might be some slight difference in what kind of OpenFlow rules ovn-controller generates depending on the OVS datapath type (features supported). On 19.05.2025 23:14, Ilya Maximets wrote: On 5/16/25 11:16 AM, Rukomoinikova Aleksandra wrote: Hi! We've encountered a strange issue while backporting patches to the version 24.03 branch (ovs v3.3.4) and running tests. Let me describe the situation: I took the upstream branch 24.03, added a stage at the beginning of the switch pipeline, and added a 'match all' flow with 'next;' action. Commit example: https://github.com/Sashhkaa/ovn/commit/f20295315c327addfeb6fe455c3b3c655d6b3666. After this change, OVN 79-82 userspace tests (ECMP symmetric reply) started failing. According to the test logs, I see the following: The test expects to see the conntrack state ct_state(+new-est-rpl+trk) in the datapath flow, but gets ct_state(+new-rpl+trk) - that is, -est disappears. I will also attach more detailed dumps below. In general, the extra -est match is harmless and doesn't affect correctness, because +new traffic is always -est. And +est traffic is always -new. So, I think, you may just update the test in your internal backport and call it a day. For the actual reason why this is happening, the answer is: OpenFlow table sharing between the switch and the router pipelines. Both the router and the switch pipelines have their OpenFlow rules in the exact same OpenFlow tables starting from table 8. This means that on 24.03 the ls_in_acl_action and the lr_in_ecmp_stateful stages are using the same OpenFlow table 17. When you add one stage to the switch pipeline, you shift all switch tables by one while keeping router pipelines in place. So, now lr_in_ecmp_stateful shares the table with ls_in_acl_eval instead. All the rules have a match on metadata fields that distinguishes switches from routers and so there are no issues with correctness caused by sharing. However, the classifier may add extra matches due to internal implementation details. Classifier will traverse all the rules in the OpenFlow table starting with the highest priority. If there are no rules that match the packet in the current priority, classifier adds a minimal match to the datapath flow that will distinguish this packet from any OpenFlow rule in this table at this priority. So, if one of the rules with the higher priority had +est in the match, classifier will add -est to the datapath flow for the packet that didn't match that flow. So, by adding an extra stage to the router pipeline, you're just restoring the mapping of switch and router pipelines to OpenFlow tables like it was before the backport. By playing with ACL priorities, you're making the classifier go to the next table before evaluating a lower priority rule that has an extra +est match. We worked on one similar issue recently: https://patchwork.ozlabs.org/project/ovn/patch/[email protected]/ Here we had a -dnat match leak from the router pipeline to the switch pipeline for the packet that does not even go through the router. And that breaks hardware offload because neither kernel nor hardware NICs support offloading of NAT flags. Leaking of match criteria between switch and router pipelines is an interesting side effect of OVN design, but should not generally cause issues, except for hardware offloading in some cases. Best regards, Ilya Maximets. The expected state should be set by matching this OpenFlow rule in table 17 (in OVN it is router pipeline table 9 - ECMP stateful): cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=6, n_bytes=636, idle_age=1, priority=100,ct_state=+new-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=14, n_bytes=1396, idle_age=0, priority=100,ct_state=+est-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) I found two logical flow changes, that work, though it's not clear why: 1) Adding a router table before ECMP processing: By inserting just one table at the very beginning of the router pipeline, before the ECMP stateful handling (for example, https://github.com/odivlad/ovn/commit/eb6d0d7409ff78f1fc0908a28225d0a2a47daa29 one table is enough), the test starts passing. The mechanism isn't clear - packets now match the default flow in table 17 and only hit the proper ECMP rule in table 18, yet this somehow resolves the issue. 2) Modifying ACL evaluation rules: The second solution is even more strange. Since this test case doesn't use ACLs or load balancers, northd adds match all' flow with 'next;' action and priority 65535 to the acl_eval table (logical table 9 in switch, OpenFlow table 17). When we lower the priority of these rules below 100(less priority for the ecmp rules), the test begins working. This suggests some hidden interaction between router and switch pipeline rules, despite their different metadata matching criteria. When examining the OVS traces for both cases - the initial failed test with just a stage addition versus the working version where we also modified the ACL eval table priority to 0 - the packet's path through the tables shows no differences except for two key aspects: first, the rule matching in ACL eval (OpenFlow table 17), and second, the resulting datapath action where the -est state unexpectedly disappears. The trace comparison reveals that only the rule priorities in table 17 actually changed, yet this somehow impacts the connection tracking state. You can see the complete trace comparison showing both scenarios - with just the stage addition and with the priority modification - along with the contents of table 17 and the diff between traces at this link: https://gist.github.com/Sashhkaa/58b2c616e7d46fc2dafb898ed832960f. I've verified this behavior persists in newer versions of Open vSwitch as well. Does anyone understand what could be causing this issue? I'd appreciate any insights or suggestions for a proper fix. Thank you! -- regards, Alexandra. _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
