On 11/11/25 6:07 PM, Numan Siddique wrote: > On Mon, Nov 3, 2025 at 2:12 PM Numan Siddique <[email protected]> wrote: >> >> On Mon, Nov 3, 2025 at 2:02 PM Ilya Maximets <[email protected]> wrote: >>> >>> On 11/3/25 7:29 PM, Numan Siddique wrote: >>>> On Mon, Nov 3, 2025 at 5:55 AM Ilya Maximets <[email protected]> wrote: >>>>> >>>>> On 11/1/25 7:23 AM, Numan Siddique wrote: >>>>>> Hello OVS folks, >>>>>> >>>>>> In our deployments we are seeing a lot of datapath flow offload issues >>>>>> with tc resulting in packets getting handled in the host and packet >>>>>> drops. >>>>>> >>>>>> We recently observed such an issue and only restart of ovs-vswitchd >>>>>> fixed it. >>>>>> >>>>>> I debugged a bit and found that all the datapath flows offloaded by >>>>>> ovs-vswitchd to tc fails if the recirculation id is greater than >>>>>> 268,435,455 (which is 0x0fffffff). >>>>>> >>>>>> We see the below error messages: >>>>>> >>>>>> -------------------------------------------------- >>>>>> 2025-11-01T03:12:18.415Z|93221|netlink_socket(handler53)|DBG|nl_sock_recv__ >>>>>> (Success): nl(len:692, type=2(error), flags=200[MATCH], seq=7af, >>>>>> pid=3613179965 error(-22(Invalid argument), in-reply-to(nl(len:624, >>>>>> type=44(family-defined), flags=409[REQUEST][ECHO][ATOMIC], seq=7af, >>>>>> pid=3613179965)) >>>>>> 2025-11-01T03:12:18.415Z|93222|netlink_socket(handler53)|DBG|received >>>>>> NAK error=22 - Specified chain index exceeds upper limit >>>>>> 2025-11-01T03:12:18.415Z|93223|dpif_netlink(handler53)|ERR|failed to >>>>>> offload flow: Invalid argument: ovn-f3902a-0 >>>>>> 2025-11-01T03:12:18.415Z|93224|dpif_netlink(handler53)|DBG|system@ovs-system: >>>>>> put[create] ufid:e287c507-e111-44be-90dd-469c242cb873 >>>>>> recirc_id(0x2660dc6d),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0x915,src=10.32.35.9,dst=10.32.5.25,ttl=59/0,tp_src=34744/0,tp_dst=6081/0,geneve({class=0x102/0,type=0x80/0,len=4/0,0x79a041a/0}),flags(-df+csum+key)),in_port(6),skb_mark(0/0),ct_state(0x21/0x3f),ct_zone(0x17f5/0),ct_mark(0/0x1),ct_label(0/0),ct_tuple4(src=172.27.61.139/0.0.0.0,dst=172.27.58.113/0.0.0.0,proto=6/0,tp_src=49588/0,tp_dst=4240/0),eth(src=be:28:87:5d:2e:28,dst=fe:6c:ee:aa:33:be),eth_type(0x0800),ipv4(src=172.27.61.139,dst=172.27.58.113,proto=6,tos=0/0,ttl=64/0,frag=no),tcp(src=49588/0x8000,dst=4240/0xf800),tcp_flags(0/0), >>>>>> actions:ct(commit,zone=6133,mark=0/0x1,nat(src)),20 >>>>>> ------------------------------------------------------------------- >>>>>> >>>>>> I was able to reproduce the issue locally with OVS main and Fedora >>>>>> kernel 6.16.10-200.fc42. I had to hack the code though. >>>>>> >>>>>> ---- >>>>>> diff --git a/ofproto/ofproto-dpif-rid.c b/ofproto/ofproto-dpif-rid.c >>>>>> index f01468025..1d577d73b 100644 >>>>>> --- a/ofproto/ofproto-dpif-rid.c >>>>>> +++ b/ofproto/ofproto-dpif-rid.c >>>>>> @@ -34,8 +34,7 @@ static struct ovs_list expiring OVS_GUARDED_BY(mutex) >>>>>> static struct ovs_list expired OVS_GUARDED_BY(mutex) >>>>>> = OVS_LIST_INITIALIZER(&expired); >>>>>> >>>>>> -static uint32_t next_id OVS_GUARDED_BY(mutex) = 1; /* Possible next >>>>>> free id. */ >>>>>> - >>>>>> +static uint32_t next_id OVS_GUARDED_BY(mutex) = 0x0fffffff; /* >>>>>> Possible next free id. */ >>>>>> #define RECIRC_POOL_STATIC_IDS 1024 >>>>>> >>>>>> static void recirc_id_node_free(struct recirc_id_node *); >>>>>> ----- >>>>>> >>>>>> Looks like kernel expects the tc flower chain id to be encoded with in >>>>>> the first 28 bits [1], where as ovs-vswitchd is using the value of >>>>>> recirc_id as chain id and if the recirc_id overflows 28 bits, the >>>>>> issue is seen. >>>>>> >>>>>> Is my analysis correct ? I'm not too familiar with the classifier and >>>>>> the offload code base. Hope the experts can take a look at it. >>>>> >>>>> Hi, Numan. Yes, your analysis seems correct. The GOTO_CHAIN action >>>>> is an "extended action", where first 4 bits are reserved for the action >>>>> type and the rest are a value: >>>>> >>>>> https://elixir.bootlin.com/linux/v6.17.6/source/tools/include/uapi/linux/pkt_cls.h#L50-L64 >>>>> >>>>> This means, we can't offload recirculations to chains above 28 bits. >>>>> >>>>> There are two things here that need fixing: >>>>> >>>>> 1. OVS doesn't seem to check that chain id fits into the action, blindly >>>>> ORing it in. That should be fixed, so we are not trying to send such >>>>> flows into kernel in the first place. >>>>> >>>>> 2. Somehow limit the recircualtion id space to 28 bits when the HW >>>>> offload is enabled. I don't like this, as we'll be just adding yet >>>>> another hack for HW offload to work, but I'm not sure what would be >>>>> a different solution here. Note: id-pool would solve the problem >>>>> by allocating densely packed IDs, but that may cause collisions as >>>>> the whole process of retiring old IDs is a bit racy and we rely on >>>>> time to guess when we can actually stop using them. Needs more >>>>> investigation. >>>>> >>>>> Best regards, Ilya Maximets. > > I'll take a look into the fix based on your suggestions. If you or > someone has already started looking into me, please let me know.
Feel free to work on these. The first one should be relatively straightforfard. The second one is a little complicated, as a straightforward solution feels a bit icky, though I'm not sure how to avoid that. Best regards, Ilya Maximets. > > Numan > >>>> >>>> >>>> Thanks for the reply, Ilya. >>>> >>>> In one of our deployment which uses OVS 3.2.0, we see the below logs >>>> and packet drops to the VM, >>>> >>>> --- >>>> >>>> 2025-10-30T04:26:28.474Z|78613|tc(handler25)|WARN|Kernel flower >>>> acknowledgment does not match request! Set dpif_netlink to dbg to see >>>> which rule caused this error. >>>> >>>> 2025-10-30T04:26:29.113Z|78614|tc(handler25)|WARN|Kernel flower >>>> acknowledgment does not match request! Set dpif_netlink to dbg to see >>>> which rule caused this error. >>>> ------ >>>> >>>> Any pointers on why we are seeing the above WARN message ? OVS 3.2.0 >>>> is missing the below backport - >>>> https://github.com/openvswitch/ovs/commit/1857c569ee9a6432ac46d31a31f882402c215437 >>>> Could it be because of this ? >>> >>> It's likely. To confirm you'll need to enbale debug logs for the tc module, >>> may also enbale dbg for the dpif_netlink, as the logs suggest. >>> >>>> Do we need to move to 3.2.2 at least for successful offloads ? >>> >>> If the issue above is indeed your issue, then update will remove the >>> warning. >>> However, this flow will not be offloaded, as it requires modification of the >>> tp_src of the outer tunnel header which TC doesn't support. Such flows >>> should >>> not be common though, so I'm not sure if you actually need them offloaded. >>> It depends on the setup. But also, you need to confirm that it is your >>> issue >>> first. >> >> Got it. Thanks. >> >>> >>> FWIW, 3.2.0 is very old and is missing a lot of fixes, so I'd suggest >>> updating >>> it anyway. Also, 3.2 is EoL, so going to at least 3.3 is recommended. >> >> Ack. >> >> >>> >>>> The kernel version is - 5.14.0-162.6.1.el9 >>>> >>>> >>>> In the below datapath flow dump, we see that there is a flow for the >>>> first packet and the final action of this dp flow is - >>>> recirc(0x94691). >>>> >>>> ------------ >>>> recirc_id(0),in_port(18),ct_state(-new-est-rpl-trk),ct_mark(0/0x2),eth(src=b0:cf:0e:b1:5f:ff,dst=5e:8e:4a:f0:44:25),eth_type(0x8100),vlan(vid=120,pcp=0),encap(eth_type(0x0800),ipv4(src=192.0.0.0/224.0.0.0,dst=160.211.64.157,proto=1,ttl=47,frag=no)), >>>> packets:2217, bytes:186228, used:0.330s, >>>> actions:pop_vlan,ct(zone=24,nat),recirc(0x94691) >>>> recirc_id(0),in_port(18),ct_state(-new-est-rpl-trk),ct_mark(0/0x2),eth(src=b0:cf:0e:b1:5f:ff,dst=5e:8e:4a:f0:44:25),eth_type(0x8100),vlan(vid=120,pcp=0),encap(eth_type(0x0800),ipv4(src=96.0.0.0/252.0.0.0,dst=160.211.64.157,proto=1,ttl=44,frag=no)), >>>> packets:1005, bytes:84420, used:0.850s, >>>> actions:pop_vlan,ct(zone=24,nat),recirc(0x94691) >>>> ---------- >>>> >>>> But in the dp flows, we never found a flow with recirc_id(0x94691). >>>> After a few minutes, we took the dump of dp flows and we noticed that >>>> there was a flow matching recirc(0x94691), but it was totally >>>> unrelated to the packet in question. >>>> >>>> >>>> We also saw the below message in the ovs logs. >>>> >>>> ----------- >>>> 2025-10-30T02:54:42.331Z|41074|ofproto_dpif_upcall(handler25)|INFO|received >>>> packet on unassociated datapath port 18 (no recirculation data for >>>> recirc_id 0x94691) >>>> >>>> 2025-10-30T03:15:42.380Z|43176|ofproto_dpif_upcall(handler25)|INFO|received >>>> packet on unassociated datapath port 18 (no recirculation data for >>>> recirc_id 0x94691) >>>> ---------- >>>> >>>> IMO the packet drops were due to the missing dp flow for the recirc_id >>>> 0x94691. >>>> >>>> Do you have any pointers on what could be going wrong ? >>> >>> Was OVS recently restarted when this was observed? >> >> No it was not restarted. >> >> That may explain the missing >>> records for the recirculation ID in userspace. But otherwise it's hard to >>> guess >>> what could've gone wrong here. >> >> Got it. Its hard to guess as there are a lot of other factors. >> >>> >>> Since you also have the recirc_id overflow issue, it might be possible that >>> the >>> ID got truncated somehwere and hence it's incorrect. >>> >> >> I see. I'll try to dig further. >> >> Numan >> >>>> >>>> Thanks for your time >>>> >>>> Numan >>>> >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> [1] - >>>>>> https://github.com/torvalds/linux/blob/v6.18-rc3/net/sched/cls_api.c#L3137 >>>>>> >>>>>> Thanks >>>>>> Numan >>>>> >>> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
