On 25 Feb 2022, at 15:09, Ilya Maximets wrote:
> On 2/24/22 20:07, Frode Nordahl wrote: >> On Fri, Apr 9, 2021 at 10:09 PM Ilya Maximets <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> I’m running ovn 20.06.2 with ovs 2.13.0. On some nodes in ovs-vswitchd.log >>>> I see lots of warnings like this: >>>> >>>> 2021-03-09T05:05:57.349Z|00062|dpif(handler3)|WARN|system at ovs-system: >>>> execute ct(commit,zone=58,label=0/0x1),33 failed (Invalid argument) on >>>> packet >>>> icmp,vlan_tci=0x0000,dl_src=<MAC1>,dl_dst=<MAC2>,nw_src=<IP1>,nw_dst=<IP2>,nw_tos=0,nw >>>> _ecn=0,nw_ttl=52,icmp_type=8,icmp_code=0 icmp_csum:e827 >>>> with metadata >>>> skb_priority(0),tunnel(tun_id=0x20009000010,src=<IP3>,dst=<IP4>,ttl=61,tp_src=37080,tp_dst=7471,flags(csum|key)),skb_mark(0),ct_state(0x21),ct_zone(0x3a),ct_tuple4(src=220.242.181.108,dst=192.168.0.5,proto=1,tp_src=8,tp_dst=0),in_port(2) >>>> mtu 0 >>> >>> Hi. Not sure if you already figured out the root cause, but >>> this looks very similar to the issue fixed by: >>> https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/ >>> >>>> >>>> How can I understand which argument is exactly invalid and what could be >>>> the reason? >>> >>> Unfortunately, it's not possible right now, so debugging is tricky. >>> There is a plan to receive meaningful error messages from the kernel via >>> netlink and print them in logs, but this is not implemented yet. >> >> I have an environment that displays similar symptoms and unfortunately >> the in-kernel fix does not help here. I'm currently in the process of >> debugging and as you point out, debugging issues with the in-kernel >> datapath or OVS's conversation with it is pretty hard. >> >> In reference to the above stated plans to add more logging to OVS in >> this area I wondered if this work has started. If not I might try to >> get that done to help resolve the issue we see here, any pointers, >> in-flight thoughts or work on how to approach it would be appreciated. >> > > We recently added to OVS support to actually print extended ack > messages received from the kernel: > https://github.com/openvswitch/ovs/commit/4a6a4734622e42367faf39cd3938bc8a57786282 > but I'm not sure if anyone started to add more messages to the kernel > itself. > > In general, we need to find all the places where execute/put/get/del > requests returns an error inside the kernel and add some meaningful > extack message to the netlink reply at that point. It's kind of a > lot of work, but it should be very beneficial for anyone who tries > to debug this kind of in-kernel problems or miscommunication between > OVS kernel and userspace. > > Aaron, Eelco, do you know if any work started in the kernel in that > direction, or maybe you worked on something like this? I’m not aware of any work in this area, other than TC. I’ve created BZ2059162, just in case someone has time to work on it. //Eelco _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
