On Sat, May 24, 2025, 7:46 AM Serhat Rıfat Demircan < [email protected]> wrote:
> Finally I can confirm and reproduce when gateway mac ages out from mac > address table(or flush) ovs starts to flood all ports and flows switch to > software path. When It learn gateway mac address by any arp requests > hardware offload works again. > > It seems increasing mac-aging on br-ex can be an option. What do you > suggest permanent solution for this issue? > If you know well in advance of the destination Mac or the traffic pattern then probably you can consider programming higher priority openflow rules in br-ex ? Thanks Numan > Thanks, > Serhat > > On Thu, May 22, 2025 at 19:50 Serhat Rıfat Demircan < > [email protected]> wrote: > >> No OVN, it is only OVS. oftrace says it flooding because "no learned MAC >> for destination". Is this expected while there is existing traffic? And >> another question is, can I prevent flooding? :) >> >> # ovs-appctl ofproto/trace br-int >>> "in_port=4,ip,eth_src=fa:16:7e:e1:50:af,eth_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y" >>> Flow: >>> ip,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >> >> >> bridge("br-int") >>> ---------------- >>> 0. priority 0, cookie 0x64ecae89c6f9a081 >>> goto_table:58 >>> 58. priority 0, cookie 0x64ecae89c6f9a081 >>> goto_table:60 >>> 60. in_port=4, priority 100, cookie 0x64ecae89c6f9a081 >>> set_field:0x4->reg5 >>> set_field:0x1->reg6 >>> resubmit(,71) >>> 71. ip,reg5=0x4,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X, >>> priority 65, cookie 0x64ecae89c6f9a081 >>> ct(table=72,zone=NXM_NX_REG6[0..15]) >>> drop >>> -> A clone of the packet is forked to recirculate. The forked >>> pipeline will be resumed at table 72. >>> -> Sets the packet to an untracked state, and clears all the >>> conntrack fields. >> >> >> Final flow: >>> ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>> Megaflow: >>> recirc_id=0,ct_state=-trk,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X,nw_proto=0,nw_frag=no >>> Datapath actions: ct(zone=1),recirc(0x1) >> >> >> >>> =============================================================================== >>> recirc(0x1) - resume conntrack with default ct_state=trk|new (use >>> --ct-next to customize) >>> >>> =============================================================================== >> >> >> Flow: >>> recirc_id=0x1,ct_state=new|trk,ct_zone=1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >> >> >> bridge("br-int") >>> ---------------- >>> thaw >>> Resuming from table 72 >>> 72. ct_state=+new-est,ip,reg5=0x4, priority 74, cookie 0x64ecae89c6f9a081 >>> resubmit(,73) >>> 73. ct_state=+new-est,ip,reg5=0x4, priority 90, cookie 0x64ecae89c6f9a081 >>> ct(commit,zone=NXM_NX_REG6[0..15]) >>> drop >>> -> Sets the packet to an untracked state, and clears all the >>> conntrack fields. >>> resubmit(,91) >>> 91. priority 1, cookie 0x64ecae89c6f9a081 >>> resubmit(,94) >>> 94. >>> reg6=0x1,dl_src=fa:16:7e:e1:50:af,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, >>> priority 10, cookie 0x64ecae89c6f9a081 >>> push_vlan:0x8100 >>> set_field:4097->vlan_vid >>> output:1 >> >> >> bridge("br-ex") >>> --------------- >>> 0. in_port=2,dl_vlan=1, priority 4, cookie 0xd9b277145604416 >>> set_field:4207->vlan_vid >>> NORMAL >>> -> no learned MAC for destination, flooding >> >> >> Final flow: >>> recirc_id=0x1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,dl_vlan=1,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>> Megaflow: >>> recirc_id=0x1,ct_state=+new-est-rel-rpl,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_frag=no >>> Datapath actions: ct(commit,zone=1),push_vlan(vid=111,pcp=0),1,5 >> >> >> Thanks, >> Serhat >> >> On Thu, May 22, 2025 at 6:41 PM Numan Siddique <[email protected]> wrote: >> >>> On Thu, May 22, 2025 at 11:33 AM Serhat Rıfat Demircan via discuss >>> <[email protected]> wrote: >>> > >>> > I think this is the actual flow change in ovs side. When br-ex(br-ex >>> is external bridge, openstack is bond interface) is added as output >>> interface it breaks offloading. Still do not know why it changes :) Do you >>> have any idea why ovs decide to redirect traffic to external bridge? >>> > >>> > Offloaded flow: >>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, >>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= >>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >>> packets:1325324668, bytes:4434013761329, used:3.390s, offloaded:yes, dp:tc, >>> actions:push_vlan(vid=111,pcp=0),openstack >>> > >>> > Not offloaded flow: >>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, >>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= >>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >>> packets:1387549198, bytes:4643405901604, used:0.000s, dp:tc, >>> actions:push_vlan(vid=111,pcp=0),br-ex,openstack >>> >>> Maybe you run ofproto/trace on the sample traffic and see why it is >>> outputting to br-ex ? >>> >>> Is it with OVN ? >>> >>> Thanks >>> Numan >>> >>> > >>> > Thanks, >>> > Serhat >>> > >>> > On Wed, May 21, 2025 at 11:51 AM Serhat Rıfat Demircan < >>> [email protected]> wrote: >>> >> >>> >> After little more digging, I see miss upcalls and flow modify logs >>> like below while tc flow switching to not_in_hw. I also see similar upcall >>> while switching back to in_hw. >>> >> >>> >> >>> >> 2025-05-21T08:14:27.189Z|19748|dpif(handler3)|DBG|system@ovs-system: >>> miss upcall: >>> >> >>> recirc_id(0),dp_hash(0),skb_priority(0),in_port(6),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:7e:75:94:76,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_A,dst=IP_B,proto=6,tos=0,ttl=64,frag=no),tcp(src=52774,dst=6835),tcp_flags(ack) >>> >> >>> tcp,vlan_tci=0x0000,dl_src=fa:16:7e:75:94:76,dl_dst=00:1c:73:aa:bb:cc,nw_src=IP_A,nw_dst=IP_B,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=52774,tp_dst=6835,tcp_flags=ack >>> tcp_csum:5abe >>> >> >>> >> >>> 2025-05-21T08:14:27.189Z|104292|dpif(revalidator10)|DBG|system@ovs-system: >>> put[modify] ufid:52588bba-04e5-43fa-b677-5a96dbd11f22 >>> recirc_id(0x5),dp_hash(0/0),skb_priority(0/0),in_port(4),skb_mark(0/0),ct_state(0x22/0xf),ct_zone(0x1/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=IP_C/ >>> 0.0.0.0,dst=IP_D/0.0.0.0 >>> ,proto=6/0,tp_src=58586/0,tp_dst=6821/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_C/ >>> 0.0.0.0,dst=IP_D/0.0.0.0,proto=6/0,tos=0/0,ttl=64/0,frag=no),tcp(src=58586/0,dst=6821/0),tcp_flags(0/0), >>> actions:push_vlan(vid=111,pcp=0),1,5 >>> >> >>> >> On Tue, May 20, 2025 at 7:49 PM Ilya Maximets <[email protected]> >>> wrote: >>> >>> >>> >>> On 5/20/25 4:32 PM, Serhat Rıfat Demircan wrote: >>> >>> > Yes, statistics are updated. >>> >>> > >>> >>> > # ovs-appctl dpctl/dump-flows type=offloaded | grep >>> dst=00:1c:73:aa:bb:cc >>> >>> > >>> recirc_id(0x5),in_port(4),ct_state(+est-rel+rpl-inv+trk),ct_zone(0x1),ct_mark(0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >>> packets:222, bytes:30340, used:0.170s, actions:push_vlan(vid=111,pcp=0),5 >>> >>> > >>> recirc_id(0x5),in_port(4),ct_state(+est-rel-rpl+trk),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >>> packets:54220862, bytes:182391451911, used:0.170s, >>> actions:push_vlan(vid=111,pcp=0),5 >>> >>> >>> >>> Then one of the options would be to turn on debug logging and try >>> >>> and catch the moment the flow actions change, and then correlate the >>> >>> change with some other events in the log. I'd suggest to start with: >>> >>> ovs-appctl vlog/set ofproto_dpif_xlate:file:dbg dpif:file:dbg >>> >>> >>> >>> Best regards, Ilya Maximets. >>> >>> >>> >>> > >>> >>> > On Tue, May 20, 2025 at 3:39 PM Ilya Maximets <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> > >>> >>> > On 5/20/25 2:20 PM, Serhat Rıfat Demircan wrote: >>> >>> > > Redirecting to br-ex happens even if there is a single >>> hardware offloaded port on hypervisor. Still can't find the actual reason. >>> >>> > > Increasing mac-aging-time and mac-table-size did not help >>> also. >>> >>> > > >>> >>> > > By the way, it is bidirectional storage(ceph) traffic. >>> >>> > >>> >>> > Is statistics on the datapath flows properly updated? >>> >>> > i.e. if you watch 'ovs-appctl dpctl/dump-flows', do you see >>> stats >>> >>> > properly incremented when the flow is offloaded? >>> >>> > >>> >>> > > >>> >>> > > >>> >>> > > >>> >>> > > On Wed, May 14, 2025 at 1:39 PM Ilya Maximets < >>> [email protected] <mailto:[email protected]> <mailto: >>> [email protected] <mailto:[email protected]>>> wrote: >>> >>> > > >>> >>> > > On 5/8/25 4:15 PM, Serhat Rıfat Demircan via discuss >>> wrote: >>> >>> > > > Hello Everyone, >>> >>> > > > >>> >>> > > > I've been testing OVS hardware offloaded ports with >>> Openstack for a while. >>> >>> > > > I have ConnectX-5 and ConnectX-6 cards in my lab. >>> Iperf tests are looking >>> >>> > > > promising, but when I try to use hardware offloaded >>> ports for krbd traffic, >>> >>> > > > after some time, traffic falls back to the software >>> path. Another interesting >>> >>> > > > thing for me is without changing anything, it can also >>> switch back to the >>> >>> > > > hardware path. I could not find the real reason. >>> >>> > > > >>> >>> > > > I'm not sure if this is a configuration issue or if >>> I'm hitting to cards limit. >>> >>> > > > If any of you guys are experienced with this all ideas >>> are more than welcome. >>> >>> > > > >>> >>> > > > Openstack version: Caracal >>> >>> > > > Operation System: Ubuntu 22.04 >>> >>> > > > OVS version: 3.3.0 >>> >>> > > > >>> >>> > > > I see tc flow change: >>> >>> > > > >>> >>> > > > *From Offloaded:* >>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress >>> >>> > > > filter protocol ip pref 2 flower chain 0 >>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >>> >>> > > > dst_mac 00:1c:73:aa:bb:cc >>> >>> > > > src_mac fa:16:7e:2e:8c:f0 >>> >>> > > > eth_type ipv4 >>> >>> > > > ip_flags nofrag >>> >>> > > > skip_sw >>> >>> > > > in_hw in_hw_count 1 >>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q >>> priority 0 pipe >>> >>> > > > index 4 ref 1 bind 1 >>> >>> > > > no_percpu >>> >>> > > > used_hw_stats delayed >>> >>> > > > >>> >>> > > > action order 2: mirred (Egress Redirect to device >>> openstack) stolen >>> >>> > > > index 4 ref 1 bind 1 >>> >>> > > > cookie 605b82e5b94ef3fa63c3e78349851a27 >>> >>> > > > no_percpu >>> >>> > > > used_hw_stats delayed >>> >>> > > > >>> >>> > > > *To software path:* >>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress >>> >>> > > > filter protocol ip pref 2 flower chain 0 >>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >>> >>> > > > dst_mac 00:1c:73:aa:bb:cc >>> >>> > > > src_mac fa:16:7e:2e:8c:f0 >>> >>> > > > eth_type ipv4 >>> >>> > > > ip_flags nofrag >>> >>> > > > not_in_hw >>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q >>> priority 0 pipe >>> >>> > > > index 4 ref 1 bind 1 >>> >>> > > > no_percpu >>> >>> > > > >>> >>> > > > action order 2: skbedit ptype host pipe >>> >>> > > > index 11 ref 1 bind 1 >>> >>> > > > >>> >>> > > > action order 3: mirred (Ingress Mirror to device >>> br-ex) pipe >>> >>> > > > index 13 ref 1 bind 1 >>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d >>> >>> > > > no_percpu >>> >>> > > > >>> >>> > > > action order 4: mirred (Egress Redirect to device >>> openstack) stolen >>> >>> > > > index 15 ref 1 bind 1 >>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d >>> >>> > > > no_percpu >>> >>> > > >>> >>> > > Here you can see that traffic is getting broadcasted on >>> all ports in the >>> >>> > > br-ex bridge, and since some of these ports are virtual, >>> this operation >>> >>> > > cannot be offloaded to hardware. >>> >>> > > >>> >>> > > I would guess, after some time the FDB entry expires and >>> OVS no longer >>> >>> > > knows on which port the traffic supposed to be sent >>> out. Is that a >>> >>> > > unidirection traffic? i.e. are you also sending traffic >>> in the opposite >>> >>> > > direction? If not, are you sending some periodic ARP >>> requests for the >>> >>> > > bridge to learn where the destination is? >>> >>> > > >>> >>> > > If the traffic is bidirectional, then it might be a NIC >>> driver/firmware >>> >>> > > issue not providing updated stats. >>> >>> > > >>> >>> > > Best regards, Ilya Maximets. >>> >>> > > >>> >>> > >>> >>> >>> > _______________________________________________ >>> > discuss mailing list >>> > [email protected] >>> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>> >>
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
