Is it expected gateway mac address ages out while there is ongoing traffic when hardware offloading is active?
# ovs-appctl fdb/show br-ex | grep 00:1c:73:aa:bb:cc > 1 111 00:1c:73:aa:bb:cc 889 Thanks, Serhat On Sun, May 25, 2025 at 6:56 PM Numan Siddique <[email protected]> wrote: > > > > > On Sat, May 24, 2025, 7:46 AM Serhat Rıfat Demircan < > [email protected]> wrote: > >> Finally I can confirm and reproduce when gateway mac ages out from mac >> address table(or flush) ovs starts to flood all ports and flows switch to >> software path. When It learn gateway mac address by any arp requests >> hardware offload works again. >> >> It seems increasing mac-aging on br-ex can be an option. What do you >> suggest permanent solution for this issue? >> > > If you know well in advance of the destination Mac or the traffic pattern > then probably you can consider programming higher priority openflow rules > in br-ex ? > > Thanks > Numan > > >> Thanks, >> Serhat >> >> On Thu, May 22, 2025 at 19:50 Serhat Rıfat Demircan < >> [email protected]> wrote: >> >>> No OVN, it is only OVS. oftrace says it flooding because "no learned MAC >>> for destination". Is this expected while there is existing traffic? And >>> another question is, can I prevent flooding? :) >>> >>> # ovs-appctl ofproto/trace br-int >>>> "in_port=4,ip,eth_src=fa:16:7e:e1:50:af,eth_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y" >>>> Flow: >>>> ip,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>> >>> >>> bridge("br-int") >>>> ---------------- >>>> 0. priority 0, cookie 0x64ecae89c6f9a081 >>>> goto_table:58 >>>> 58. priority 0, cookie 0x64ecae89c6f9a081 >>>> goto_table:60 >>>> 60. in_port=4, priority 100, cookie 0x64ecae89c6f9a081 >>>> set_field:0x4->reg5 >>>> set_field:0x1->reg6 >>>> resubmit(,71) >>>> 71. ip,reg5=0x4,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X, >>>> priority 65, cookie 0x64ecae89c6f9a081 >>>> ct(table=72,zone=NXM_NX_REG6[0..15]) >>>> drop >>>> -> A clone of the packet is forked to recirculate. The forked >>>> pipeline will be resumed at table 72. >>>> -> Sets the packet to an untracked state, and clears all the >>>> conntrack fields. >>> >>> >>> Final flow: >>>> ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>>> Megaflow: >>>> recirc_id=0,ct_state=-trk,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X,nw_proto=0,nw_frag=no >>>> Datapath actions: ct(zone=1),recirc(0x1) >>> >>> >>> >>>> =============================================================================== >>>> recirc(0x1) - resume conntrack with default ct_state=trk|new (use >>>> --ct-next to customize) >>>> >>>> =============================================================================== >>> >>> >>> Flow: >>>> recirc_id=0x1,ct_state=new|trk,ct_zone=1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>> >>> >>> bridge("br-int") >>>> ---------------- >>>> thaw >>>> Resuming from table 72 >>>> 72. ct_state=+new-est,ip,reg5=0x4, priority 74, cookie >>>> 0x64ecae89c6f9a081 >>>> resubmit(,73) >>>> 73. ct_state=+new-est,ip,reg5=0x4, priority 90, cookie >>>> 0x64ecae89c6f9a081 >>>> ct(commit,zone=NXM_NX_REG6[0..15]) >>>> drop >>>> -> Sets the packet to an untracked state, and clears all the >>>> conntrack fields. >>>> resubmit(,91) >>>> 91. priority 1, cookie 0x64ecae89c6f9a081 >>>> resubmit(,94) >>>> 94. >>>> reg6=0x1,dl_src=fa:16:7e:e1:50:af,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, >>>> priority 10, cookie 0x64ecae89c6f9a081 >>>> push_vlan:0x8100 >>>> set_field:4097->vlan_vid >>>> output:1 >>> >>> >>> bridge("br-ex") >>>> --------------- >>>> 0. in_port=2,dl_vlan=1, priority 4, cookie 0xd9b277145604416 >>>> set_field:4207->vlan_vid >>>> NORMAL >>>> -> no learned MAC for destination, flooding >>> >>> >>> Final flow: >>>> recirc_id=0x1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,dl_vlan=1,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no >>>> Megaflow: >>>> recirc_id=0x1,ct_state=+new-est-rel-rpl,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_frag=no >>>> Datapath actions: ct(commit,zone=1),push_vlan(vid=111,pcp=0),1,5 >>> >>> >>> Thanks, >>> Serhat >>> >>> On Thu, May 22, 2025 at 6:41 PM Numan Siddique <[email protected]> wrote: >>> >>>> On Thu, May 22, 2025 at 11:33 AM Serhat Rıfat Demircan via discuss >>>> <[email protected]> wrote: >>>> > >>>> > I think this is the actual flow change in ovs side. When br-ex(br-ex >>>> is external bridge, openstack is bond interface) is added as output >>>> interface it breaks offloading. Still do not know why it changes :) Do you >>>> have any idea why ovs decide to redirect traffic to external bridge? >>>> > >>>> > Offloaded flow: >>>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, >>>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= >>>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >>>> packets:1325324668, bytes:4434013761329, used:3.390s, offloaded:yes, dp:tc, >>>> actions:push_vlan(vid=111,pcp=0),openstack >>>> > >>>> > Not offloaded flow: >>>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, >>>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= >>>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >>>> packets:1387549198, bytes:4643405901604, used:0.000s, dp:tc, >>>> actions:push_vlan(vid=111,pcp=0),br-ex,openstack >>>> >>>> Maybe you run ofproto/trace on the sample traffic and see why it is >>>> outputting to br-ex ? >>>> >>>> Is it with OVN ? >>>> >>>> Thanks >>>> Numan >>>> >>>> > >>>> > Thanks, >>>> > Serhat >>>> > >>>> > On Wed, May 21, 2025 at 11:51 AM Serhat Rıfat Demircan < >>>> [email protected]> wrote: >>>> >> >>>> >> After little more digging, I see miss upcalls and flow modify logs >>>> like below while tc flow switching to not_in_hw. I also see similar upcall >>>> while switching back to in_hw. >>>> >> >>>> >> >>>> >> 2025-05-21T08:14:27.189Z|19748|dpif(handler3)|DBG|system@ovs-system: >>>> miss upcall: >>>> >> >>>> recirc_id(0),dp_hash(0),skb_priority(0),in_port(6),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:7e:75:94:76,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_A,dst=IP_B,proto=6,tos=0,ttl=64,frag=no),tcp(src=52774,dst=6835),tcp_flags(ack) >>>> >> >>>> tcp,vlan_tci=0x0000,dl_src=fa:16:7e:75:94:76,dl_dst=00:1c:73:aa:bb:cc,nw_src=IP_A,nw_dst=IP_B,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=52774,tp_dst=6835,tcp_flags=ack >>>> tcp_csum:5abe >>>> >> >>>> >> >>>> 2025-05-21T08:14:27.189Z|104292|dpif(revalidator10)|DBG|system@ovs-system: >>>> put[modify] ufid:52588bba-04e5-43fa-b677-5a96dbd11f22 >>>> recirc_id(0x5),dp_hash(0/0),skb_priority(0/0),in_port(4),skb_mark(0/0),ct_state(0x22/0xf),ct_zone(0x1/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=IP_C/ >>>> 0.0.0.0,dst=IP_D/0.0.0.0 >>>> ,proto=6/0,tp_src=58586/0,tp_dst=6821/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_C/ >>>> 0.0.0.0,dst=IP_D/0.0.0.0,proto=6/0,tos=0/0,ttl=64/0,frag=no),tcp(src=58586/0,dst=6821/0),tcp_flags(0/0), >>>> actions:push_vlan(vid=111,pcp=0),1,5 >>>> >> >>>> >> On Tue, May 20, 2025 at 7:49 PM Ilya Maximets <[email protected]> >>>> wrote: >>>> >>> >>>> >>> On 5/20/25 4:32 PM, Serhat Rıfat Demircan wrote: >>>> >>> > Yes, statistics are updated. >>>> >>> > >>>> >>> > # ovs-appctl dpctl/dump-flows type=offloaded | grep >>>> dst=00:1c:73:aa:bb:cc >>>> >>> > >>>> recirc_id(0x5),in_port(4),ct_state(+est-rel+rpl-inv+trk),ct_zone(0x1),ct_mark(0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >>>> packets:222, bytes:30340, used:0.170s, actions:push_vlan(vid=111,pcp=0),5 >>>> >>> > >>>> recirc_id(0x5),in_port(4),ct_state(+est-rel-rpl+trk),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >>>> packets:54220862, bytes:182391451911, used:0.170s, >>>> actions:push_vlan(vid=111,pcp=0),5 >>>> >>> >>>> >>> Then one of the options would be to turn on debug logging and try >>>> >>> and catch the moment the flow actions change, and then correlate the >>>> >>> change with some other events in the log. I'd suggest to start >>>> with: >>>> >>> ovs-appctl vlog/set ofproto_dpif_xlate:file:dbg dpif:file:dbg >>>> >>> >>>> >>> Best regards, Ilya Maximets. >>>> >>> >>>> >>> > >>>> >>> > On Tue, May 20, 2025 at 3:39 PM Ilya Maximets <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>> > >>>> >>> > On 5/20/25 2:20 PM, Serhat Rıfat Demircan wrote: >>>> >>> > > Redirecting to br-ex happens even if there is a single >>>> hardware offloaded port on hypervisor. Still can't find the actual reason. >>>> >>> > > Increasing mac-aging-time and mac-table-size did not help >>>> also. >>>> >>> > > >>>> >>> > > By the way, it is bidirectional storage(ceph) traffic. >>>> >>> > >>>> >>> > Is statistics on the datapath flows properly updated? >>>> >>> > i.e. if you watch 'ovs-appctl dpctl/dump-flows', do you see >>>> stats >>>> >>> > properly incremented when the flow is offloaded? >>>> >>> > >>>> >>> > > >>>> >>> > > >>>> >>> > > >>>> >>> > > On Wed, May 14, 2025 at 1:39 PM Ilya Maximets < >>>> [email protected] <mailto:[email protected]> <mailto: >>>> [email protected] <mailto:[email protected]>>> wrote: >>>> >>> > > >>>> >>> > > On 5/8/25 4:15 PM, Serhat Rıfat Demircan via discuss >>>> wrote: >>>> >>> > > > Hello Everyone, >>>> >>> > > > >>>> >>> > > > I've been testing OVS hardware offloaded ports with >>>> Openstack for a while. >>>> >>> > > > I have ConnectX-5 and ConnectX-6 cards in my lab. >>>> Iperf tests are looking >>>> >>> > > > promising, but when I try to use hardware offloaded >>>> ports for krbd traffic, >>>> >>> > > > after some time, traffic falls back to the software >>>> path. Another interesting >>>> >>> > > > thing for me is without changing anything, it can >>>> also switch back to the >>>> >>> > > > hardware path. I could not find the real reason. >>>> >>> > > > >>>> >>> > > > I'm not sure if this is a configuration issue or if >>>> I'm hitting to cards limit. >>>> >>> > > > If any of you guys are experienced with this all >>>> ideas are more than welcome. >>>> >>> > > > >>>> >>> > > > Openstack version: Caracal >>>> >>> > > > Operation System: Ubuntu 22.04 >>>> >>> > > > OVS version: 3.3.0 >>>> >>> > > > >>>> >>> > > > I see tc flow change: >>>> >>> > > > >>>> >>> > > > *From Offloaded:* >>>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress >>>> >>> > > > filter protocol ip pref 2 flower chain 0 >>>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >>>> >>> > > > dst_mac 00:1c:73:aa:bb:cc >>>> >>> > > > src_mac fa:16:7e:2e:8c:f0 >>>> >>> > > > eth_type ipv4 >>>> >>> > > > ip_flags nofrag >>>> >>> > > > skip_sw >>>> >>> > > > in_hw in_hw_count 1 >>>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q >>>> priority 0 pipe >>>> >>> > > > index 4 ref 1 bind 1 >>>> >>> > > > no_percpu >>>> >>> > > > used_hw_stats delayed >>>> >>> > > > >>>> >>> > > > action order 2: mirred (Egress Redirect to device >>>> openstack) stolen >>>> >>> > > > index 4 ref 1 bind 1 >>>> >>> > > > cookie 605b82e5b94ef3fa63c3e78349851a27 >>>> >>> > > > no_percpu >>>> >>> > > > used_hw_stats delayed >>>> >>> > > > >>>> >>> > > > *To software path:* >>>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress >>>> >>> > > > filter protocol ip pref 2 flower chain 0 >>>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >>>> >>> > > > dst_mac 00:1c:73:aa:bb:cc >>>> >>> > > > src_mac fa:16:7e:2e:8c:f0 >>>> >>> > > > eth_type ipv4 >>>> >>> > > > ip_flags nofrag >>>> >>> > > > not_in_hw >>>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q >>>> priority 0 pipe >>>> >>> > > > index 4 ref 1 bind 1 >>>> >>> > > > no_percpu >>>> >>> > > > >>>> >>> > > > action order 2: skbedit ptype host pipe >>>> >>> > > > index 11 ref 1 bind 1 >>>> >>> > > > >>>> >>> > > > action order 3: mirred (Ingress Mirror to device >>>> br-ex) pipe >>>> >>> > > > index 13 ref 1 bind 1 >>>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d >>>> >>> > > > no_percpu >>>> >>> > > > >>>> >>> > > > action order 4: mirred (Egress Redirect to device >>>> openstack) stolen >>>> >>> > > > index 15 ref 1 bind 1 >>>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d >>>> >>> > > > no_percpu >>>> >>> > > >>>> >>> > > Here you can see that traffic is getting broadcasted on >>>> all ports in the >>>> >>> > > br-ex bridge, and since some of these ports are >>>> virtual, this operation >>>> >>> > > cannot be offloaded to hardware. >>>> >>> > > >>>> >>> > > I would guess, after some time the FDB entry expires >>>> and OVS no longer >>>> >>> > > knows on which port the traffic supposed to be sent >>>> out. Is that a >>>> >>> > > unidirection traffic? i.e. are you also sending >>>> traffic in the opposite >>>> >>> > > direction? If not, are you sending some periodic ARP >>>> requests for the >>>> >>> > > bridge to learn where the destination is? >>>> >>> > > >>>> >>> > > If the traffic is bidirectional, then it might be a NIC >>>> driver/firmware >>>> >>> > > issue not providing updated stats. >>>> >>> > > >>>> >>> > > Best regards, Ilya Maximets. >>>> >>> > > >>>> >>> > >>>> >>> >>>> > _______________________________________________ >>>> > discuss mailing list >>>> > [email protected] >>>> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>>> >>>
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
