Hi all,
Enabling the debugging, we can see the following:
2024-04-26T20:54:25.880Z|00004|dpif(handler75)|DBG|system@ovs-system: miss
upcall:
recirc_id(0),dp_hash(0),skb_priority(0),in_port(1706),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=129,code=0)
icmp6,vlan_tci=0x0000,dl_src=fa:16:3e:d7:c9:46,dl_dst=fa:16:3e:9b:b3:c6,ipv6_src=2001:db8:2:a::301,ipv6_dst=2001:db8:1:2::10,ipv6_label=0xbde5f,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=129,icmp_code=0
icmp6_csum:e5d6
#
2024-04-26T20:54:29.933Z|00042|dpif(revalidator131)|DBG|system@ovs-system:
flow_del ufid:82c1cdb8-d56e-4d0b-b65c-96edecef76e1
recirc_id(0),dp_hash(0),skb_priority(0),in_port(1706),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=129,code=0),
packets:0, bytes:0, used:never
#
2024-04-26T20:54:37.632Z|00173|dpif(handler23)|DBG|system@ovs-system:
put[create] ufid:82c1cdb8-d56e-4d0b-b65c-96edecef76e1
recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(1706),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=129/0xfc,code=0/0),
actions:ct(zone=7185),recirc(0x5ef91)
There is this miss upcall and then the revalidator removes the flow from
the datapath and it is added again. So, probably this is creating the
jitter behavior.
Regards,
Tiago Pires
On Fri, Apr 26, 2024 at 3:33 PM Tiago Pires <tiago.pi...@luizalabs.com>
wrote:
> Hi all,
>
> While testing the upgrade path from OVN 22.03.1/OVS 2.17.2 to OVN
> 23.03.1/OVS 3.1.3 on Ubuntu 22.04/kernel 5.15 and 6.5 we are seeing a
> strange behavior for icmpv6 traffic.
> Before the upgrade a simple north-south or west-east ping between IPv6
> hosts would have a low jitter like below:
>
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2712 ttl=62 time=0.676 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2713 ttl=62 time=0.829 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2714 ttl=62 time=0.568 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2715 ttl=62 time=0.700 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2716 ttl=62 time=0.768 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2717 ttl=62 time=0.599 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2718 ttl=62 time=0.656 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2719 ttl=62 time=0.689 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2720 ttl=62 time=0.724 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2721 ttl=62 time=0.419 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2722 ttl=62 time=0.732 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2723 ttl=62 time=0.717 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2724 ttl=62 time=0.755 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2725 ttl=62 time=0.765 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2726 ttl=62 time=0.535 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2727 ttl=62 time=0.865 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2728 ttl=62 time=0.692 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2729 ttl=62 time=0.597 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2730 ttl=62 time=0.661 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=2731 ttl=62 time=0.558 ms
>
> But after the upgrade, the same ping started to have a higher jitter:
>
> 64 bytes from 2001:db8:2:a::301: icmp_seq=37 ttl=253 time=2.14 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=38 ttl=253 time=50.2 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=39 ttl=253 time=57.0 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=40 ttl=253 time=61.5 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=41 ttl=253 time=2.16 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=42 ttl=253 time=1.68 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=43 ttl=253 time=1.63 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=44 ttl=253 time=3.32 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=45 ttl=253 time=1.87 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=46 ttl=253 time=39.6 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=47 ttl=253 time=2.87 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=48 ttl=253 time=60.0 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=49 ttl=253 time=1.79 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=50 ttl=253 time=2.06 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=51 ttl=253 time=2.45 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=52 ttl=253 time=2.10 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=53 ttl=253 time=4.39 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=54 ttl=253 time=2.91 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=55 ttl=253 time=1.79 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=56 ttl=253 time=1.80 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=57 ttl=253 time=2.26 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=58 ttl=253 time=55.1 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=59 ttl=253 time=57.2 ms
> 64 bytes from 2001:db8:2:a::301: icmp_seq=60 ttl=253 time=3.34 ms
>
> --- 2001:db8:2:a::301 ping statistics ---
> 60 packets transmitted, 60 received, 0% packet loss, time 59120ms
> rtt min/avg/max/mdev = 0.531/16.329/61.464/23.395 ms
>
> The icmp v4 is not affected and we have the same jitter before and after
> the upgrade. Regarding throughput, I ran a TCP/UDP (v4/v6) throughput test
> before and after the upgrade and the numbers are similar, so it seems it
> happens only in special with icmpv6 traffic.
>
> Checking the datapath, I can see the flow related with the in_port(1706)
> where the VM is connected being removed and installed again:
>
> # ovs-dpctl dump-flows | grep 2001:db8:2:a::301
> recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
> packets:7, bytes:826, used:0.674s, actions:1706
> recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
> packets:7, bytes:826, used:0.709s, actions:ct(zone=7185),recirc(0x1ad3d)
>
> # ovs-dpctl dump-flows | grep 2001:db8:2:a::301
> recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
> packets:11, bytes:1298, used:0.190s, actions:1706
> recirc_id(0),in_port(1706),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,proto=58,hlimit=255,frag=no),icmpv6(type=128/0xfc),
> packets:0, bytes:0, used:never, actions:ct(zone=7185),recirc(0x1b13c)
> recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
> packets:11, bytes:1298, used:0.237s, actions:ct(zone=7185),recirc(0x1ad3d)
>
> (Note: no OVS HW Offloading)
>
> So, it seems there is a flow missing, the flow goes to userspace and it is
> installed again on the datapath. Maybe it can explain the higher jitter.
>
> From what I explained, does anyone remember any recent changes that could
> introduce such behavior? If anyone who has an IPv6 deployment and is using
> the last OVN/OVS could test if there is similar behavior, I would
> appreciate it.
>
> Regards,
>
> Tiago Pires
>
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no
cabeçalho inicial. Se você não está listado nos endereços constantes no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão
imediatamente anuladas e proibidas’._
* **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja
presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
