Hi all,
While testing the upgrade path from OVN 22.03.1/OVS 2.17.2 to OVN
23.03.1/OVS 3.1.3 on Ubuntu 22.04/kernel 5.15 and 6.5 we are seeing a
strange behavior for icmpv6 traffic.
Before the upgrade a simple north-south or west-east ping between IPv6
hosts would have a low jitter like below:
64 bytes from 2001:db8:2:a::301: icmp_seq=2712 ttl=62 time=0.676 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2713 ttl=62 time=0.829 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2714 ttl=62 time=0.568 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2715 ttl=62 time=0.700 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2716 ttl=62 time=0.768 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2717 ttl=62 time=0.599 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2718 ttl=62 time=0.656 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2719 ttl=62 time=0.689 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2720 ttl=62 time=0.724 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2721 ttl=62 time=0.419 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2722 ttl=62 time=0.732 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2723 ttl=62 time=0.717 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2724 ttl=62 time=0.755 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2725 ttl=62 time=0.765 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2726 ttl=62 time=0.535 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2727 ttl=62 time=0.865 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2728 ttl=62 time=0.692 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2729 ttl=62 time=0.597 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2730 ttl=62 time=0.661 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=2731 ttl=62 time=0.558 ms
But after the upgrade, the same ping started to have a higher jitter:
64 bytes from 2001:db8:2:a::301: icmp_seq=37 ttl=253 time=2.14 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=38 ttl=253 time=50.2 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=39 ttl=253 time=57.0 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=40 ttl=253 time=61.5 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=41 ttl=253 time=2.16 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=42 ttl=253 time=1.68 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=43 ttl=253 time=1.63 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=44 ttl=253 time=3.32 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=45 ttl=253 time=1.87 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=46 ttl=253 time=39.6 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=47 ttl=253 time=2.87 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=48 ttl=253 time=60.0 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=49 ttl=253 time=1.79 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=50 ttl=253 time=2.06 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=51 ttl=253 time=2.45 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=52 ttl=253 time=2.10 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=53 ttl=253 time=4.39 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=54 ttl=253 time=2.91 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=55 ttl=253 time=1.79 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=56 ttl=253 time=1.80 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=57 ttl=253 time=2.26 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=58 ttl=253 time=55.1 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=59 ttl=253 time=57.2 ms
64 bytes from 2001:db8:2:a::301: icmp_seq=60 ttl=253 time=3.34 ms
--- 2001:db8:2:a::301 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59120ms
rtt min/avg/max/mdev = 0.531/16.329/61.464/23.395 ms
The icmp v4 is not affected and we have the same jitter before and after
the upgrade. Regarding throughput, I ran a TCP/UDP (v4/v6) throughput test
before and after the upgrade and the numbers are similar, so it seems it
happens only in special with icmpv6 traffic.
Checking the datapath, I can see the flow related with the in_port(1706)
where the VM is connected being removed and installed again:
# ovs-dpctl dump-flows | grep 2001:db8:2:a::301
recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:7, bytes:826, used:0.674s, actions:1706
recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:7, bytes:826, used:0.709s, actions:ct(zone=7185),recirc(0x1ad3d)
# ovs-dpctl dump-flows | grep 2001:db8:2:a::301
recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:11, bytes:1298, used:0.190s, actions:1706
recirc_id(0),in_port(1706),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,proto=58,hlimit=255,frag=no),icmpv6(type=128/0xfc),
packets:0, bytes:0, used:never, actions:ct(zone=7185),recirc(0x1b13c)
recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:11, bytes:1298, used:0.237s, actions:ct(zone=7185),recirc(0x1ad3d)
(Note: no OVS HW Offloading)
So, it seems there is a flow missing, the flow goes to userspace and it is
installed again on the datapath. Maybe it can explain the higher jitter.
>From what I explained, does anyone remember any recent changes that could
introduce such behavior? If anyone who has an IPv6 deployment and is using
the last OVN/OVS could test if there is similar behavior, I would
appreciate it.
Regards,
Tiago Pires
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no
cabeçalho inicial. Se você não está listado nos endereços constantes no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão
imediatamente anuladas e proibidas’._
* **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja
presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
