Hi all, On Thu, Jul 31, 2025 at 11:49 AM Tiago Pires <[email protected]> wrote: > > Hi all, > > I have a potential issue to report where the use case is using > dnat_and_snat and it is keeping many SYN_SENT with UNREPLIED status on > the conntrack table. > The TCP handshake is working fine, but when a user establishes a > connection, there is always a SYN_SENT left on the conntrack table. > > I was able to figure out that this patch [0] (Fix direct access to > SNAT network) introduced this behavior. > > [0] > https://github.com/ovn-org/ovn/commit/40136a2f2c84dca48f7b67c7dc63759e82a44668 > > When we have many connections on the chassis gateways, more than 50% > of the conntrack entries stay in this UNREPLIED state until the entry > is expired according with the nf_conntrack_tcp_timeout_syn_sent > setting. > > How to reproduce: > > #Setup > OS_BASE=ubuntu OS_IMAGE=docker.io/library/ubuntu:jammy ENABLE_SSL=no > CENTRAL_COUNT=1 CHASSIS_COUNT=2 GW_COUNT=1 ./ovn_cluster.sh start > podman exec -it ovn-gw-1 ip link set dev br-ex up > podman exec -it ovn-central-az1-1 ovn-nbctl lr-add lr11 > podman exec -it ovn-central-az1-1 ovn-nbctl ls-add bar11 > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 lr11-public1 > 00:00:00:00:aa:02 172.16.1.100/24 > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-set-gateway-chassis > lr11-public1 ovn-gw-1 > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add public1 lr11-lsp-public1 > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-type lr11-lsp-public1 > router > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-addresses > lr11-lsp-public1 router > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-options > lr11-lsp-public1 router-port=lr11-public1 > podman exec -it ovn-central-az1-1 ovn-nbctl lr-route-add lr11 > 0.0.0.0/0 172.16.1.1 > podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 > dnat_and_snat 172.16.1.101 192.168.10.10 > podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 snat > 172.16.1.100 192.168.10.0/24 > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 rp11-bar > 00:01:ff:22:00:01 192.168.10.1/24 > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 haproxy1 \ > -- lsp-set-addresses haproxy1 "00:00:ff:ff:ff:21 192.168.10.10" > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 bar-rp11 -- > set Logical_Switch_Port bar-rp11 type=router > options:router-port=rp11-bar -- lsp-set-addresses bar-rp11 router > podman exec -it ovn-chassis-1 ip netns add haproxy1 > podman exec -it ovn-chassis-1 ovs-vsctl add-port br-int haproxy1 -- > set interface haproxy1 type=internal > podman exec -it ovn-chassis-1 ip link set haproxy1 netns haproxy1 > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip addr add > 192.168.10.10/24 dev haproxy1 > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set dev > haproxy1 mtu 1400 > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set > haproxy1 address 00:00:ff:ff:ff:21 > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set haproxy1 up > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip route add > default via 192.168.10.1 > podman exec -it ovn-chassis-1 ovs-vsctl set Interface haproxy1 > external_ids:iface-id=haproxy1 > podman exec ovn-chassis-1 ip netns exec haproxy1 python3 -m > http.server 80 -d /tmp/ >/dev/null 2>&1 & > > #Test (client IP - 172.16.1.50) > ip netns exec ovnfake-ext1 curl http://172.16.1.101 > > #tcpdump on ovn-gw-1 (eth2) > > 13:50:31.703106 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [S], seq > 2055144661, win 64240, options [mss 1460,sackOK,TS val 2795500537 ecr > 0,nop,wscale 7], length 0 > 13:50:31.705836 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [S.], > seq 3211627047, ack 2055144662, win 64704, options [mss 1360,sackOK,TS > val 875814075 ecr 2795500537,nop,wscale 7], length 0 > 13:50:31.705973 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > 1, win 502, options [nop,nop,TS val 2795500540 ecr 875814075], length > 0 > 13:50:31.706065 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [P.], > seq 1:77, ack 1, win 502, options [nop,nop,TS val 2795500540 ecr > 875814075], length 76: HTTP: GET / HTTP/1.1 > 13:50:31.706323 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack > 77, win 505, options [nop,nop,TS val 875814077 ecr 2795500540], length > 0 > 13:50:31.708484 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], > seq 1:187, ack 77, win 505, options [nop,nop,TS val 875814079 ecr > 2795500540], length 186: HTTP: HTTP/1.0 200 OK > 13:50:31.708494 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > 187, win 501, options [nop,nop,TS val 2795500542 ecr 875814079], > length 0 > 13:50:31.708567 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], > seq 187:248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr > 2795500542], length 61: HTTP > 13:50:31.708571 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > 248, win 501, options [nop,nop,TS val 2795500543 ecr 875814080], > length 0 > 13:50:31.708605 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [F.], > seq 248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr > 2795500543], length 0 > 13:50:31.708673 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [F.], > seq 77, ack 249, win 501, options [nop,nop,TS val 2795500543 ecr > 875814080], length 0 > 13:50:31.708759 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack > 78, win 505, options [nop,nop,TS val 875814080 ecr 2795500543], length > 0 > > > #conntrack UNREPLIED entries on ovn-gw-1 > # conntrack -L | grep -v 6081 | grep -v 6642 | grep UNREP > conntrack v1.4.6 (conntrack-tools): 4 flow entries have been shown. > tcp 6 118 SYN_SENT src=172.16.1.50 dst=192.168.10.10 sport=44742 > dport=80 [UNREPLIED] src=192.168.10.10 dst=172.16.1.50 sport=80 > dport=44742 mark=0 zone=13 use=1 > > Even though the packets are replied and NATed, it seems there is > something wrong that is keeping this conntrack entry there. > I was able to check there was a later patch [1] that fixed some issues > with the patch [0] but it did not address this specific behavior. > > [1] > https://github.com/ovn-org/ovn/commit/8114a5c606eb5d0b3f04cf3e8219e094f1cc1b37 > > The patch is creating the way for an external IP to communicate with a > private IP on the overlay without NATing when having DNAT/SNAT > configured on the LR. > > Could you give a hint how we could work to solve this issue? > > If Martin or Ales could check, I would appreciate it. >
The following line is creating the behavior when using true: https://github.com/ovn-org/ovn/blob/main/northd/northd.c#L16516 So, changing it to false solves my problem but I believe it would create an issue for the main purpose of the patch. Checking the logical flows I was able to verify that there is a difference in the direction of the flow on the lr_out_snat stage: #true table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.dst == 192.168.10.0/24 && inport == "lr11-public1" && is_chassis_resident("cr-lr11-public1")), action=(ct_snat;) table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == 192.168.10.0/24 && outport == "lr11-public1" && is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), action=(ct_snat(172.16.1.100);) #false table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == 192.168.10.0/24 && outport == "lr11-public1" && is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), action=(ct_snat(172.16.1.100);) table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == 192.168.10.0/24 && outport == "lr11-public1" && is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), action=(ct_snat;) Regards, Tiago Pires > Thanks. > > Regards, > > Tiago Pires -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
