Hi all, On Thu, Jul 31, 2025 at 6:27 PM Tiago Pires <[email protected]> wrote: > > Hi all, > > On Thu, Jul 31, 2025 at 11:49 AM Tiago Pires <[email protected]> > wrote: > > > > Hi all, > > > > I have a potential issue to report where the use case is using > > dnat_and_snat and it is keeping many SYN_SENT with UNREPLIED status on > > the conntrack table. > > The TCP handshake is working fine, but when a user establishes a > > connection, there is always a SYN_SENT left on the conntrack table. > > > > I was able to figure out that this patch [0] (Fix direct access to > > SNAT network) introduced this behavior. > > > > [0] > > https://github.com/ovn-org/ovn/commit/40136a2f2c84dca48f7b67c7dc63759e82a44668 > > > > When we have many connections on the chassis gateways, more than 50% > > of the conntrack entries stay in this UNREPLIED state until the entry > > is expired according with the nf_conntrack_tcp_timeout_syn_sent > > setting. > > > > How to reproduce: > > > > #Setup > > OS_BASE=ubuntu OS_IMAGE=docker.io/library/ubuntu:jammy ENABLE_SSL=no > > CENTRAL_COUNT=1 CHASSIS_COUNT=2 GW_COUNT=1 ./ovn_cluster.sh start > > podman exec -it ovn-gw-1 ip link set dev br-ex up > > podman exec -it ovn-central-az1-1 ovn-nbctl lr-add lr11 > > podman exec -it ovn-central-az1-1 ovn-nbctl ls-add bar11 > > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 lr11-public1 > > 00:00:00:00:aa:02 172.16.1.100/24 > > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-set-gateway-chassis > > lr11-public1 ovn-gw-1 > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add public1 lr11-lsp-public1 > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-type lr11-lsp-public1 > > router > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-addresses > > lr11-lsp-public1 router > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-options > > lr11-lsp-public1 router-port=lr11-public1 > > podman exec -it ovn-central-az1-1 ovn-nbctl lr-route-add lr11 > > 0.0.0.0/0 172.16.1.1 > > podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 > > dnat_and_snat 172.16.1.101 192.168.10.10 > > podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 snat > > 172.16.1.100 192.168.10.0/24 > > podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 rp11-bar > > 00:01:ff:22:00:01 192.168.10.1/24 > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 haproxy1 \ > > -- lsp-set-addresses haproxy1 "00:00:ff:ff:ff:21 192.168.10.10" > > podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 bar-rp11 -- > > set Logical_Switch_Port bar-rp11 type=router > > options:router-port=rp11-bar -- lsp-set-addresses bar-rp11 router > > podman exec -it ovn-chassis-1 ip netns add haproxy1 > > podman exec -it ovn-chassis-1 ovs-vsctl add-port br-int haproxy1 -- > > set interface haproxy1 type=internal > > podman exec -it ovn-chassis-1 ip link set haproxy1 netns haproxy1 > > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip addr add > > 192.168.10.10/24 dev haproxy1 > > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set dev > > haproxy1 mtu 1400 > > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set > > haproxy1 address 00:00:ff:ff:ff:21 > > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set haproxy1 up > > podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip route add > > default via 192.168.10.1 > > podman exec -it ovn-chassis-1 ovs-vsctl set Interface haproxy1 > > external_ids:iface-id=haproxy1 > > podman exec ovn-chassis-1 ip netns exec haproxy1 python3 -m > > http.server 80 -d /tmp/ >/dev/null 2>&1 & > > > > #Test (client IP - 172.16.1.50) > > ip netns exec ovnfake-ext1 curl http://172.16.1.101 > > > > #tcpdump on ovn-gw-1 (eth2) > > > > 13:50:31.703106 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [S], seq > > 2055144661, win 64240, options [mss 1460,sackOK,TS val 2795500537 ecr > > 0,nop,wscale 7], length 0 > > 13:50:31.705836 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [S.], > > seq 3211627047, ack 2055144662, win 64704, options [mss 1360,sackOK,TS > > val 875814075 ecr 2795500537,nop,wscale 7], length 0 > > 13:50:31.705973 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > > 1, win 502, options [nop,nop,TS val 2795500540 ecr 875814075], length > > 0 > > 13:50:31.706065 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [P.], > > seq 1:77, ack 1, win 502, options [nop,nop,TS val 2795500540 ecr > > 875814075], length 76: HTTP: GET / HTTP/1.1 > > 13:50:31.706323 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack > > 77, win 505, options [nop,nop,TS val 875814077 ecr 2795500540], length > > 0 > > 13:50:31.708484 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], > > seq 1:187, ack 77, win 505, options [nop,nop,TS val 875814079 ecr > > 2795500540], length 186: HTTP: HTTP/1.0 200 OK > > 13:50:31.708494 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > > 187, win 501, options [nop,nop,TS val 2795500542 ecr 875814079], > > length 0 > > 13:50:31.708567 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], > > seq 187:248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr > > 2795500542], length 61: HTTP > > 13:50:31.708571 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack > > 248, win 501, options [nop,nop,TS val 2795500543 ecr 875814080], > > length 0 > > 13:50:31.708605 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [F.], > > seq 248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr > > 2795500543], length 0 > > 13:50:31.708673 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [F.], > > seq 77, ack 249, win 501, options [nop,nop,TS val 2795500543 ecr > > 875814080], length 0 > > 13:50:31.708759 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack > > 78, win 505, options [nop,nop,TS val 875814080 ecr 2795500543], length > > 0 > > > > > > #conntrack UNREPLIED entries on ovn-gw-1 > > # conntrack -L | grep -v 6081 | grep -v 6642 | grep UNREP > > conntrack v1.4.6 (conntrack-tools): 4 flow entries have been shown. > > tcp 6 118 SYN_SENT src=172.16.1.50 dst=192.168.10.10 sport=44742 > > dport=80 [UNREPLIED] src=192.168.10.10 dst=172.16.1.50 sport=80 > > dport=44742 mark=0 zone=13 use=1 > > > > Even though the packets are replied and NATed, it seems there is > > something wrong that is keeping this conntrack entry there. > > I was able to check there was a later patch [1] that fixed some issues > > with the patch [0] but it did not address this specific behavior. > > > > [1] > > https://github.com/ovn-org/ovn/commit/8114a5c606eb5d0b3f04cf3e8219e094f1cc1b37 > > > > The patch is creating the way for an external IP to communicate with a > > private IP on the overlay without NATing when having DNAT/SNAT > > configured on the LR. > > > > Could you give a hint how we could work to solve this issue? > > > > If Martin or Ales could check, I would appreciate it. > > > > The following line is creating the behavior when using true: > > https://github.com/ovn-org/ovn/blob/main/northd/northd.c#L16516 > > So, changing it to false solves my problem but I believe it would > create an issue for the main purpose of the patch. > Checking the logical flows I was able to verify that there is a > difference in the direction of the flow on the lr_out_snat stage: > > #true > table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.dst == > 192.168.10.0/24 && inport == "lr11-public1" && > is_chassis_resident("cr-lr11-public1")), action=(ct_snat;) > table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == > 192.168.10.0/24 && outport == "lr11-public1" && > is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), > action=(ct_snat(172.16.1.100);) > > #false > table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == > 192.168.10.0/24 && outport == "lr11-public1" && > is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), > action=(ct_snat(172.16.1.100);) > table=3 (lr_out_snat ), priority=153 , match=(ip && ip4.src == > 192.168.10.0/24 && outport == "lr11-public1" && > is_chassis_resident("cr-lr11-public1") && (!ct.trk || !ct.rpl)), > action=(ct_snat;) >
I was able to fix the behavior regarding my conntrack's issue and keep the traffic initiated from an external network going directly into a network configured with SNAT. I will submit a patch for it. Regards, Tiago pires > Regards, > > Tiago Pires > > > > Thanks. > > > > Regards, > > > > Tiago Pires -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
