Hi all, I have a potential issue to report where the use case is using dnat_and_snat and it is keeping many SYN_SENT with UNREPLIED status on the conntrack table. The TCP handshake is working fine, but when a user establishes a connection, there is always a SYN_SENT left on the conntrack table.
I was able to figure out that this patch [0] (Fix direct access to SNAT network) introduced this behavior. [0] https://github.com/ovn-org/ovn/commit/40136a2f2c84dca48f7b67c7dc63759e82a44668 When we have many connections on the chassis gateways, more than 50% of the conntrack entries stay in this UNREPLIED state until the entry is expired according with the nf_conntrack_tcp_timeout_syn_sent setting. How to reproduce: #Setup OS_BASE=ubuntu OS_IMAGE=docker.io/library/ubuntu:jammy ENABLE_SSL=no CENTRAL_COUNT=1 CHASSIS_COUNT=2 GW_COUNT=1 ./ovn_cluster.sh start podman exec -it ovn-gw-1 ip link set dev br-ex up podman exec -it ovn-central-az1-1 ovn-nbctl lr-add lr11 podman exec -it ovn-central-az1-1 ovn-nbctl ls-add bar11 podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 lr11-public1 00:00:00:00:aa:02 172.16.1.100/24 podman exec -it ovn-central-az1-1 ovn-nbctl lrp-set-gateway-chassis lr11-public1 ovn-gw-1 podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add public1 lr11-lsp-public1 podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-type lr11-lsp-public1 router podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-addresses lr11-lsp-public1 router podman exec -it ovn-central-az1-1 ovn-nbctl lsp-set-options lr11-lsp-public1 router-port=lr11-public1 podman exec -it ovn-central-az1-1 ovn-nbctl lr-route-add lr11 0.0.0.0/0 172.16.1.1 podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 dnat_and_snat 172.16.1.101 192.168.10.10 podman exec -it ovn-central-az1-1 ovn-nbctl lr-nat-add lr11 snat 172.16.1.100 192.168.10.0/24 podman exec -it ovn-central-az1-1 ovn-nbctl lrp-add lr11 rp11-bar 00:01:ff:22:00:01 192.168.10.1/24 podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 haproxy1 \ -- lsp-set-addresses haproxy1 "00:00:ff:ff:ff:21 192.168.10.10" podman exec -it ovn-central-az1-1 ovn-nbctl lsp-add bar11 bar-rp11 -- set Logical_Switch_Port bar-rp11 type=router options:router-port=rp11-bar -- lsp-set-addresses bar-rp11 router podman exec -it ovn-chassis-1 ip netns add haproxy1 podman exec -it ovn-chassis-1 ovs-vsctl add-port br-int haproxy1 -- set interface haproxy1 type=internal podman exec -it ovn-chassis-1 ip link set haproxy1 netns haproxy1 podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip addr add 192.168.10.10/24 dev haproxy1 podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set dev haproxy1 mtu 1400 podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set haproxy1 address 00:00:ff:ff:ff:21 podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip link set haproxy1 up podman exec -it ovn-chassis-1 ip netns exec haproxy1 ip route add default via 192.168.10.1 podman exec -it ovn-chassis-1 ovs-vsctl set Interface haproxy1 external_ids:iface-id=haproxy1 podman exec ovn-chassis-1 ip netns exec haproxy1 python3 -m http.server 80 -d /tmp/ >/dev/null 2>&1 & #Test (client IP - 172.16.1.50) ip netns exec ovnfake-ext1 curl http://172.16.1.101 #tcpdump on ovn-gw-1 (eth2) 13:50:31.703106 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [S], seq 2055144661, win 64240, options [mss 1460,sackOK,TS val 2795500537 ecr 0,nop,wscale 7], length 0 13:50:31.705836 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [S.], seq 3211627047, ack 2055144662, win 64704, options [mss 1360,sackOK,TS val 875814075 ecr 2795500537,nop,wscale 7], length 0 13:50:31.705973 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2795500540 ecr 875814075], length 0 13:50:31.706065 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 2795500540 ecr 875814075], length 76: HTTP: GET / HTTP/1.1 13:50:31.706323 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack 77, win 505, options [nop,nop,TS val 875814077 ecr 2795500540], length 0 13:50:31.708484 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], seq 1:187, ack 77, win 505, options [nop,nop,TS val 875814079 ecr 2795500540], length 186: HTTP: HTTP/1.0 200 OK 13:50:31.708494 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack 187, win 501, options [nop,nop,TS val 2795500542 ecr 875814079], length 0 13:50:31.708567 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [P.], seq 187:248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr 2795500542], length 61: HTTP 13:50:31.708571 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [.], ack 248, win 501, options [nop,nop,TS val 2795500543 ecr 875814080], length 0 13:50:31.708605 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [F.], seq 248, ack 77, win 505, options [nop,nop,TS val 875814080 ecr 2795500543], length 0 13:50:31.708673 IP 172.16.1.50.44742 > 172.16.1.101.80: Flags [F.], seq 77, ack 249, win 501, options [nop,nop,TS val 2795500543 ecr 875814080], length 0 13:50:31.708759 IP 172.16.1.101.80 > 172.16.1.50.44742: Flags [.], ack 78, win 505, options [nop,nop,TS val 875814080 ecr 2795500543], length 0 #conntrack UNREPLIED entries on ovn-gw-1 # conntrack -L | grep -v 6081 | grep -v 6642 | grep UNREP conntrack v1.4.6 (conntrack-tools): 4 flow entries have been shown. tcp 6 118 SYN_SENT src=172.16.1.50 dst=192.168.10.10 sport=44742 dport=80 [UNREPLIED] src=192.168.10.10 dst=172.16.1.50 sport=80 dport=44742 mark=0 zone=13 use=1 Even though the packets are replied and NATed, it seems there is something wrong that is keeping this conntrack entry there. I was able to check there was a later patch [1] that fixed some issues with the patch [0] but it did not address this specific behavior. [1] https://github.com/ovn-org/ovn/commit/8114a5c606eb5d0b3f04cf3e8219e094f1cc1b37 The patch is creating the way for an external IP to communicate with a private IP on the overlay without NATing when having DNAT/SNAT configured on the LR. Could you give a hint how we could work to solve this issue? If Martin or Ales could check, I would appreciate it. Thanks. Regards, Tiago Pires -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
