On Wed, Jan 24, 2024 at 6:05 PM Martin Kalcok via discuss
wrote:
>
> Hi everyone,
> We are encountering an issue with our (Openstack) topology where we
> need to directly access hosts in SNATed network from external network.
> While the traffic like ICMP works fine, TCP traffic gets SNATed on the
> way back **after** the initial handshake. Our setup looks something
> like this:
>
> OVN from git (0ce21f2)
> OVS from git (4102674)
>
> 3x Nova Compute hypervisor, each with GW Chassis
>
> External network: 10.5.0./16
> Host on ext. net (S1): 10.5.0.16
> Logical Router (r1) ext. IP:10.5.150.143
> Internal network (a_net) with SNAT: 192.168.10.0/24
> Guest VM (A1) on the int. network: 192.168.10.83
>
> (Host S1 has static routes for the internal network via r1's external
> IP)
>
> I think the problem is best demonstrated on the tcpdump captured on the
> S1 trying to communicate with A1 [0] (I'm going to use pastebin to keep
> this mail shorter, I hope that it's OK). As you can see, after the
> initial handshake, S1 pushes data, but ACK comes back from SNATed
> router's IP.
>
> I also took OVN traces:
>
> * S1 -> A1 (New direction)[1]
> * A1 -> S1 (Reply direction)[2]
>
> The traces themselves look good as they don't show any SNAT in the
> reply direction but I'm not sure that this is what's actually happening
> when the packet goes through the router. I see that the SNAT rules
> should not match for packets in "reply" direction [3] but I don't see
> (in the trace) that the router would apply connection tracking on the
> packets with "ct_next". If I understand this correctly, if the router
> does not commit flow into the connection tracking, it is not able to
> match on ct_state.
>
> I tried to apply following changes that initiate CT and commit packets
> to the conntrack and it seems to fix the issue without breaking
> anything. The gist of the change is that the router initiates CT in the
> ingress and commits the packet into CT before moving it to the egress
> pipeline. That way, when the packets in the reply direction show up,
> router can determine that they are replies and does not perform the
> SNAT.
>
> I very much just eyeballed where to put the "ct_next" and "ct_commit"
> so I'm sure that this is not the final solution, but I'd just like to
> know if I'm going in the right direction.
>
> diff --git a/northd/northd.c b/northd/northd.c
> index 952f8200d..4eea38cbd 100644
> --- a/northd/northd.c
> +++ b/northd/northd.c
> @@ -11792,7 +11792,7 @@ add_route(struct hmap *lflows, struct
> ovn_datapath *od,
>"eth.src = %s; "
>"outport = %s; "
>"flags.loopback = 1; "
> - "next;",
> + "ct_next;",
>is_ipv4 ? REG_SRC_IPV4 : REG_SRC_IPV6,
>lrp_addr_s,
>op->lrp_networks.ea_s,
> @@ -14331,7 +14331,7 @@ build_arp_request_flows_for_lrouter(
>"};",
>copp_meter_get(COPP_ND_NS_RESOLVE, od->nbr-
> >copp,
> meter_groups));
> -ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_REQUEST, 0, "1",
> "output;");
> +ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_REQUEST, 0, "1",
> "ct_commit { }; output;");
> }
>
> /* Logical router egress table DELIVERY: Delivery (priority 100-110).
> @@ -15692,7 +15692,7 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> - REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> + REGBIT_DST_NAT_IP_LOCAL" = 0; ct_next;");
> ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
>
>
> Thank you for any suggestions,
> Martin.
>
> -
> [0] https://pastebin.com/gmbNP1Bg
> [1] https://pastebin.com/155aVzYj
> [2] https://pastebin.com/pcAuCFhT
> [3]
> https://github.com/ovn-org/ovn/blob/0ce21f2adda1edeeafe10a1d62cd976039a42492/northd/northd.c#L15418
Thanks alot for starting this discussion, Martin!
I just wanted to chime in with the desire to see this use case working
and I know we have examples of end users coming from ML2/OVS with this
expectation. There are also other facets of this problem complex with
issues such as internal IP access to a DNAT'ed address backed by an IP
on the same network etc. but I'm sure we will get onto those as part
of the discussion.
FWIW; there are some descriptions of the handling of this use case in
OpenStack documentation and specs, so I thought it would be useful to
reference that here [4] to support the discussion.
4:
