On 23/06/2021 03:00, Han Zhou wrote: > On Sat, Jun 19, 2021 at 2:52 AM Mark Gray <mark.d.g...@redhat.com> wrote: >> >> This patch addresses a number of interconnected issues with Gateway > Routers >> that have Load Balancing enabled: >> >> 1) In the router pipeline, we have the following stages to handle >> dnat and unsnat. >> >> - Stage 4 : lr_in_defrag (dnat zone) >> - Stage 5 : lr_in_unsnat (snat zone) >> - Stage 6 : lr_in_dnat (dnat zone) >> >> In the reply direction, the order of traversal of the tables >> "lr_in_defrag", "lr_in_unsnat" and "lr_in_dnat" adds incorrect >> datapath flows that check ct_state in the wrong conntrack zone. >> This is illustrated below where reply trafic enters the physical host >> port (6) and traverses DNAT zone (14), SNAT zone (default), back to the >> DNAT zone and then on to Logical Switch Port zone (22). The third >> flow is incorrectly checking the state from the SNAT zone instead >> of the DNAT zone. >> >> recirc_id(0),in_port(6),ct_state(-new-est-rel-rpl-trk) > actions:ct_clear,ct(zone=14),recirc(0xf) >> recirc_id(0xf),in_port(6) actions:ct(nat),recirc(0x10) >> recirc_id(0x10),in_port(6),ct_state(-new+est+trk) > actions:ct(zone=14,nat),recirc(0x11) >> recirc_id(0x11),in_port(6),ct_state(+new-est-rel-rpl+trk) actions: > ct(zone=22,nat),recirc(0x12) >> recirc_id(0x12),in_port(6),ct_state(-new+est-rel+rpl+trk) actions:5 >> >> Update the order of these tables to resolve this. >> >> 2) Efficiencies can be gained by using the ct_dnat action in the >> table "lr_in_defrag" instead of ct_next. This removes the need for the >> ct_dnat action for established Load Balancer flows avoiding a >> recirculation. >> >> 3) On a Gateway router with DNAT flows configured, the router will > translate >> the destination IP address from (A) to (B). Reply packets from (B) are >> correctly UNDNATed in the reverse direction. >> >> However, if a new connection is established from (B), this flow is never >> committed to conntrack and, as such, is never established. This will >> cause OVS datapath flows to be added that match on the ct.new flag. >> >> For software-only datapaths this is not a problem. However, for >> datapaths that offload these flows to hardware, this may be problematic >> as some devices are unable to offload flows that match on ct.new. >> >> This patch resolves this by committing these flows to the DNAT zone in >> the new "lr_out_post_undnat" stage. Although this could be done in the >> DNAT zone, by doing this in the new zone we can avoid a recirculation. >> >> Co-authored-by: Numan Siddique <num...@ovn.org> >> Signed-off-by: Mark Gray <mark.d.g...@redhat.com> >> Signed-off-by: Numan Siddique <num...@ovn.org> > > Thanks Mark and Numan. Please see some non-critical comments inlined. > >> --- >> northd/ovn-northd.8.xml | 143 +++++---- >> northd/ovn-northd.c | 111 +++++-- >> northd/ovn_northd.dl | 113 +++++-- >> tests/ovn-northd.at | 674 +++++++++++++++++++++++++++++++++++----- >> tests/ovn.at | 6 +- >> tests/system-ovn.at | 13 +- >> 6 files changed, 853 insertions(+), 207 deletions(-) >> >> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml >> index 4074646029b4..d56a121d4d2e 100644 >> --- a/northd/ovn-northd.8.xml >> +++ b/northd/ovn-northd.8.xml >> @@ -2628,39 +2628,9 @@ icmp6 { >> </li> >> </ul> >> >> - <h3>Ingress Table 4: DEFRAG</h3> >> >> - <p> >> - This is to send packets to connection tracker for tracking and >> - defragmentation. It contains a priority-0 flow that simply moves > traffic >> - to the next table. >> - </p> >> - >> - <p> >> - If load balancing rules with virtual IP addresses (and ports) are >> - configured in <code>OVN_Northbound</code> database for a Gateway > router, >> - a priority-100 flow is added for each configured virtual IP address >> - <var>VIP</var>. For IPv4 <var>VIPs</var> the flow matches <code>ip >> - && ip4.dst == <var>VIP</var></code>. For IPv6 > <var>VIPs</var>, >> - the flow matches <code>ip && ip6.dst == > <var>VIP</var></code>. >> - The flow uses the action <code>ct_next;</code> to send IP packets > to the >> - connection tracker for packet de-fragmentation and tracking before >> - sending it to the next table. >> - </p> >> - >> - <p> >> - If ECMP routes with symmetric reply are configured in the >> - <code>OVN_Northbound</code> database for a gateway router, a > priority-300 >> - flow is added for each router port on which symmetric replies are >> - configured. The matching logic for these ports essentially > reverses the >> - configured logic of the ECMP route. So for instance, a route with a >> - destination routing policy will instead match if the source IP > address >> - matches the static route's prefix. The flow uses the action >> - <code>ct_next</code> to send IP packets to the connection tracker > for >> - packet de-fragmentation and tracking before sending it to the next > table. >> - </p> >> >> - <h3>Ingress Table 5: UNSNAT</h3> >> + <h3>Ingress Table 4: UNSNAT</h3> >> >> <p> >> This is for already established connections' reverse traffic. >> @@ -2669,7 +2639,7 @@ icmp6 { >> unSNATted here. >> </p> >> >> - <p>Ingress Table 5: UNSNAT on Gateway and Distributed Routers</p> >> + <p>Ingress Table 4: UNSNAT on Gateway and Distributed Routers</p> >> <ul> >> <li> >> <p> >> @@ -2696,7 +2666,7 @@ icmp6 { >> </li> >> </ul> >> >> - <p>Ingress Table 5: UNSNAT on Gateway Routers</p> >> + <p>Ingress Table 4: UNSNAT on Gateway Routers</p> >> >> <ul> >> <li> >> @@ -2713,9 +2683,10 @@ icmp6 { >> <code>lb_force_snat_ip=router_ip</code> then for every logical > router >> port <var>P</var> attached to the Gateway router with the > router ip >> <var>B</var>, a priority-110 flow is added with the match >> - <code>inport == <var>P</var> && ip4.dst == > <var>B</var></code> or >> - <code>inport == <var>P</var> && ip6.dst == > <var>B</var></code> >> - with an action <code>ct_snat; </code>. >> + <code>inport == <var>P</var> && >> + ip4.dst == <var>B</var></code> or <code>inport == <var>P</var> >> + && ip6.dst == <var>B</var></code> with an action >> + <code>ct_snat; </code>. >> </p> >> >> <p> >> @@ -2745,7 +2716,7 @@ icmp6 { >> </li> >> </ul> >> >> - <p>Ingress Table 5: UNSNAT on Distributed Routers</p> >> + <p>Ingress Table 4: UNSNAT on Distributed Routers</p> >> >> <ul> >> <li> >> @@ -2776,6 +2747,40 @@ icmp6 { >> </li> >> </ul> >> >> + <h3>Ingress Table 5: DEFRAG</h3> >> + >> + <p> >> + This is to send packets to connection tracker for tracking and >> + defragmentation. It contains a priority-0 flow that simply moves > traffic >> + to the next table. >> + </p> >> + >> + <p> >> + If load balancing rules with virtual IP addresses (and ports) are >> + configured in <code>OVN_Northbound</code> database for a Gateway > router, >> + a priority-100 flow is added for each configured virtual IP address >> + <var>VIP</var>. For IPv4 <var>VIPs</var> the flow matches <code>ip >> + && ip4.dst == <var>VIP</var></code>. For IPv6 > <var>VIPs</var>, >> + the flow matches <code>ip && ip6.dst == > <var>VIP</var></code>. >> + The flow applies the action <code>reg0 = <var>VIP</var> > > nit: for IPv6 it is xxreg0. > >> + && ct_dnat;</code> to send IP packets to the >> + connection tracker for packet de-fragmentation and to dnat the >> + destination IP for the committed connection before sending it to > the >> + next table. >> + </p> >> + >> + <p> >> + If ECMP routes with symmetric reply are configured in the >> + <code>OVN_Northbound</code> database for a gateway router, a > priority-300 >> + flow is added for each router port on which symmetric replies are >> + configured. The matching logic for these ports essentially > reverses the >> + configured logic of the ECMP route. So for instance, a route with a >> + destination routing policy will instead match if the source IP > address >> + matches the static route's prefix. The flow uses the action >> + <code>ct_next</code> to send IP packets to the connection tracker > for >> + packet de-fragmentation and tracking before sending it to the next > table. >> + </p> >> + >> <h3>Ingress Table 6: DNAT</h3> >> >> <p> >> @@ -2828,19 +2833,28 @@ icmp6 { >> </li> >> >> <li> >> - For all the configured load balancing rules for a router in >> - <code>OVN_Northbound</code> database that includes a L4 port >> - <var>PORT</var> of protocol <var>P</var> and IPv4 or IPv6 address >> - <var>VIP</var>, a priority-120 flow that matches on >> - <code>ct.est && ip && ip4.dst == <var>VIP</var> >> - && <var>P</var> && <var>P</var>.dst == <var>PORT >> - </var></code> (<code>ip6.dst == <var>VIP</var></code> in the > IPv6 case) >> - with an action of <code>ct_dnat;</code>. If the router is >> - configured to force SNAT any load-balanced packets, the above > action >> - will be replaced by <code>flags.force_snat_for_lb = 1; > ct_dnat;</code>. >> - If the load balancing rule is configured with > <code>skip_snat</code> >> - set to true, the above action will be replaced by >> - <code>flags.skip_snat_for_lb = 1; ct_dnat;</code>. >> + <p> >> + For all the configured load balancing rules for a router in >> + <code>OVN_Northbound</code> database that includes a L4 port >> + <var>PORT</var> of protocol <var>P</var> and IPv4 or IPv6 > address >> + <var>VIP</var>, a priority-120 flow that matches on >> + <code>ct.est && ip && reg0 == <var>VIP</var> >> + && <var>P</var> && <var>P</var>.dst == > <var>PORT >> + </var></code> (<code>xxreg0 == <var>VIP</var></code> in the >> + IPv6 case) with an action of <code>next;</code>. If the router > is >> + configured to force SNAT any load-balanced packets, the above > action >> + will be replaced by <code>flags.force_snat_for_lb = 1; > next;</code>. >> + If the load balancing rule is configured with > <code>skip_snat</code> >> + set to true, the above action will be replaced by >> + <code>flags.skip_snat_for_lb = 1; next;</code>. >> + </p> >> + >> + <p> >> + Previous table <code>lr_in_defrag</code> sets the register >> + <code>reg0</code> (or <code>xxreg0</code> for IPv6) and does >> + <code>ct_dnat</code>. Hence for established traffic, this >> + table just advances the packet to the next stage. >> + </p> >> </li> >> > In this section "Ingress Table 6: DNAT", there are several paragraphs with > very similar text but only minor differences on the match condition, such > as "includes just an IP address ...", and you need to update for all those > paragraphs. I understand it is painful to update the redundant information, > and it may be also painful for the readers, but it seems even worse if it > is incorrect. Not sure if there is a better way to maintain this document. > (the DDlog code is easier to understand than the document)
Sorry, this appears to be an unintentional omission. I will update this section. It would be good if we found an easier way to keep this up-to-date, and as you state, keep it correct. It is a key resource for debugging. Unfortunately the code isn't particularly self-documenting :(. I guess its something to have a think about for the future .. > >> <li> >> @@ -3876,7 +3890,26 @@ nd_ns { >> </li> >> </ul> >> >> - <h3>Egress Table 1: SNAT</h3> >> + <h3>Egress Table 1: Post UNDNAT on Gateway Routers</h3> >> + >> + <p> >> + <ul> >> + <li> >> + A priority-50 logical flow is added that commits any untracked > flows >> + from the previous table <code>lr_out_undnat</code>. This flow >> + matches on <code>ct.new && ip</code> with action >> + <code>ct_commit { } ; next; </code>. >> + </li> >> + >> + <li> >> + A priority-0 logical flow with match <code>1</code> has actions >> + <code>next;</code>. >> + </li> >> + >> + </ul> >> + </p> >> + >> + <h3>Egress Table 2: SNAT</h3> >> >> <p> >> Packets that are configured to be SNATed get their source IP > address >> @@ -3892,7 +3925,7 @@ nd_ns { >> </li> >> </ul> >> >> - <p>Egress Table 1: SNAT on Gateway Routers</p> >> + <p>Egress Table 2: SNAT on Gateway Routers</p> >> >> <ul> >> <li> >> @@ -3991,7 +4024,7 @@ nd_ns { >> </li> >> </ul> >> >> - <p>Egress Table 1: SNAT on Distributed Routers</p> >> + <p>Egress Table 2: SNAT on Distributed Routers</p> >> >> <ul> >> <li> >> @@ -4051,7 +4084,7 @@ nd_ns { >> </li> >> </ul> >> >> - <h3>Egress Table 2: Egress Loopback</h3> >> + <h3>Egress Table 3: Egress Loopback</h3> >> >> <p> >> For distributed logical routers where one of the logical router >> @@ -4120,7 +4153,7 @@ clone { >> </li> >> </ul> >> >> - <h3>Egress Table 3: Delivery</h3> >> + <h3>Egress Table 4: Delivery</h3> >> >> <p> >> Packets that reach this table are ready for delivery. It contains: >> diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c >> index d97ab4a5b39c..27e5fbea9f4f 100644 >> --- a/northd/ovn-northd.c >> +++ b/northd/ovn-northd.c >> @@ -187,8 +187,8 @@ enum ovn_stage { >> PIPELINE_STAGE(ROUTER, IN, LOOKUP_NEIGHBOR, 1, > "lr_in_lookup_neighbor") \ >> PIPELINE_STAGE(ROUTER, IN, LEARN_NEIGHBOR, 2, > "lr_in_learn_neighbor") \ >> PIPELINE_STAGE(ROUTER, IN, IP_INPUT, 3, "lr_in_ip_input") > \ >> - PIPELINE_STAGE(ROUTER, IN, DEFRAG, 4, "lr_in_defrag") > \ >> - PIPELINE_STAGE(ROUTER, IN, UNSNAT, 5, "lr_in_unsnat") > \ >> + PIPELINE_STAGE(ROUTER, IN, UNSNAT, 4, "lr_in_unsnat") > \ >> + PIPELINE_STAGE(ROUTER, IN, DEFRAG, 5, "lr_in_defrag") > \ >> PIPELINE_STAGE(ROUTER, IN, DNAT, 6, "lr_in_dnat") > \ >> PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, > "lr_in_ecmp_stateful") \ >> PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, > "lr_in_nd_ra_options") \ >> @@ -204,10 +204,11 @@ enum ovn_stage { >> PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, > "lr_in_arp_request") \ >> \ >> /* Logical router egress stages. */ \ >> - PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ >> - PIPELINE_STAGE(ROUTER, OUT, SNAT, 1, "lr_out_snat") \ >> - PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP, 2, "lr_out_egr_loop") \ >> - PIPELINE_STAGE(ROUTER, OUT, DELIVERY, 3, "lr_out_delivery") >> + PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ >> + PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat") \ >> + PIPELINE_STAGE(ROUTER, OUT, SNAT, 2, "lr_out_snat") \ >> + PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP, 3, "lr_out_egr_loop") \ >> + PIPELINE_STAGE(ROUTER, OUT, DELIVERY, 4, "lr_out_delivery") >> >> #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME) \ >> S_##DP_TYPE##_##PIPELINE##_##STAGE \ >> @@ -643,6 +644,12 @@ struct ovn_datapath { >> /* Multicast data. */ >> struct mcast_info mcast_info; >> >> + /* Applies to only logical router datapath. >> + * True if logical router is a gateway router. i.e options:chassis > is set. >> + * If this is true, then 'l3dgw_port' and 'l3redirect_port' will be >> + * ignored. */ >> + bool is_gw_router; >> + > > This looks redundant with the field l3dgw_port. The newly added bool is > used only once, and the other places are still using !l3dgw_port to check > if it is GW router. I suggest either keep using !l3dgw_port or unifying all > the checks to use the new bool. Numans updated this to use 'is_gw_router' everywhere > >> /* OVN northd only needs to know about the logical router gateway > port for >> * NAT on a distributed router. This "distributed gateway port" is >> * populated only when there is a gateway chassis specified for one > of >> @@ -1247,6 +1254,9 @@ join_datapaths(struct northd_context *ctx, struct > hmap *datapaths, >> init_mcast_info_for_datapath(od); >> init_nat_entries(od); >> init_lb_ips(od); >> + if (smap_get(&od->nbr->options, "chassis")) { >> + od->is_gw_router = true; >> + } >> ovs_list_push_back(lr_list, &od->lr_list); >> } >> } >> @@ -8731,20 +8741,33 @@ add_router_lb_flow(struct hmap *lflows, struct > ovn_datapath *od, >> } >> >> /* A match and actions for established connections. */ >> - char *est_match = xasprintf("ct.est && %s", ds_cstr(match)); >> + struct ds est_match = DS_EMPTY_INITIALIZER; >> + ds_put_format(&est_match, >> + "ct.est && ip && %sreg0 == %s && ct_label.natted == 1", >> + IN6_IS_ADDR_V4MAPPED(&lb_vip->vip) ? "" : "xx", >> + lb_vip->vip_str); >> + if (lb_vip->vip_port) { >> + ds_put_format(&est_match, " && %s", proto); >> + } >> + if (od->l3redirect_port && >> + (lb_vip->n_backends || !lb_vip->empty_backend_rej)) { >> + ds_put_format(&est_match, " && is_chassis_resident(%s)", >> + od->l3redirect_port->json_key); >> + } > > This part is a little redundant with constructing the "match" in the caller > of this function. The only difference is matching ip.dst or "reg0/xxreg0". > It is not anything critical but it may be better to put the logic at the > same place, maybe just move the logic from the caller to this function. I have refactored this as you suggested, I think it looks marginally cleaner. Although this whole section of the code could do with a larger refactor IMO. A task for another day ... > >> if (snat_type == FORCE_SNAT || snat_type == SKIP_SNAT) { >> - char *est_actions = xasprintf("flags.%s_snat_for_lb = 1; > ct_dnat;", >> + char *est_actions = xasprintf("flags.%s_snat_for_lb = 1; next;", >> snat_type == SKIP_SNAT ? "skip" : "force"); >> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, >> - est_match, est_actions, &lb->header_); >> + ds_cstr(&est_match), est_actions, >> + &lb->header_); >> free(est_actions); >> } else { >> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, >> - est_match, "ct_dnat;", &lb->header_); >> + ds_cstr(&est_match), "next;", > &lb->header_); >> } >> >> free(new_match); >> - free(est_match); >> + ds_destroy(&est_match); >> >> const char *ip_match = NULL; >> if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) { >> @@ -8829,8 +8852,8 @@ add_router_lb_flow(struct hmap *lflows, struct > ovn_datapath *od, >> static void >> build_lrouter_lb_flows(struct hmap *lflows, struct ovn_datapath *od, >> struct hmap *lbs, struct shash *meter_groups, >> - struct sset *nat_entries, struct ds *match, >> - struct ds *actions) >> + struct sset *nat_entries, >> + struct ds *match, struct ds *actions) >> { >> /* A set to hold all ips that need defragmentation and tracking. */ >> struct sset all_ips = SSET_INITIALIZER(&all_ips); >> @@ -8852,10 +8875,17 @@ build_lrouter_lb_flows(struct hmap *lflows, > struct ovn_datapath *od, >> for (size_t j = 0; j < lb->n_vips; j++) { >> struct ovn_lb_vip *lb_vip = &lb->vips[j]; >> struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[j]; >> + >> + bool is_udp = nullable_string_is_equal(nb_lb->protocol, > "udp"); >> + bool is_sctp = nullable_string_is_equal(nb_lb->protocol, >> + "sctp"); >> + const char *proto = is_udp ? "udp" : is_sctp ? "sctp" : > "tcp"; >> + >> ds_clear(actions); >> build_lb_vip_actions(lb_vip, lb_vip_nb, actions, >> lb->selection_fields, false); >> >> + struct ds defrag_actions = DS_EMPTY_INITIALIZER; >> if (!sset_contains(&all_ips, lb_vip->vip_str)) { >> sset_add(&all_ips, lb_vip->vip_str); >> /* If there are any load balancing rules, we should send >> @@ -8867,17 +8897,28 @@ build_lrouter_lb_flows(struct hmap *lflows, > struct ovn_datapath *od, >> * 2. If there are L4 ports in load balancing rules, we >> * need the defragmentation to match on L4 ports. */ >> ds_clear(match); >> + ds_clear(&defrag_actions); >> if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) { >> ds_put_format(match, "ip && ip4.dst == %s", >> lb_vip->vip_str); >> + ds_put_format(&defrag_actions, "reg0 = %s; ct_dnat;", >> + lb_vip->vip_str); >> } else { >> ds_put_format(match, "ip && ip6.dst == %s", >> lb_vip->vip_str); >> + ds_put_format(&defrag_actions, "xxreg0 = %s; > ct_dnat;", >> + lb_vip->vip_str); >> + } >> + >> + if (lb_vip->vip_port) { >> + ds_put_format(match, " && %s", proto); >> } >> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DEFRAG, >> - 100, ds_cstr(match), "ct_next;", >> + 100, ds_cstr(match), >> + ds_cstr(&defrag_actions), >> &nb_lb->header_); >> } >> + ds_destroy(&defrag_actions); >> >> /* Higher priority rules are added for load-balancing in DNAT >> * table. For every match (on a VIP[:port]), we add two > flows >> @@ -8886,18 +8927,14 @@ build_lrouter_lb_flows(struct hmap *lflows, > struct ovn_datapath *od, >> * flow is for ct.est with an action of "ct_dnat;". */ >> ds_clear(match); >> if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) { >> - ds_put_format(match, "ip && ip4.dst == %s", >> + ds_put_format(match, "ip && reg0 == %s", >> lb_vip->vip_str); >> } else { >> - ds_put_format(match, "ip && ip6.dst == %s", >> + ds_put_format(match, "ip && xxreg0 == %s", >> lb_vip->vip_str); >> } >> >> int prio = 110; >> - bool is_udp = nullable_string_is_equal(nb_lb->protocol, > "udp"); >> - bool is_sctp = nullable_string_is_equal(nb_lb->protocol, >> - "sctp"); >> - const char *proto = is_udp ? "udp" : is_sctp ? "sctp" : > "tcp"; >> >> if (lb_vip->vip_port) { >> ds_put_format(match, " && %s && %s.dst == %d", proto, >> @@ -11400,8 +11437,7 @@ build_lrouter_out_undnat_flow(struct hmap > *lflows, struct ovn_datapath *od, >> * part of a reply. We undo the DNAT here. >> * >> * Note that this only applies for NAT on a distributed router. >> - * Undo DNAT on a gateway router is done in the ingress DNAT >> - * pipeline stage. */ >> + */ >> if (!od->l3dgw_port || >> (strcmp(nat->type, "dnat") && strcmp(nat->type, > "dnat_and_snat"))) { >> return; >> @@ -11681,9 +11717,22 @@ build_lrouter_nat_defrag_and_lb(struct > ovn_datapath *od, >> ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;"); >> ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;"); >> ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;"); >> + ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;"); >> ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;"); >> ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", > "next;"); >> >> + /* For Gateway routers, if the gateway router has load balancer or > DNAT >> + * rules, we commit newly initiated connections in the reply > direction >> + * to the DNAT zone. This ensures that these flows are tracked. If > the flow >> + * was not committed, it would produce ongoing datapath flows with > the >> + * ct.new flag set. Some NICs are unable to offload these flows. >> + */ >> + if (od->is_gw_router && >> + (od->nbr->n_nat || od->nbr->n_load_balancer)) { >> + ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50, >> + "ip && ct.new", "ct_commit { } ; next; "); >> + } >> + > > Why do the commit for GW routers ONLY? I think it may be better to keep the > behavior consistent for both distributed routers and GW routers. I > understand that distributed routers has flows that match each individual > backends IP [&& port], so there are less chances to go through this stage, > but it is still possible if a backend initiates a connection to external > (when L4 port is not specified for the VIP, or the source port happens to > be the VIP's port). And it seems not harmful to add the flow for > distributed routers if only very few connections would hit this flow. > Ok - makes sense. Updated > Moreover, I think it is better to keep the behavior consistent between the > two types of routers also regarding the individual backends IP [&& port] > check. There were control plane scale concerns discussed. However, from > what I observed with DP group feature enabled, the impact should be > minimal. So I'd suggest introducing the same checks for GW router datapaths > before going through the UNDNAT stage, to avoid the extra recirculation for > most use cases. It would be great to have a knob to turn it off when there > is scale concerns. However, please feel free to add it as a follow-up patch > if it is ok. > >> /* Send the IPv6 NS packets to next table. When ovn-controller >> * generates IPv6 NS (for the action - nd_ns{}), the injected >> * packet would go through conntrack - which is not required. */ >> @@ -11848,18 +11897,12 @@ build_lrouter_nat_defrag_and_lb(struct > ovn_datapath *od, >> od->lb_force_snat_addrs.ipv6_addrs[0].addr_s, "lb"); >> } >> } >> - >> - /* For gateway router, re-circulate every packet through >> - * the DNAT zone. This helps with the following. >> - * >> - * Any packet that needs to be unDNATed in the reverse >> - * direction gets unDNATed. Ideally this could be done in >> - * the egress pipeline. But since the gateway router >> - * does not have any feature that depends on the source >> - * ip address being external IP address for IP routing, >> - * we can do it here, saving a future re-circulation. */ >> - ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 50, >> - "ip", "flags.loopback = 1; ct_dnat;"); >> + /* For gateway router, re-circulate every packet through the > DNAT zone >> + * so that packets that need to be unDNATed in the reverse > direction >> + * get unDNATed. >> + */ >> + ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50, >> + "ip", "flags.loopback = 1; ct_dnat;"); > > This change looks reasonable to me, because I don't understand the comment > above regarding how doing the UNDNAT in the ROUTER_IN_DNAT stage would save > a recirculation. > Could you update this part in the ovn-northd.xml document to reflect this > change as well? It is still mentioned in the document "For Gateway routers, > the unDNAT processing is carried out in the ingress DNAT table." Done > > Thanks, > Han > >> } >> >> /* Load balancing and packet defrag are only valid on >> diff --git a/northd/ovn_northd.dl b/northd/ovn_northd.dl >> index 3afa80a3b549..727641ac6ae1 100644 >> --- a/northd/ovn_northd.dl >> +++ b/northd/ovn_northd.dl >> @@ -1454,8 +1454,8 @@ function s_ROUTER_IN_ADMISSION(): Stage { > Stage{Ingress, 0, "lr_in_admiss >> function s_ROUTER_IN_LOOKUP_NEIGHBOR(): Stage { Stage{Ingress, 1, > "lr_in_lookup_neighbor"} } >> function s_ROUTER_IN_LEARN_NEIGHBOR(): Stage { Stage{Ingress, 2, > "lr_in_learn_neighbor"} } >> function s_ROUTER_IN_IP_INPUT(): Stage { Stage{Ingress, 3, > "lr_in_ip_input"} } >> -function s_ROUTER_IN_DEFRAG(): Stage { Stage{Ingress, 4, > "lr_in_defrag"} } >> -function s_ROUTER_IN_UNSNAT(): Stage { Stage{Ingress, 5, > "lr_in_unsnat"} } >> +function s_ROUTER_IN_UNSNAT(): Stage { Stage{Ingress, 4, > "lr_in_unsnat"} } >> +function s_ROUTER_IN_DEFRAG(): Stage { Stage{Ingress, 5, > "lr_in_defrag"} } >> function s_ROUTER_IN_DNAT(): Stage { Stage{Ingress, 6, > "lr_in_dnat"} } >> function s_ROUTER_IN_ECMP_STATEFUL(): Stage { Stage{Ingress, 7, > "lr_in_ecmp_stateful"} } >> function s_ROUTER_IN_ND_RA_OPTIONS(): Stage { Stage{Ingress, 8, > "lr_in_nd_ra_options"} } >> @@ -1472,9 +1472,10 @@ function s_ROUTER_IN_ARP_REQUEST(): Stage { > Stage{Ingress, 18, "lr_in_arp_re >> >> /* Logical router egress stages. */ >> function s_ROUTER_OUT_UNDNAT(): Stage { Stage{ Egress, 0, > "lr_out_undnat"} } >> -function s_ROUTER_OUT_SNAT(): Stage { Stage{ Egress, 1, > "lr_out_snat"} } >> -function s_ROUTER_OUT_EGR_LOOP(): Stage { Stage{ Egress, 2, > "lr_out_egr_loop"} } >> -function s_ROUTER_OUT_DELIVERY(): Stage { Stage{ Egress, 3, > "lr_out_delivery"} } >> +function s_ROUTER_OUT_POST_UNDNAT(): Stage { Stage{ Egress, 1, > "lr_out_post_undnat"} } >> +function s_ROUTER_OUT_SNAT(): Stage { Stage{ Egress, 2, > "lr_out_snat"} } >> +function s_ROUTER_OUT_EGR_LOOP(): Stage { Stage{ Egress, 3, > "lr_out_egr_loop"} } >> +function s_ROUTER_OUT_DELIVERY(): Stage { Stage{ Egress, 4, > "lr_out_delivery"} } >> >> /* >> * OVS register usage: >> @@ -2886,7 +2887,8 @@ for (&Switch(._uuid = ls_uuid)) { >> function get_match_for_lb_key(ip_address: v46_ip, >> port: bit<16>, >> protocol: Option<string>, >> - redundancy: bool): string = { >> + redundancy: bool, >> + use_nexthop_reg: bool): string = { >> var port_match = if (port != 0) { >> var proto = if (protocol == Some{"udp"}) { >> "udp" >> @@ -2900,8 +2902,18 @@ function get_match_for_lb_key(ip_address: v46_ip, >> }; >> >> var ip_match = match (ip_address) { >> - IPv4{ipv4} -> "ip4.dst == ${ipv4}", >> - IPv6{ipv6} -> "ip6.dst == ${ipv6}" >> + IPv4{ipv4} -> >> + if (use_nexthop_reg) { >> + "${rEG_NEXT_HOP()} == ${ipv4}" >> + } else { >> + "ip4.dst == ${ipv4}" >> + }, >> + IPv6{ipv6} -> >> + if (use_nexthop_reg) { >> + "xx${rEG_NEXT_HOP()} == ${ipv6}" >> + } else { >> + "ip6.dst == ${ipv6}" >> + } >> }; >> >> if (redundancy) { "ip && " } else { "" } ++ ip_match ++ port_match >> @@ -2935,7 +2947,11 @@ function build_lb_vip_actions(lbvip: > Intern<LBVIPWithStatus>, >> for (pair in lbvip.backends) { >> (var backend, var up) = pair; >> if (up) { >> - > up_backends.insert("${backend.ip.to_bracketed_string()}:${backend.port}") >> + if (backend.port != 0) { >> + > up_backends.insert("${backend.ip.to_bracketed_string()}:${backend.port}") >> + } else { >> + up_backends.insert("${backend.ip.to_bracketed_string()}") >> + } >> } >> }; >> >> @@ -2981,7 +2997,7 @@ Flow(.logical_datapath = sw._uuid, >> >> build_lb_vip_actions(lbvip, s_SWITCH_OUT_QOS_MARK(), actions0 ++ > actions1) >> }, >> - var __match = "ct.new && " ++ get_match_for_lb_key(lbvip.vip_addr, > lbvip.vip_port, lb.protocol, false). >> + var __match = "ct.new && " ++ get_match_for_lb_key(lbvip.vip_addr, > lbvip.vip_port, lb.protocol, false, false). >> >> /* Ingress Pre-Hairpin/Nat-Hairpin/Hairpin tabled (Priority 0). >> * Packets that don't need hairpinning should continue processing. >> @@ -3019,7 +3035,7 @@ for (&Switch(._uuid = ls_uuid, .has_lb_vip = true)) > { >> .__match = "ip && ct.new && ct.trk && ${rEGBIT_HAIRPIN()} == 1", >> .actions = "ct_snat_to_vip; next;", >> .external_ids = stage_hint(ls_uuid)); >> - >> + >> /* If packet needs to be hairpinned, for established sessions there >> * should already be an SNAT conntrack entry. >> */ >> @@ -5379,13 +5395,14 @@ function default_allow_flow(datapath: uuid, > stage: Stage): Flow { >> .actions = "next;", >> .external_ids = map_empty()} >> } >> -for (&Router(._uuid = lr_uuid)) { >> +for (r in &Router(._uuid = lr_uuid)) { >> /* Packets are allowed by default. */ >> Flow[default_allow_flow(lr_uuid, s_ROUTER_IN_DEFRAG())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_IN_UNSNAT())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_OUT_SNAT())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_IN_DNAT())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_OUT_UNDNAT())]; >> + Flow[default_allow_flow(lr_uuid, s_ROUTER_OUT_POST_UNDNAT())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_OUT_EGR_LOOP())]; >> Flow[default_allow_flow(lr_uuid, s_ROUTER_IN_ECMP_STATEFUL())]; >> >> @@ -5400,6 +5417,25 @@ for (&Router(._uuid = lr_uuid)) { >> .external_ids = map_empty()) >> } >> >> +for (r in &Router(._uuid = lr_uuid, >> + .is_gateway = is_gateway, >> + .nat = nat, >> + .load_balancer = load_balancer) >> + if is_gateway and (not is_empty(nat) or not > is_empty(load_balancer))) { >> + /* For Gateway routers, if the gateway router has load balancer or > DNAT >> + * rules, we commit newly initiated connections in the reply > direction >> + * to the DNAT zone. This ensures that these flows are tracked. If > the flow >> + * was not committed, it would produce ongoing datapath flows with > the >> + * ct.new flag set. Some NICs are unable to offload these flows. >> + */ >> + Flow(.logical_datapath = lr_uuid, >> + .stage = s_ROUTER_OUT_POST_UNDNAT(), >> + .priority = 50, >> + .__match = "ip && ct.new", >> + .actions = "ct_commit { } ; next; ", >> + .external_ids = map_empty()) >> +} >> + >> Flow(.logical_datapath = lr, >> .stage = s_ROUTER_OUT_SNAT(), >> .priority = 120, >> @@ -5438,7 +5474,7 @@ function lrouter_nat_add_ext_ip_match( >> Some{AllowedExtIps{__as}} -> (" && ${ipX}.${dir} == $${__as.name}", > None), >> Some{ExemptedExtIps{__as}} -> { >> /* Priority of logical flows corresponding to > exempted_ext_ips is >> - * +1 of the corresponding regulr NAT rule. >> + * +1 of the corresponding regular NAT rule. >> * For example, if we have following NAT rule and we > associate >> * exempted external ips to it: >> * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 > 50.0.0.11" >> @@ -5746,8 +5782,7 @@ for (r in &Router(._uuid = lr_uuid, >> * part of a reply. We undo the DNAT here. >> * >> * Note that this only applies for NAT on a distributed > router. >> - * Undo DNAT on a gateway router is done in the ingress DNAT >> - * pipeline stage. */ >> + */ >> if ((nat.nat.__type == "dnat" or nat.nat.__type == > "dnat_and_snat")) { >> Some{var gwport} = l3dgw_port in >> var __match = >> @@ -5953,16 +5988,11 @@ for (r in &Router(._uuid = lr_uuid, >> .context = "lb"); >> >> /* For gateway router, re-circulate every packet through >> - * the DNAT zone. This helps with the following. >> - * >> - * Any packet that needs to be unDNATed in the reverse >> - * direction gets unDNATed. Ideally this could be done in >> - * the egress pipeline. But since the gateway router >> - * does not have any feature that depends on the source >> - * ip address being external IP address for IP routing, >> - * we can do it here, saving a future re-circulation. */ >> + * the DNAT zone so that packets that need to be unDNATed in the > reverse >> + * direction get unDNATed. >> + */ >> Flow(.logical_datapath = lr_uuid, >> - .stage = s_ROUTER_IN_DNAT(), >> + .stage = s_ROUTER_OUT_UNDNAT(), >> .priority = 50, >> .__match = "ip", >> .actions = "flags.loopback = 1; ct_dnat;", >> @@ -6024,7 +6054,16 @@ for (RouterLBVIP( >> * pick a DNAT ip address from a group. >> * 2. If there are L4 ports in load balancing rules, we >> * need the defragmentation to match on L4 ports. */ >> - var __match = "ip && ${ipX}.dst == ${ip_address}" in >> + var match1 = "ip && ${ipX}.dst == ${ip_address}" in >> + var match2 = >> + if (port != 0) { >> + " && ${proto}" >> + } else { >> + "" >> + } in >> + var __match = match1 ++ match2 in >> + var xx = ip_address.xxreg() in >> + var __actions = "${xx}${rEG_NEXT_HOP()} = ${ip_address}; > ct_dnat;" in >> /* One of these flows must be created for each unique LB VIP > address. >> * We create one for each VIP:port pair; flows with the same IP > and >> * different port numbers will produce identical flows that will >> @@ -6033,7 +6072,7 @@ for (RouterLBVIP( >> .stage = s_ROUTER_IN_DEFRAG(), >> .priority = 100, >> .__match = __match, >> - .actions = "ct_next;", >> + .actions = __actions, >> .external_ids = stage_hint(lb._uuid)); >> >> /* Higher priority rules are added for load-balancing in DNAT >> @@ -6041,7 +6080,8 @@ for (RouterLBVIP( >> * via add_router_lb_flow(). One flow is for specific matching >> * on ct.new with an action of "ct_lb($targets);". The other >> * flow is for ct.est with an action of "ct_dnat;". */ >> - var match1 = "ip && ${ipX}.dst == ${ip_address}" in >> + var xx = ip_address.xxreg() in >> + var match1 = "ip && ${xx}${rEG_NEXT_HOP()} == ${ip_address}" in >> (var prio, var match2) = >> if (port != 0) { >> (120, " && ${proto} && ${proto}.dst == ${port}") >> @@ -6056,12 +6096,21 @@ for (RouterLBVIP( >> var snat_for_lb = snat_for_lb(r.options, lb) in >> { >> /* A match and actions for established connections. */ >> - var est_match = "ct.est && " ++ __match in >> + var est_match = "ct.est && " ++ match1 ++ " && > ct_label.natted == 1" ++ >> + if (port != 0) { >> + " && ${proto}" >> + } else { >> + "" >> + } ++ >> + match ((l3dgw_port, backends != "" or > lb.options.get_bool_def("reject", false))) { >> + (Some{gwport}, true) -> " && > is_chassis_resident(${redirect_port_name})", >> + _ -> "" >> + } in >> var actions = >> match (snat_for_lb) { >> - SkipSNAT -> "flags.skip_snat_for_lb = 1; ct_dnat;", >> - ForceSNAT -> "flags.force_snat_for_lb = 1; ct_dnat;", >> - _ -> "ct_dnat;" >> + SkipSNAT -> "flags.skip_snat_for_lb = 1; next;", >> + ForceSNAT -> "flags.force_snat_for_lb = 1; next;", >> + _ -> "next;" >> } in >> Flow(.logical_datapath = lr_uuid, >> .stage = s_ROUTER_IN_DNAT(), >> @@ -6152,7 +6201,7 @@ Flow(.logical_datapath = r._uuid, >> r.load_balancer.contains(lb._uuid), >> var __match >> = "ct.new && " ++ >> - get_match_for_lb_key(lbvip.vip_addr, lbvip.vip_port, > lb.protocol, true) ++ >> + get_match_for_lb_key(lbvip.vip_addr, lbvip.vip_port, > lb.protocol, true, true) ++ >> match (r.l3dgw_port) { >> Some{gwport} -> " && > is_chassis_resident(${r.redirect_port_name})", >> _ -> "" >> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at >> index d81975cb18a6..accaa87033f4 100644 >> --- a/tests/ovn-northd.at >> +++ b/tests/ovn-northd.at >> @@ -1406,40 +1406,39 @@ AT_SETUP([ovn -- Load balancer VIP in NAT > entries]) >> AT_SKIP_IF([test $HAVE_PYTHON = no]) >> ovn_start >> >> -ovn-nbctl lr-add lr0 >> -ovn-nbctl lrp-add lr0 lr0-public 00:00:01:01:02:04 192.168.2.1/24 >> -ovn-nbctl lrp-add lr0 lr0-join 00:00:01:01:02:04 10.10.0.1/24 >> +check ovn-nbctl lr-add lr0 >> +check ovn-nbctl lrp-add lr0 lr0-public 00:00:01:01:02:04 192.168.2.1/24 >> +check ovn-nbctl lrp-add lr0 lr0-join 00:00:01:01:02:04 10.10.0.1/24 >> >> -ovn-nbctl set logical_router lr0 options:chassis=ch1 >> +check ovn-nbctl set logical_router lr0 options:chassis=ch1 >> >> -ovn-nbctl lb-add lb1 "192.168.2.1:8080" "10.0.0.4:8080" >> -ovn-nbctl lb-add lb2 "192.168.2.4:8080" "10.0.0.5:8080" udp >> -ovn-nbctl lb-add lb3 "192.168.2.5:8080" "10.0.0.6:8080" >> -ovn-nbctl lb-add lb4 "192.168.2.6:8080" "10.0.0.7:8080" >> +check ovn-nbctl lb-add lb1 "192.168.2.1:8080" "10.0.0.4:8080" >> +check ovn-nbctl lb-add lb2 "192.168.2.4:8080" "10.0.0.5:8080" udp >> +check ovn-nbctl lb-add lb3 "192.168.2.5:8080" "10.0.0.6:8080" >> +check ovn-nbctl lb-add lb4 "192.168.2.6:8080" "10.0.0.7:8080" >> >> -ovn-nbctl lr-lb-add lr0 lb1 >> -ovn-nbctl lr-lb-add lr0 lb2 >> -ovn-nbctl lr-lb-add lr0 lb3 >> -ovn-nbctl lr-lb-add lr0 lb4 >> +check ovn-nbctl lr-lb-add lr0 lb1 >> +check ovn-nbctl lr-lb-add lr0 lb2 >> +check ovn-nbctl lr-lb-add lr0 lb3 >> +check ovn-nbctl lr-lb-add lr0 lb4 >> >> -ovn-nbctl lr-nat-add lr0 snat 192.168.2.1 10.0.0.0/24 >> -ovn-nbctl lr-nat-add lr0 dnat_and_snat 192.168.2.4 10.0.0.4 >> +check ovn-nbctl lr-nat-add lr0 snat 192.168.2.1 10.0.0.0/24 >> +check ovn-nbctl lr-nat-add lr0 dnat_and_snat 192.168.2.4 10.0.0.4 >> check ovn-nbctl --wait=sb lr-nat-add lr0 dnat 192.168.2.5 10.0.0.5 >> >> ovn-sbctl dump-flows lr0 > sbflows >> AT_CAPTURE_FILE([sbflows]) >> >> -OVS_WAIT_UNTIL([test 1 = $(grep lr_in_unsnat sbflows | \ >> -grep "ip4 && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 8080" -c) ]) >> - >> -AT_CHECK([test 1 = $(grep lr_in_unsnat sbflows | \ >> -grep "ip4 && ip4.dst == 192.168.2.4 && udp && udp.dst == 8080" -c) ]) >> - >> -AT_CHECK([test 1 = $(grep lr_in_unsnat sbflows | \ >> -grep "ip4 && ip4.dst == 192.168.2.5 && tcp && tcp.dst == 8080" -c) ]) >> - >> -AT_CHECK([test 0 = $(grep lr_in_unsnat sbflows | \ >> -grep "ip4 && ip4.dst == 192.168.2.6 && tcp && tcp.dst == 8080" -c) ]) >> +# There shoule be no flows for LB VIPs in lr_in_unsnat if the VIP is not > a >> +# dnat_and_snat or snat entry. >> +AT_CHECK([grep "lr_in_unsnat" sbflows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=120 , match=(ip4 && ip4.dst > == 192.168.2.1 && tcp && tcp.dst == 8080), action=(next;) >> + table=4 (lr_in_unsnat ), priority=120 , match=(ip4 && ip4.dst > == 192.168.2.4 && udp && udp.dst == 8080), action=(next;) >> + table=4 (lr_in_unsnat ), priority=120 , match=(ip4 && ip4.dst > == 192.168.2.5 && tcp && tcp.dst == 8080), action=(next;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 192.168.2.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 192.168.2.4), action=(ct_snat;) >> +]) >> >> AT_CLEANUP >> ]) >> @@ -1458,8 +1457,8 @@ ovn-nbctl set logical_router lr0 > options:dnat_force_snat_ip=192.168.2.3 >> ovn-nbctl --wait=sb sync >> >> AT_CHECK([ovn-sbctl lflow-list lr0 | grep lr_in_unsnat | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(ip4 && ip4.dst > == 192.168.2.3), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(ip4 && ip4.dst > == 192.168.2.3), action=(ct_snat;) >> ]) >> >> AT_CLEANUP >> @@ -3163,14 +3162,28 @@ ovn-sbctl dump-flows lr0 > lr0flows >> AT_CAPTURE_FILE([lr0flows]) >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> ]) >> >> AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), action=(ct_dnat;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(ct_lb(backends=10.0.0.4:8080);) >> - table=6 (lr_in_dnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), > action=(ct_lb(backends=10.0.0.4:8080);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> ]) >> >> check ovn-nbctl --wait=sb set logical_router lr0 > options:lb_force_snat_ip="20.0.0.4 aef0::4" >> @@ -3180,23 +3193,37 @@ AT_CAPTURE_FILE([lr0flows]) >> >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(ip4 && ip4.dst > == 20.0.0.4), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(ip6 && ip6.dst > == aef0::4), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(ip4 && ip4.dst > == 20.0.0.4), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(ip6 && ip6.dst > == aef0::4), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> ]) >> >> AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_dnat;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.4:8080);) >> - table=6 (lr_in_dnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> ]) >> >> AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> - table=1 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> - table=1 (lr_out_snat ), priority=100 , > match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);) >> - table=1 (lr_out_snat ), priority=100 , > match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);) >> - table=1 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=100 , > match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);) >> + table=2 (lr_out_snat ), priority=100 , > match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> ]) >> >> check ovn-nbctl --wait=sb set logical_router lr0 > options:lb_force_snat_ip="router_ip" >> @@ -3208,25 +3235,39 @@ AT_CHECK([grep "lr_in_ip_input" lr0flows | grep > "priority=60" | sort], [0], [dnl >> ]) >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> ]) >> >> AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_dnat;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.4:8080);) >> - table=6 (lr_in_dnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> ]) >> >> AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> - table=1 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.100);) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), > action=(ct_snat(20.0.0.1);) >> - table=1 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.100);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), > action=(ct_snat(20.0.0.1);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> ]) >> >> check ovn-nbctl --wait=sb remove logical_router lr0 options chassis >> @@ -3235,12 +3276,12 @@ ovn-sbctl dump-flows lr0 > lr0flows >> AT_CAPTURE_FILE([lr0flows]) >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> ]) >> >> AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> - table=1 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> - table=1 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> ]) >> >> check ovn-nbctl set logical_router lr0 options:chassis=ch1 >> @@ -3250,27 +3291,41 @@ ovn-sbctl dump-flows lr0 > lr0flows >> AT_CAPTURE_FILE([lr0flows]) >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip6.dst == bef0::1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip6.dst == bef0::1), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> ]) >> >> AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_dnat;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.4:8080);) >> - table=6 (lr_in_dnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> ]) >> >> AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> - table=1 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.100);) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), > action=(ct_snat(20.0.0.1);) >> - table=1 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), > action=(ct_snat(bef0::1);) >> - table=1 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.100);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), > action=(ct_snat(20.0.0.1);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), > action=(ct_snat(bef0::1);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> ]) >> >> check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080 >> @@ -3280,20 +3335,35 @@ check ovn-nbctl --wait=sb lb-del lb1 >> ovn-sbctl dump-flows lr0 > lr0flows >> >> AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> - table=5 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> - table=5 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip6.dst == bef0::1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.100), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip4.dst == 20.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw1" && ip6.dst == bef0::1), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.20 && tcp), action=(reg0 = 10.0.0.20; ct_dnat;) >> ]) >> >> AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], > [0], [dnl >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > ip4.dst == 10.0.0.20 && tcp && tcp.dst == 80), > action=(flags.skip_snat_for_lb = 1; ct_dnat;) >> - table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.20 && tcp && tcp.dst == 80), > action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.20 && ct_label.natted == 1 && tcp), > action=(flags.skip_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.20 && tcp && tcp.dst == 80), action=(flags.skip_snat_for_lb > = 1; ct_lb(backends=10.0.0.40:8080);) >> ]) >> >> AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], > [0], [dnl >> - table=1 (lr_out_snat ), priority=120 , > match=(flags.skip_snat_for_lb == 1 && ip), action=(next;) >> + table=2 (lr_out_snat ), priority=120 , > match=(flags.skip_snat_for_lb == 1 && ip), action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> ]) >> >> AT_CLEANUP >> @@ -3737,3 +3807,451 @@ AT_CHECK([ovn-trace --minimal 'inport == > "sw1-port1" && eth.src == 50:54:00:00:0 >> >> AT_CLEANUP >> ]) >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([ovn -- LR NAT flows]) >> +ovn_start >> + >> +check ovn-nbctl \ >> + -- ls-add sw0 \ >> + -- lb-add lb0 10.0.0.10:80 10.0.0.4:8080 \ >> + -- ls-lb-add sw0 lb0 >> + >> +check ovn-nbctl lr-add lr0 >> +check ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 >> +check ovn-nbctl lsp-add sw0 sw0-lr0 >> +check ovn-nbctl lsp-set-type sw0-lr0 router >> +check ovn-nbctl lsp-set-addresses sw0-lr0 00:00:00:00:ff:01 >> +check ovn-nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 >> + >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> +]) >> + >> +# Create few dnat_and_snat entries >> + >> +check ovn-nbctl lr-nat-add lr0 snat 172.168.0.10 10.0.0.0/24 >> +check ovn-nbctl lr-nat-add lr0 dnat_and_snat 172.168.0.20 10.0.0.3 >> +check ovn-nbctl lr-nat-add lr0 snat 172.168.0.30 10.0.0.10 >> + >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> +]) >> + >> +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 >> + >> +# Create a distributed gw port on lr0 >> +check ovn-nbctl ls-add public >> +check ovn-nbctl lrp-add lr0 lr0-public 00:00:00:00:ff:02 172.168.0.10/24 >> +check ovn-nbctl lrp-set-gateway-chassis lr0-public gw1 >> + >> +ovn-nbctl lsp-add public public-lr0 -- set Logical_Switch_Port > public-lr0 \ >> + type=router options:router-port=lr0-public \ >> + -- lsp-set-addresses public-lr0 router >> + >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.10 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.30 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=100 , match=(ip && ip4.src == > 10.0.0.3 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=153 , match=(ip && ip4.src == > 10.0.0.0/24 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=161 , match=(ip && ip4.src == > 10.0.0.10 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=161 , match=(ip && ip4.src == > 10.0.0.3 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +# Associate load balancer to lr0 >> + >> +check ovn-nbctl lb-add lb0 172.168.0.100:8082 "10.0.0.50:82,10.0.0.60:82" >> + >> +# No L4 >> +check ovn-nbctl lb-add lb1 172.168.0.200 "10.0.0.80,10.0.0.81" >> +check ovn-nbctl lb-add lb2 172.168.0.210:60 "10.0.0.50:6062, > 10.0.0.60:6062" udp >> + >> +check ovn-nbctl lr-lb-add lr0 lb0 >> +check ovn-nbctl lr-lb-add lr0 lb1 >> +check ovn-nbctl lr-lb-add lr0 lb2 >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.10 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.30 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.100 && tcp), action=(reg0 = 172.168.0.100; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.200), action=(reg0 = 172.168.0.200; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.210 && udp), action=(reg0 = 172.168.0.210; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20 && inport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.est && ip && > reg0 == 172.168.0.200 && ct_label.natted == 1 && > is_chassis_resident("cr-lr0-public")), action=(next;) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.new && ip && > reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), > action=(ct_lb(backends=10.0.0.80,10.0.0.81);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp && > is_chassis_resident("cr-lr0-public")), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.100 && ct_label.natted == 1 && tcp && > is_chassis_resident("cr-lr0-public")), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.210 && ct_label.natted == 1 && udp && > is_chassis_resident("cr-lr0-public")), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80 && > is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.4:8080 > );) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.100 && tcp && tcp.dst == 8082 && > is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.210 && udp && udp.dst == 60 && > is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062 > ,10.0.0.60:6062);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=100 , match=(ip && ip4.src == > 10.0.0.3 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) >> + table=0 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) >> + table=0 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) > && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), > action=(ct_dnat;) >> + table=0 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == > 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), > action=(ct_dnat;) >> + table=0 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=153 , match=(ip && ip4.src == > 10.0.0.0/24 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=161 , match=(ip && ip4.src == > 10.0.0.10 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=161 , match=(ip && ip4.src == > 10.0.0.3 && outport == "lr0-public" && > is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +# Make the logical router as Gateway router >> +check ovn-nbctl clear logical_router_port lr0-public gateway_chassis >> +check ovn-nbctl set logical_router lr0 options:chassis=gw1 >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.20), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.30), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.100 && tcp), action=(reg0 = 172.168.0.100; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.200), action=(reg0 = 172.168.0.200; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.210 && udp), action=(reg0 = 172.168.0.210; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20), action=(flags.loopback = 1; ct_dnat(10.0.0.3);) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.est && ip && > reg0 == 172.168.0.200 && ct_label.natted == 1), action=(next;) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.new && ip && > reg0 == 172.168.0.200), action=(ct_lb(backends=10.0.0.80,10.0.0.81);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.100 && ct_label.natted == 1 && tcp), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.210 && ct_label.natted == 1 && udp), action=(next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), > action=(ct_lb(backends=10.0.0.4:8080);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.100 && tcp && tcp.dst == 8082), action=(ct_lb(backends= > 10.0.0.50:82,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.210 && udp && udp.dst == 60), action=(ct_lb(backends= > 10.0.0.50:6062,10.0.0.60:6062);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=25 , match=(ip && ip4.src == > 10.0.0.0/24), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.10), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.3), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +# Set lb force snat logical router. >> +check ovn-nbctl --wait=sb set logical_router lr0 > options:lb_force_snat_ip="router_ip" >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.20), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.30), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.100 && tcp), action=(reg0 = 172.168.0.100; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.200), action=(reg0 = 172.168.0.200; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.210 && udp), action=(reg0 = 172.168.0.210; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20), action=(flags.loopback = 1; ct_dnat(10.0.0.3);) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.est && ip && > reg0 == 172.168.0.200 && ct_label.natted == 1), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.new && ip && > reg0 == 172.168.0.200), action=(flags.force_snat_for_lb = 1; > ct_lb(backends=10.0.0.80,10.0.0.81);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.100 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.210 && ct_label.natted == 1 && udp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.100 && tcp && tcp.dst == 8082), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.210 && udp && udp.dst == 60), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062 > ,10.0.0.60:6062);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=25 , match=(ip && ip4.src == > 10.0.0.0/24), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.10), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.3), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +# Add a LB VIP same as router ip. >> +check ovn-nbctl lb-add lb0 172.168.0.10:9082 "10.0.0.50:82,10.0.0.60:82" >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=120 , match=(ip4 && ip4.dst > == 172.168.0.10 && tcp && tcp.dst == 9082), action=(next;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.20), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.30), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.10 && tcp), action=(reg0 = 172.168.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.100 && tcp), action=(reg0 = 172.168.0.100; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.200), action=(reg0 = 172.168.0.200; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.210 && udp), action=(reg0 = 172.168.0.210; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20), action=(flags.loopback = 1; ct_dnat(10.0.0.3);) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.est && ip && > reg0 == 172.168.0.200 && ct_label.natted == 1), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.new && ip && > reg0 == 172.168.0.200), action=(flags.force_snat_for_lb = 1; > ct_lb(backends=10.0.0.80,10.0.0.81);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.100 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.210 && ct_label.natted == 1 && udp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.10 && tcp && tcp.dst == 9082), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.100 && tcp && tcp.dst == 8082), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.210 && udp && udp.dst == 60), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062 > ,10.0.0.60:6062);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=25 , match=(ip && ip4.src == > 10.0.0.0/24), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.10), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.3), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +# Add IPv6 router port and LB. >> +check ovn-nbctl lrp-del lr0-sw0 >> +check ovn-nbctl lrp-del lr0-public >> +check ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 aef0::1 >> +check ovn-nbctl lrp-add lr0 lr0-public 00:00:00:00:ff:02 172.168.0.10/24 > def0::10 >> + >> +lb1_uuid=$(fetch_column nb:Load_Balancer _uuid name=lb1) >> +ovn-nbctl set load_balancer $lb1_uuid > vips:'"[[def0::2]]:8000"'='"@<:@aef0::2@:>@:80,@<:@aef0::3@:>@:80"' >> + >> +ovn-nbctl list load_Balancer >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl >> + table=4 (lr_in_unsnat ), priority=0 , match=(1), > action=(next;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip4.dst == 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-public" && ip6.dst == def0::10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip4.dst == 10.0.0.1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=110 , match=(inport == > "lr0-sw0" && ip6.dst == aef0::1), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=120 , match=(ip4 && ip4.dst > == 172.168.0.10 && tcp && tcp.dst == 9082), action=(next;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.10), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.20), action=(ct_snat;) >> + table=4 (lr_in_unsnat ), priority=90 , match=(ip && ip4.dst == > 172.168.0.30), action=(ct_snat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl >> + table=5 (lr_in_defrag ), priority=0 , match=(1), > action=(next;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 10.0.0.10 && tcp), action=(reg0 = 10.0.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.10 && tcp), action=(reg0 = 172.168.0.10; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.100 && tcp), action=(reg0 = 172.168.0.100; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.200), action=(reg0 = 172.168.0.200; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == > 172.168.0.210 && udp), action=(reg0 = 172.168.0.210; ct_dnat;) >> + table=5 (lr_in_defrag ), priority=100 , match=(ip && ip6.dst == > def0::2 && tcp), action=(xxreg0 = def0::2; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl >> + table=6 (lr_in_dnat ), priority=0 , match=(1), > action=(next;) >> + table=6 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == > 172.168.0.20), action=(flags.loopback = 1; ct_dnat(10.0.0.3);) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.est && ip && > reg0 == 172.168.0.200 && ct_label.natted == 1), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=110 , match=(ct.new && ip && > reg0 == 172.168.0.200), action=(flags.force_snat_for_lb = 1; > ct_lb(backends=10.0.0.80,10.0.0.81);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 10.0.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.10 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.100 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > reg0 == 172.168.0.210 && ct_label.natted == 1 && udp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.est && ip && > xxreg0 == def0::2 && ct_label.natted == 1 && tcp), > action=(flags.force_snat_for_lb = 1; next;) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 10.0.0.10 && tcp && tcp.dst == 80), action=(flags.force_snat_for_lb > = 1; ct_lb(backends=10.0.0.4:8080);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.10 && tcp && tcp.dst == 9082), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.100 && tcp && tcp.dst == 8082), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:82 > ,10.0.0.60:82);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > reg0 == 172.168.0.210 && udp && udp.dst == 60), > action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062 > ,10.0.0.60:6062);) >> + table=6 (lr_in_dnat ), priority=120 , match=(ct.new && ip && > xxreg0 == def0::2 && tcp && tcp.dst == 8000), > action=(flags.force_snat_for_lb = 1; > ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);) >> +]) >> + >> +AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl >> + table=0 (lr_out_undnat ), priority=0 , match=(1), > action=(next;) >> + table=0 (lr_out_undnat ), priority=50 , match=(ip), > action=(flags.loopback = 1; ct_dnat;) >> +]) >> + >> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl >> + table=1 (lr_out_post_undnat ), priority=0 , match=(1), > action=(next;) >> + table=1 (lr_out_post_undnat ), priority=50 , match=(ip && ct.new), > action=(ct_commit { } ; next; ) >> +]) >> + >> +AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl >> + table=2 (lr_out_snat ), priority=0 , match=(1), > action=(next;) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), > action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), > action=(ct_snat(10.0.0.1);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), > action=(ct_snat(def0::10);) >> + table=2 (lr_out_snat ), priority=110 , > match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), > action=(ct_snat(aef0::1);) >> + table=2 (lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) >> + table=2 (lr_out_snat ), priority=25 , match=(ip && ip4.src == > 10.0.0.0/24), action=(ct_snat(172.168.0.10);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.10), action=(ct_snat(172.168.0.30);) >> + table=2 (lr_out_snat ), priority=33 , match=(ip && ip4.src == > 10.0.0.3), action=(ct_snat(172.168.0.20);) >> +]) >> + >> +AT_CLEANUP >> +]) >> diff --git a/tests/ovn.at b/tests/ovn.at >> index bc494fcad9bb..ea1593197f21 100644 >> --- a/tests/ovn.at >> +++ b/tests/ovn.at >> @@ -20571,7 +20571,7 @@ AT_CAPTURE_FILE([sbflows2]) >> OVS_WAIT_FOR_OUTPUT( >> [ovn-sbctl dump-flows > sbflows2 >> ovn-sbctl dump-flows lr0 | grep ct_lb | grep priority=120 | sed > 's/table=..//'], 0, >> - [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && > ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && > is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80, > 20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) >> + [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && reg0 > == 10.0.0.10 && tcp && tcp.dst == 80 && > is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80, > 20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) >> ]) >> >> # get the svc monitor mac. >> @@ -20612,8 +20612,8 @@ AT_CHECK( >> AT_CAPTURE_FILE([sbflows4]) >> ovn-sbctl dump-flows lr0 > sbflows4 >> AT_CHECK([grep lr_in_dnat sbflows4 | grep priority=120 | sed > 's/table=..//' | sort], [0], [dnl >> - (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst > == 10.0.0.10 && tcp && tcp.dst == 80 && > is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) >> - (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst > == 10.0.0.10 && tcp && tcp.dst == 80 && > is_chassis_resident("cr-lr0-public")), action=(drop;) >> + (lr_in_dnat ), priority=120 , match=(ct.est && ip && reg0 == > 10.0.0.10 && ct_label.natted == 1 && tcp && > is_chassis_resident("cr-lr0-public")), action=(next;) >> + (lr_in_dnat ), priority=120 , match=(ct.new && ip && reg0 == > 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), > action=(drop;) >> ]) >> >> # Delete sw0-p1 >> diff --git a/tests/system-ovn.at b/tests/system-ovn.at >> index 552fdae52665..4f104171bdba 100644 >> --- a/tests/system-ovn.at >> +++ b/tests/system-ovn.at >> @@ -116,6 +116,7 @@ NS_CHECK_EXEC([alice1], [ping -q -c 3 -i 0.3 -w 2 > 30.0.0.2 | FORMAT_PING], \ >> # Check conntrack entries. >> AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.2) | \ >> sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl >> > +icmp,orig=(src=172.16.1.2,dst=192.168.1.2,id=<cleared>,type=8,code=0),reply=(src=192.168.1.2,dst=172.16.1.2,id=<cleared>,type=0,code=0),zone=<cleared> >> > > icmp,orig=(src=172.16.1.2,dst=30.0.0.2,id=<cleared>,type=8,code=0),reply=(src=192.168.1.2,dst=172.16.1.2,id=<cleared>,type=0,code=0),zone=<cleared> >> ]) >> >> @@ -298,6 +299,7 @@ NS_CHECK_EXEC([alice1], [ping6 -q -c 3 -i 0.3 -w 2 > fd30::2 | FORMAT_PING], \ >> # Check conntrack entries. >> AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd21::2) | \ >> sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl >> > +icmpv6,orig=(src=fd21::2,dst=fd11::2,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd21::2,id=<cleared>,type=129,code=0),zone=<cleared> >> > > icmpv6,orig=(src=fd21::2,dst=fd30::2,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd21::2,id=<cleared>,type=129,code=0),zone=<cleared> >> ]) >> >> @@ -2197,11 +2199,12 @@ > tcp,orig=(src=172.16.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(sr >> ]) >> >> check_est_flows () { >> - n=$(ovs-ofctl dump-flows br-int table=15 | grep \ >> > -"priority=120,ct_state=+est+trk,tcp,metadata=0x2,nw_dst=30.0.0.2,tp_dst=8000" > \ >> -| grep nat | sed -n 's/.*n_packets=\([[0-9]]\{1,\}\).*/\1/p') >> + n=$(ovs-ofctl dump-flows br-int table=13 | grep \ >> +"priority=100,tcp,metadata=0x2,nw_dst=30.0.0.2" | grep nat | >> +sed -n 's/.*n_packets=\([[0-9]]\{1,\}\).*/\1/p') >> >> echo "n_packets=$n" >> + test ! -z $n >> test "$n" != 0 >> } >> >> @@ -2222,7 +2225,7 @@ ovn-nbctl set load_balancer $uuid vips:'" > 30.0.0.2:8000"'='"192.168.1.2:80,192.16 >> >> ovn-nbctl list load_balancer >> ovn-sbctl dump-flows R2 >> -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=41 | \ >> +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \ >> grep 'nat(src=20.0.0.2)']) >> >> dnl Test load-balancing that includes L4 ports in NAT. >> @@ -2260,7 +2263,7 @@ ovn-nbctl set load_balancer $uuid vips:'" > 30.0.0.2:8000"'='"192.168.1.2:80,192.16 >> >> ovn-nbctl list load_balancer >> ovn-sbctl dump-flows R2 >> -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=41 | \ >> +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \ >> grep 'nat(src=20.0.0.2)']) >> >> rm -f wget*.log >> -- >> 2.27.0 >> >> >> _______________________________________________ >> dev mailing list >> d...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev