One OpenFlow flow's actions are translated into one datapath flow. "resubmit" is followed during that translation, so a flood that resubmits once per port becomes a single action list, and a large enough multicast group overruns the limits on how big that list may be: the FLOW_MOD that carries it, the datapath's own 16-bit Netlink attribute length, and the number of resubmits one translation may perform.
Neither failure is graceful. Past the FLOW_MOD limit ovn-controller discards the flow outright and the flood reaches nobody. Past the datapath limit ofproto stops translating without reporting an error, so the flow installs and quietly delivers to a prefix of the group: in testing, 712 of 800 remote ports, with nothing to say the rest were dropped. Install such a group as a chain of flows instead. Each one carries a slice of the outputs and, except the last, ends with "recirc" and a resubmit back to the same table. "recirc" ends the datapath flow, so the resubmit after it runs in the continuation and looks up the next slice by an id published in reg0; "resubmit" alone would be composed back into the one flow this is trying to avoid. reg0 is saved on the stack across that lookup, because the egress pipeline reads it, and MLF_MC_CHUNK marks the lookup so only the continuation matches it. A slice is cut only after a completed delivery. Cutting between a port's register loads and the resubmit that consumes them would deliver that port to the multicast key the continuation restores. Slice size is charged against all three limits at once, because no single byte count stands in for the others: a local port is a resubmit that expands into a whole egress pipeline, while a remote port is field writes that collapse into one encapsulation header. ovs-vswitchd may not support "recirc". In that case install the single large flow as before and log a warning, since there is nothing better to offer and the degradation would otherwise be silent. Assisted-by: Claude Opus 5, Cursor Signed-off-by: Han Zhou <[email protected]> --- controller/physical.c | 250 +++++++++++++++++++++++++++++++- include/ovn/logical-fields.h | 5 + ovn-architecture.7.xml | 10 ++ tests/ovn.at | 267 +++++++++++++++++++++++++++++++++++ 4 files changed, 527 insertions(+), 5 deletions(-) diff --git a/controller/physical.c b/controller/physical.c index ea1f2e0d1a53..add327625a99 100644 --- a/controller/physical.c +++ b/controller/physical.c @@ -46,6 +46,7 @@ #include "lib/ovn-sb-idl.h" #include "lib/ovn-util.h" #include "ovn/actions.h" +#include "ovn/features.h" #include "if-status.h" #include "physical.h" #include "pinctrl.h" @@ -2938,6 +2939,245 @@ flood_querier_port_add(struct mcast_querier_port *flood_ports, (*n_flood_ports)++; } +/* One OpenFlow flow's actions are translated into one datapath flow, and + * resubmit is followed during that translation, so a flood of per-port + * resubmits becomes a single action list. Three limits bound that list: + * the FLOW_MOD carrying it, the datapath's nl_attr_oversized() (which + * applies to the kernel and the userspace datapath alike), and OVS's + * resubmit count. A flood too big for one flow is split into slices, + * each sized against all three. + * + * recirc ends a slice; the resubmit after it runs in the continuation and + * looks the next slice up by its id in reg0. reg0 is stacked around that + * lookup so the egress pipeline still sees what the logical pipeline left + * there, and MLF_MC_CHUNK marks the lookup only. + * + * A datapath caps recirculation depth, so a chain longer than nine slices + * silently truncates the flood; the budgets below stay large enough that + * nine slices cover any plausible group. */ + +/* Encoded OpenFlow action bytes. add_flow_mod() has about 65,440 to + * spend on actions once the headers and the match are accounted for. */ +#define MC_CHUNK_OF_MAX (32 * 1024) + +/* Estimated datapath action bytes, against nl_attr_oversized's 65,531. */ +#define MC_CHUNK_DP_MAX (48 * 1024) + +/* Resubmits into the egress pipeline. Each one costs as many of OVS's + * 4096 resubmits as that pipeline is deep, so keep this well under it. */ +#define MC_CHUNK_RESUBMITS_MAX 128 + +struct mc_chunk_cost { + size_t of_bytes; + size_t dp_bytes; + size_t resubmits; +}; + +/* What one ofpact costs against each of the three limits. 'scratch' is + * reused across calls. */ +static void +mc_chunk_cost(const struct ofpact *a, struct ofpbuf *scratch, + struct mc_chunk_cost *cost) +{ + /* The internal ofpact size is no guide to the encoded size: a + * set_field on a tunnel metadata field carries a 128-byte value and + * mask but encodes to 16 bytes. Encode it and measure. */ + ofpbuf_clear(scratch); + ofpacts_put_openflow_actions(a, OFPACT_ALIGN(a->len), scratch, + OFP15_VERSION); + cost->of_bytes = scratch->size; + + /* An output to a tunnel port pushes an encapsulation header, and a + * resubmit can reach one. */ + cost->dp_bytes = (a->type == OFPACT_OUTPUT + || a->type == OFPACT_RESUBMIT) ? 128 : 32; + cost->resubmits = a->type == OFPACT_RESUBMIT; +} + +static void +mc_chunk_cost_add(struct mc_chunk_cost *sum, + const struct mc_chunk_cost *cost) +{ + sum->of_bytes += cost->of_bytes; + sum->dp_bytes += cost->dp_bytes; + sum->resubmits += cost->resubmits; +} + +static bool +mc_chunk_cost_exceeded(const struct mc_chunk_cost *sum) +{ + return sum->of_bytes > MC_CHUNK_OF_MAX + || sum->dp_bytes > MC_CHUNK_DP_MAX + || sum->resubmits > MC_CHUNK_RESUBMITS_MAX; +} + +static bool +mc_fits_one_chunk(const struct ofpbuf *ofpacts) +{ + struct mc_chunk_cost sum = { 0, 0, 0 }; + struct ofpbuf scratch; + const struct ofpact *a; + + ofpbuf_init(&scratch, 64); + OFPACT_FOR_EACH (a, (const struct ofpact *) ofpacts->data, ofpacts->size) { + struct mc_chunk_cost cost; + + mc_chunk_cost(a, &scratch, &cost); + mc_chunk_cost_add(&sum, &cost); + } + ofpbuf_uninit(&scratch); + + return !mc_chunk_cost_exceeded(&sum); +} + +static void +mc_chunk_match(const struct sbrec_multicast_group *mc, + const struct mc_flow_ctx *ctx, uint32_t chunk_id, + struct match *match) +{ + uint32_t flags = ctx->flags; + uint32_t mask = ctx->flags_mask | MLF_MC_CHUNK; + + match_init_catchall(match); + match_outport_dp_and_port_keys(match, mc->datapath->tunnel_key, + mc->tunnel_key); + if (chunk_id > 0) { + flags |= MLF_MC_CHUNK; + match_set_reg(match, MFF_REG0 - MFF_REG0, chunk_id); + } + match_set_reg_masked(match, MFF_LOG_FLAGS - MFF_REG0, flags, mask); +} + +static void +mc_compose_chunk(struct ofpbuf *dst, const struct ofpbuf *body, + bool first, bool last, uint32_t next_chunk, + int64_t mc_key, uint8_t same_table, bool resume_next, + uint8_t next_table) +{ + ofpbuf_clear(dst); + if (!first) { + put_stack(MFF_REG0, ofpact_put_STACK_POP(dst)); + put_load(0, MFF_LOG_FLAGS, MLF_MC_CHUNK_BIT, 1, dst); + } + ofpbuf_put(dst, body->data, body->size); + if (!last) { + /* The body may have overwritten the outport this table matches, + * and the egress pipeline reads reg0. */ + put_stack(MFF_REG0, ofpact_put_STACK_PUSH(dst)); + put_load(mc_key, MFF_LOG_OUTPORT, 0, 32, dst); + put_load(next_chunk, MFF_REG0, 0, 32, dst); + put_load(1, MFF_LOG_FLAGS, MLF_MC_CHUNK_BIT, 1, dst); + ofpact_put_RECIRC(dst); + put_resubmit(same_table, dst); + } else if (resume_next) { + put_load(mc_key, MFF_LOG_OUTPORT, 0, 32, dst); + put_resubmit(next_table, dst); + } +} + +static void +mc_add_chunk_flow(const struct sbrec_multicast_group *mc, + const struct mc_flow_ctx *ctx, uint32_t chunk_id, + const struct ofpbuf *actions, + struct ovn_desired_flow_table *flow_table) +{ + struct match match; + + mc_chunk_match(mc, ctx, chunk_id, &match); + ofctrl_add_flow(flow_table, ctx->stage, ctx->prio, + mc->header_.uuid.parts[0], &match, actions, + &mc->header_.uuid); +} + +static void +mc_split_flows(const struct sbrec_multicast_group *mc, + struct mc_flow_ctx *ctx, bool resume_next, uint8_t next_table, + struct ovn_desired_flow_table *flow_table) +{ + struct ofpbuf body, actions, scratch; + struct mc_chunk_cost sum = { 0, 0, 0 }; + const struct ofpact *a; + const struct ofpact *base = ctx->ofpacts.data; + size_t total = ctx->ofpacts.size; + uint32_t chunk_id = 0; + + ofpbuf_init(&body, 0); + ofpbuf_init(&actions, 0); + ofpbuf_init(&scratch, 64); + + /* A slice has to end on a completed delivery: cutting between a port's + * register loads and the resubmit that consumes them would deliver that + * port to the multicast key the continuation restores. */ + const struct ofpact *last = NULL; + OFPACT_FOR_EACH (a, base, total) { + struct mc_chunk_cost cost, next = sum; + bool boundary = last && (last->type == OFPACT_RESUBMIT + || last->type == OFPACT_OUTPUT); + + mc_chunk_cost(a, &scratch, &cost); + mc_chunk_cost_add(&next, &cost); + + if (body.size > 0 && mc_chunk_cost_exceeded(&next) && boundary) { + mc_compose_chunk(&actions, &body, chunk_id == 0, false, + chunk_id + 1, mc->tunnel_key, ctx->stage, + resume_next, next_table); + mc_add_chunk_flow(mc, ctx, chunk_id, &actions, flow_table); + chunk_id++; + ofpbuf_clear(&body); + last = NULL; + memset(&sum, 0, sizeof sum); + next = cost; + } + ofpbuf_put(&body, a, OFPACT_ALIGN(a->len)); + sum = next; + last = a; + } + + mc_compose_chunk(&actions, &body, chunk_id == 0, true, 0, + mc->tunnel_key, ctx->stage, resume_next, next_table); + mc_add_chunk_flow(mc, ctx, chunk_id, &actions, flow_table); + + ofpbuf_uninit(&body); + ofpbuf_uninit(&actions); + ofpbuf_uninit(&scratch); +} + +/* Install the multicast output in 'ctx', splitting it across several flows + * if it is too large for one. When 'resume_next' is true the last slice + * resubmits to 'next_table'. */ +static void +mc_install_flows(const struct sbrec_multicast_group *mc, + struct mc_flow_ctx *ctx, bool resume_next, uint8_t next_table, + struct ovn_desired_flow_table *flow_table) +{ + size_t plain = ctx->ofpacts.size; + + if (resume_next) { + put_resubmit(next_table, &ctx->ofpacts); + } + if (mc_fits_one_chunk(&ctx->ofpacts)) { + mc_ofctrl_add_flow(mc, ctx, flow_table); + return; + } + + if (!ovs_feature_is_supported(OVS_RECIRC_ACTION_SUPPORT)) { + /* Nothing better to offer than the oversized flow: ovs-vswitchd may + * reject it, or install it and silently drop the destinations that + * do not fit. */ + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); + VLOG_WARN_RL(&rl, "Multicast group %s is too large for one OpenFlow " + "flow, and ovs-vswitchd does not support the recirc " + "action needed to split it. The flood may not reach " + "every port.", mc->name); + mc_ofctrl_add_flow(mc, ctx, flow_table); + return; + } + + ofpbuf_truncate(&ctx->ofpacts, plain); + mc_split_flows(mc, ctx, resume_next, next_table, flow_table); + ofpbuf_clear(&ctx->ofpacts); +} + static void consider_mc_group(const struct physical_ctx *ctx, const struct sbrec_multicast_group *mc, @@ -3118,7 +3358,7 @@ consider_mc_group(const struct physical_ctx *ctx, if (local_lports) { put_load(mc->tunnel_key, MFF_LOG_OUTPORT, 0, 32, &local_ctx->ofpacts); - mc_ofctrl_add_flow(mc, local_ctx, flow_table); + mc_install_flows(mc, local_ctx, false, 0, flow_table); } if (remote_ports) { @@ -3150,14 +3390,14 @@ consider_mc_group(const struct physical_ctx *ctx, remote_ports = remote_ctx->ofpacts.size > 0; if (remote_ports) { - put_resubmit(OFTABLE_REMOTE_VTEP_OUTPUT, &remote_ctx->ofpacts); - mc_ofctrl_add_flow(mc, remote_ctx, flow_table); + mc_install_flows(mc, remote_ctx, true, OFTABLE_REMOTE_VTEP_OUTPUT, + flow_table); } if (ramp_ports && has_vtep) { put_load(mc->tunnel_key, MFF_LOG_OUTPORT, 0, 32, &ramp_ctx->ofpacts); - put_resubmit(OFTABLE_REMOTE_VTEP_OUTPUT, &ramp_ctx->ofpacts); - mc_ofctrl_add_flow(mc, ramp_ctx, flow_table); + mc_install_flows(mc, ramp_ctx, true, OFTABLE_REMOTE_VTEP_OUTPUT, + flow_table); } for (size_t i = 0; i < MC_FLOWS_MAX; i++) { diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h index de81a7fe255a..433ba35d168f 100644 --- a/include/ovn/logical-fields.h +++ b/include/ovn/logical-fields.h @@ -144,6 +144,7 @@ enum mff_log_flags_bits { MLF_RECIRC_BIT = 24, MLF_EVPN_LOOKUP_BIT = 25, MLF_POST_NF_LOOP_BACK_BIT = 26, + MLF_MC_CHUNK_BIT = 27, MLF_NETWORK_ID_START_BIT = 28, MLF_NETWORK_ID_END_BIT = 31, }; @@ -224,6 +225,10 @@ enum mff_log_flags { /* Set on a post-NF packet going back to the original ingress port. */ MLF_POST_NF_LOOP_BACK = (1 << MLF_POST_NF_LOOP_BACK_BIT), + /* Set on a multicast flood that was split across several OpenFlow + * flows, for the table lookup of the next chunk only. */ + MLF_MC_CHUNK = (1 << MLF_MC_CHUNK_BIT), + /* Assign network ID to packet to choose correct network for snat when * lb_force_snat_ip=router_ip. */ MLF_NETWORK_ID = (OVN_MAX_NETWORK_ID << MLF_NETWORK_ID_START_BIT), diff --git a/ovn-architecture.7.xml b/ovn-architecture.7.xml index 00728ee9b7e6..8f6739ee9ad2 100644 --- a/ovn-architecture.7.xml +++ b/ovn-architecture.7.xml @@ -1553,6 +1553,16 @@ multicast group includes a logical port or ports on the local hypervisor, then its actions also resubmit to <code>OFTABLE_LOCAL_OUTPUT</code>. + A multicast group with enough ports is installed as several flows + in this table, or in <code>OFTABLE_LOCAL_OUTPUT</code>. Each flow + carries one slice of the outputs and, except the last, ends with + <code>recirc</code> and <code>resubmit</code> back to the same + table. The continuation matches <code>MLF_MC_CHUNK</code> and + <code>reg0</code>. <code>reg0</code> is saved on the stack across + that lookup and restored before the next slice runs, and + <code>MLF_MC_CHUNK</code> is cleared before the slice runs the + egress pipeline. <code>resubmit</code> alone would be composed + into one datapath flow and hit the action-list limit. <code>OFTABLE_REMOTE_OUTPUT</code> also includes: </p> diff --git a/tests/ovn.at b/tests/ovn.at index 780149467448..823fd6ccd3e7 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -47167,6 +47167,273 @@ OVN_CLEANUP([hv1]) AT_CLEANUP ]) +OVN_FOR_EACH_NORTHD([ +AT_SETUP([broadcast to many local ports stays in the datapath]) +ovn_start + +dnl A local port is delivered to by resubmitting into the egress +dnl pipeline, which costs about 19 more resubmits. A full flood of +dnl roughly 215 ports therefore reaches OVS's 4096-resubmit limit and +dnl translation aborts, dropping the broadcast. That limit is enforced +dnl while OVS translates the flow, so it applies to both the kernel and +dnl the userspace datapath. 280 ports is past that point. The split +dnl installs several table-56 flows chained by recirc, and the trace of +dnl a broadcast must still reach every port. +check ovn-nbctl ls-add ls1 + +nbctl_cmd="" +for i in $(seq 1 280); do + idx=$(printf "%04d" $i) + hex_hi=$(printf "%02x" $(($i / 256))) + hex_lo=$(printf "%02x" $(($i % 256))) + mac="f0:00:00:00:${hex_hi}:${hex_lo}" + nbctl_cmd="$nbctl_cmd -- lsp-add ls1 vif-$idx" + nbctl_cmd="$nbctl_cmd -- lsp-set-addresses vif-$idx \"$mac\"" +done +eval check ovn-nbctl $nbctl_cmd + +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 + +vsctl_cmd="" +for i in $(seq 1 280); do + idx=$(printf "%04d" $i) + vsctl_cmd="$vsctl_cmd -- add-port br-int vif-$idx" + vsctl_cmd="$vsctl_cmd -- set Interface vif-$idx external-ids:iface-id=vif-$idx" +done +eval check ovs-vsctl $vsctl_cmd + +OVN_POPULATE_ARP +wait_for_ports_up +check ovn-nbctl --wait=hv sync + +dnl At least one table-56 flow chains to the next slice with recirc. +dnl grep -c exits 1 when the count is zero. +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=OFTABLE_LOCAL_OUTPUT | + grep -c recirc], [0], + [ignore]) + +in_ofport=$(as hv1 ovs-vsctl get Interface vif-0001 ofport) +flow="in_port=$in_ofport,dl_src=f0:00:00:00:00:01,dl_dst=ff:ff:ff:ff:ff:ff,arp,arp_op=1,arp_spa=10.0.0.1,arp_tpa=10.0.0.250,arp_sha=f0:00:00:00:00:01,arp_tha=ff:ff:ff:ff:ff:ff" +dnl ofproto/trace stops at recirc. Follow each continuation immediately, +dnl while the recirculation context is still allocated. +: > all.trace +rid=0 +for _ in $(seq 0 8); do + if test "$rid" = 0; then + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow"], [0], [stdout]) + else + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow,recirc_id=$rid"], [0], [stdout]) + fi + cat stdout >> all.trace + rid=$(grep 'recirc(0x' stdout | tail -1 | sed 's/.*recirc(0x//;s/).*//') + if test -z "$rid"; then + break + fi +done +cp all.trace stdout +AT_CHECK([grep "Translation failed" stdout], [1]) +AT_CHECK([grep "over 4096 resubmit" stdout], [1]) +AT_CHECK([grep "over 64 kB of actions" stdout], [1]) +AT_CHECK([grep "handled by the userspace slow path" stdout], [1]) +AT_CHECK([grep "Recirculation context not found" stdout], [1]) +dnl The flood is split, so a later slice is reached only through recirc. +AT_CHECK([test $(grep -c "Restoring actions" stdout) -ge 1]) +dnl Every port except the ingress port is output across the slices. +AT_CHECK([test $(grep -c "output:" stdout) -ge 279]) + +OVN_CLEANUP([hv1]) +AT_CLEANUP +]) + +AT_SETUP([broadcast to many remote chassis stays in the datapath]) +ovn_start +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 16 + +dnl A remote destination costs one output action, but that output pushes +dnl a tunnel header, so it becomes about 100 datapath bytes. Past roughly +dnl 710 chassis the action list crosses the 16-bit Netlink attribute limit. +dnl Translation then stops partway through and the remaining chassis are +dnl dropped from the flow without any error, so those nodes silently never +dnl receive the broadcast. 800 chassis is past that point. +N=800 + +check ovn-nbctl ls-add ls1 +check ovn-nbctl lsp-add ls1 vif-local \ + -- lsp-set-addresses vif-local "f0:00:00:00:00:01" +check ovs-vsctl add-port br-int vif-local \ + -- set Interface vif-local external-ids:iface-id=vif-local + +nbctl_cmd="" +sbctl_cmd="" +for i in $(seq 1 $N); do + a=$(($i / 250 + 1)); b=$(($i % 250 + 1)) + hi=$(printf "%02x" $(($i / 256))); lo=$(printf "%02x" $(($i % 256))) + nbctl_cmd="$nbctl_cmd -- lsp-add ls1 rp-$i" + nbctl_cmd="$nbctl_cmd -- lsp-set-addresses rp-$i \"f0:00:01:00:$hi:$lo\"" + sbctl_cmd="$sbctl_cmd -- chassis-add rch-$i geneve 192.168.$a.$b" +done +eval check ovn-nbctl $nbctl_cmd +eval check ovn-sbctl $sbctl_cmd +check ovn-nbctl --wait=sb sync + +bind_cmd="" +for i in $(seq 1 $N); do + bind_cmd="$bind_cmd -- lsp-bind rp-$i rch-$i" +done +eval check ovn-sbctl $bind_cmd + +dnl Resolve the tunnel next hops so every output actually encapsulates. +for i in $(seq 1 $N); do + a=$(($i / 250 + 1)); b=$(($i % 250 + 1)) + hi=$(printf "%02x" $(($i / 256))); lo=$(printf "%02x" $(($i % 256))) + as hv1 ovs-appctl tnl/neigh/set br-phys 192.168.$a.$b f0:00:02:00:$hi:$lo +done + +check ovn-nbctl --wait=hv sync +OVS_WAIT_UNTIL([test $(as hv1 ovs-vsctl list-ports br-int | + grep -c '^ovn-rch') -ge $N]) +check ovn-nbctl --wait=hv sync + +dnl The remote output is split, so at least one table-54 flow chains to +dnl the next slice with recirc. grep -c exits 1 when the count is zero. +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=OFTABLE_REMOTE_OUTPUT | + grep -c recirc], [0], [ignore]) + +in_ofport=$(as hv1 ovs-vsctl get Interface vif-local ofport) +flow="in_port=$in_ofport,dl_src=f0:00:00:00:00:01,dl_dst=ff:ff:ff:ff:ff:ff,arp,arp_op=1,arp_spa=10.0.0.1,arp_tpa=10.0.0.250,arp_sha=f0:00:00:00:00:01,arp_tha=ff:ff:ff:ff:ff:ff" +dnl ofproto/trace stops at recirc. Follow each continuation immediately, +dnl while the recirculation context is still allocated. +: > all.trace +rid=0 +for _ in $(seq 0 8); do + if test "$rid" = 0; then + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow"], [0], [stdout]) + else + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow,recirc_id=$rid"], [0], [stdout]) + fi + cat stdout >> all.trace + rid=$(grep 'recirc(0x' stdout | tail -1 | sed 's/.*recirc(0x//;s/).*//') + if test -z "$rid"; then + break + fi +done +cp all.trace stdout + +AT_CHECK([grep "Translation failed" stdout], [1]) +AT_CHECK([grep "over 4096 resubmit" stdout], [1]) +AT_CHECK([grep "over 64 kB of actions" stdout], [1]) +AT_CHECK([grep "handled by the userspace slow path" stdout], [1]) +AT_CHECK([grep "Recirculation context not found" stdout], [1]) +dnl Every chassis is encapsulated to across the slices. +AT_CHECK([test $(grep -o 'dst=192\.168\.[[0-9]]*\.[[0-9]]*' stdout | + sort -u | wc -l) -eq $N]) + +OVN_CLEANUP([hv1]) +AT_CLEANUP + +AT_SETUP([broadcast to many remote ports stays in the datapath]) +ovn_start +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.11 16 + +dnl Tunnels to a chassis in another availability zone are only built when +dnl this chassis is an interconnection gateway. +check ovs-vsctl set open . external_ids:ovn-is-interconn=true + +dnl A type=remote port models a port in another availability zone. Unlike +dnl a port on a peer chassis in this zone, it cannot share an output with +dnl its neighbours: the remote zone has no copy of this multicast group, so +dnl the destination's own tunnel key has to go into the Geneve option. Each +dnl one therefore costs three field writes plus an output: about 80 bytes of +dnl OpenFlow, of which some 817 fit in a FLOW_MOD, and 92 bytes of datapath +dnl actions, of which some 712 fit in one action list. 800 remote ports is +dnl past the datapath limit, which is the one that binds here: the field +dnl writes are large in OpenFlow but fold into the tunnel header. Overrunning +dnl it drops the destinations that do not fit, without an error. +N=800 + +check ovn-sbctl chassis-add rch geneve 192.168.0.12 \ + -- set chassis rch other_config:is-remote=true + +check ovn-nbctl ls-add ls1 +check ovn-nbctl lsp-add ls1 vif-local \ + -- lsp-set-addresses vif-local "f0:00:00:00:00:01" +check ovs-vsctl add-port br-int vif-local \ + -- set Interface vif-local external-ids:iface-id=vif-local + +dnl Every remote port is encapsulated to individually, so they can all sit +dnl behind one remote chassis. +nbctl_cmd="" +for i in $(seq 1 $N); do + hi=$(printf "%02x" $(($i / 256))); lo=$(printf "%02x" $(($i % 256))) + nbctl_cmd="$nbctl_cmd -- lsp-add ls1 rp-$i" + nbctl_cmd="$nbctl_cmd -- lsp-set-type rp-$i remote" + nbctl_cmd="$nbctl_cmd -- lsp-set-options rp-$i requested-chassis=rch" + nbctl_cmd="$nbctl_cmd -- lsp-set-addresses rp-$i \"f0:00:01:00:$hi:$lo\"" +done +eval check ovn-nbctl $nbctl_cmd +check ovn-nbctl --wait=sb sync + +dnl Resolve the tunnel next hop so the outputs actually encapsulate. +as hv1 ovs-appctl tnl/neigh/set br-phys 192.168.0.12 f0:00:02:00:00:01 +OVS_WAIT_UNTIL([as hv1 ovs-vsctl list-ports br-int | grep -q '^ovn-rch']) +check ovn-nbctl --wait=hv sync + +dnl The flood is split, so at least one table-54 flow chains to the next +dnl slice with recirc. grep -c exits 1 when the count is zero. +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=OFTABLE_REMOTE_OUTPUT | + grep 'reg15=0x8000' | grep -c recirc], [0], [ignore]) + +dnl Nothing was dropped for being too big to install. +AT_CHECK([grep "FLOW_MOD message is too big" hv1/ovn-controller.log], [1]) + +in_ofport=$(as hv1 ovs-vsctl get Interface vif-local ofport) +flow="in_port=$in_ofport,dl_src=f0:00:00:00:00:01,dl_dst=ff:ff:ff:ff:ff:ff,arp,arp_op=1,arp_spa=10.0.0.1,arp_tpa=10.0.0.250,arp_sha=f0:00:00:00:00:01,arp_tha=ff:ff:ff:ff:ff:ff" +dnl ofproto/trace stops at recirc. Follow each continuation immediately, +dnl while the recirculation context is still allocated. Nine iterations is +dnl all the userspace datapath would execute, so a flood needing more +dnl slices than this fails the destination count below. +: > all.trace +rid=0 +for _ in $(seq 0 8); do + if test "$rid" = 0; then + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow"], [0], [stdout]) + else + AT_CHECK([as hv1 ovs-appctl ofproto/trace br-int "$flow,recirc_id=$rid"], [0], [stdout]) + fi + cat stdout >> all.trace + rid=$(grep 'recirc(0x' stdout | tail -1 | sed 's/.*recirc(0x//;s/).*//') + if test -z "$rid"; then + break + fi +done +cp all.trace stdout + +AT_CHECK([grep "Translation failed" stdout], [1]) +AT_CHECK([grep "over 4096 resubmit" stdout], [1]) +AT_CHECK([grep "over 64 kB of actions" stdout], [1]) +AT_CHECK([grep "handled by the userspace slow path" stdout], [1]) +AT_CHECK([grep "Recirculation context not found" stdout], [1]) +dnl Every remote port is encapsulated to across the slices. They share one +dnl tunnel, so the destinations are told apart by the Geneve option that +dnl carries the destination's own tunnel key. +AT_CHECK([test $(grep -o 'len=4,0x[[0-9a-f]]*' stdout | + sort -u | wc -l) -eq $N]) + +OVN_CLEANUP([hv1]) +AT_CLEANUP + OVN_FOR_EACH_NORTHD([ AT_SETUP([Port security - VRRPv3 ARP/ND]) CHECK_SCAPY -- 2.38.1 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
