Today, for a load balancer on a logical router with a distributed gateway
port, the conntrack state of a load balanced connection lives on the chassis
that currently hosts the DGP: a backend is picked for the first packet only,
and the rest of the connection follows the existing conntrack entry. This
pins a connection to a single node, so gateway nodes cannot be scaled
horizontally - all the traffic of a VIP keeps flowing through the same
chassis.
Synchronizing conntrack state between gateways has been proposed before, but
that direction looks rather dated. This series takes the opposite one: keep
no conntrack state on the gateway at all, select a backend statelessly for
every packet, and keep the conntrack state on the node that hosts the
backend. That splits into two halves, one per patch: northd has to express
such a pipeline, and ovn-controller has to make a per-packet selection
stable across backend changes.
A new 'deferred-nat' load balancer option makes the gateway route only, and
defers the NAT to the hypervisor hosting the selected backend:
1. A packet for the VIP arrives at the gateway.
2. A backend is picked with select() in lr_in_ip_routing (priority 1050),
without touching conntrack.
3. The packet is routed straight to that backend: eth.dst becomes the
backend's MAC, ip.dst stays the VIP.
4. It goes through the ingress pipeline of the backend's switch and over
the tunnel to the node hosting the backend.
5. There the DNAT is done in the new ls_out_lb stage, with ct_lb_mark() for
that single backend, and the conntrack state is created.
6. The reply is unDNATed back to the VIP in the ingress pipeline of the
same switch, where the connection is known.
OVN already has 'use_stateless_nat', but it SNATs the reply traffic 1:1 to
the VIP, which is only correct if a backend belongs to a single VIP. Here
the translation is done per backend on the backend's own node, so a backend
may be a member of several VIPs without any ambiguity on the reply path.
The trade-off is that the packet is delivered to the backend by MAC, i.e. it
is no longer routed: every backend must sit on a logical switch directly
connected to the router that references the load balancer, and backends
reachable only via a next hop are ignored. ip_port_mappings is therefore
required, as the logical port of each backend has to be known to resolve its
MAC and its switch.
Selecting a backend for every packet only works if the selection is stable.
select() maps to an OpenFlow group, and a group is currently identified by a
hash of its whole content, so any change to the member list yields a
different group: a new group_id is allocated, the group is installed from
scratch, every logical flow that referenced the old one is rewritten, and the
old group is deleted. With per-packet selection that is not merely churn -
it reshuffles the connections already established on the backends that did
not change.
So the "select" action gets an optional "group_key" modifier. When present,
the group is identified by that key plus the parts of the group that do not
depend on its members, so its group_id no longer changes as members come and
go. The members are tracked individually as buckets, keyed by the value the
bucket loads into the result field, each with a stable bucket_id derived from
that key; ovn-controller diffs the desired buckets against the installed ones
and emits OFPGC15_INSERT_BUCKET / OFPGC15_REMOVE_BUCKET instead of replacing
the whole group. Adding or removing a backend then only affects the traffic
of that backend - provided the "hash" selection method is used, since
dp_hash reassigns traffic across all buckets whenever their number changes.
Known gaps:
- I-P is not supported for 'deferred-nat' load balancers yet: northd falls
back to a full recompute when such a load balancer changes.
- Incremental bucket updates only preserve established connections with the
"hash" selection method, i.e. selection_fields has to be set on the load
balancer. That is not great, but its seems to be no options on this point.
Alexandra Rukomoinikova (2):
controller: Update select group buckets incrementally.
northd: Add stateless load balancing with deferred NAT.
controller/ofctrl.c | 174 ++++++++-
include/ovn/actions.h | 4 +-
lib/actions.c | 130 +++++--
lib/extend-table.c | 274 +++++++++++++-
lib/extend-table.h | 47 +++
lib/ovn-util.c | 4 +-
lib/ovn-util.h | 2 +-
northd/en-lb-data.c | 12 +
northd/en-lb-data.h | 2 +
northd/en-ls-stateful.c | 3 +
northd/en-ls-stateful.h | 9 +
northd/lb.c | 19 +-
northd/lb.h | 3 +
northd/northd.c | 445 +++++++++++++++++++----
northd/northd.h | 31 +-
ovn-nb.xml | 56 ++-
tests/ovn-northd.at | 785 +++++++++++++++++++++++++++++++++++++++-
tests/ovn.at | 26 +-
tests/system-ovn.at | 372 +++++++++++++++++++
19 files changed, 2270 insertions(+), 128 deletions(-)
--
2.48.1
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
