This RFC adds IPsec ESP tunnels to the userspace datapath, so that
OVS-DPDK can encrypt traffic in its PMD threads. Today, IPsec in OVS
relies on the kernel's xfrm, which the userspace datapath cannot use
without sending traffic through the kernel.
I would like feedback on the design and interfaces before polishing it.
The questions are below.
What it does
============
A new 'esp' tunnel port carries IPv4 and IPv6 packets in ESP tunnel mode
(RFC 4303) with AES-GCM (RFC 4106), over IPv4 or IPv6:
$ ovs-vsctl add-port br0 esp0 -- set int esp0 type=esp \
options:remote_ip=192.168.25.2 \
options:esp_keying=ike options:esp_if_id=7
- Keys come from an IKE daemon through new 'esp/*' unixctl commands, or
are configured manually.
- It supports Extended Sequence Numbers, anti-replay, UDP encapsulation
for NAT traversal, and software TSO.
- Rekeying swaps the outbound SA atomically, without changing datapath
flows.
- AES-GCM uses the Intel IPsec Multi-Buffer library if available at
build time, and OpenSSL otherwise.
For IKE, I wrote a strongSwan plugin, 'kernel-ovs', which implements
strongSwan's kernel_ipsec_t over the unixctl socket:
https://github.com/trajs/strongswan/tree/kernel-ovs
It is not upstream in strongSwan. It builds with --enable-kernel-ovs on
that branch, based on 5.9.13. My tests ran it as a separately built
module in Ubuntu's strongSwan 5.9.13.
The OVS series is also at:
https://github.com/trajs/ovs/tree/ipsec-dpdk
Patches
=======
1 netdev: push_header() may now fail, for all tunnel types.
2 The ESP library: SAs, AES-GCM, anti-replay.
3 The 'esp' tunnel port.
4 Per-SA statistics and 'esp/show'.
5 Software TSO for ESP.
6 UDP encapsulation.
7 unixctl commands with secret arguments.
8 IKE keying through 'esp/*' commands.
9 Documentation.
10 IPsec Multi-Buffer library backend.
11 Per-thread packet counts, so one SA scales across PMDs.
12 Reporting of missing SAs.
Patch 1 changes shared code and could go in by itself.
Results
=======
One PMD thread, one SA, Intel Core Ultra 9 285HX (AVX2), E810 25G, with
pktgen as the traffic source. The numbers are averages of two or three
runs, and were measured before patch 11, which changes the per-packet
counters. Its effect is shown below.
ESP alone, by crypto backend (also in patch 10):
64 bytes 512 bytes 1400 bytes
encrypt
OpenSSL 3.7 Mpps 9.4 Gbit/s 16.6 Gbit/s
IPsec-MB 6.0 Mpps 13.6 Gbit/s 19.3 Gbit/s
decrypt
OpenSSL 4.0 Mpps 9.5 Gbit/s 18.7 Gbit/s
IPsec-MB 6.5 Mpps 12.2 Gbit/s 18.9 Gbit/s
Geneve in ESP on one core, against kernel OVS with xfrm on the same
packets. Kernel encapsulation is at full rate. Kernel decapsulation
livelocks at full line rate, so its figures are the best lossless rate
from a sweep of the offered rate:
OVS-DPDK kernel
encap 64 5.02 Mpps 0.88 Mpps 5.7x
512 2.78 Mpps 11.4 Gbit/s 0.77 Mpps 3.6x
1400 1.66 Mpps 18.6 Gbit/s 0.66 Mpps 2.5x
decap 64 4.62 Mpps 0.80 Mpps 5.8x
512 2.47 Mpps 10.1 Gbit/s 0.70 Mpps 3.5x
1400 1.60 Mpps 18.0 Gbit/s 0.60 Mpps 2.7x
Round-trip latency added by ESP on top of Geneve, median:
OVS-DPDK kernel
64 +0.1 us +1.9 us
512 +0.7 us +2.2 us
1400 +0.9 us +2.3 us (1350 bytes)
The OVS-DPDK latency used pktgen probes over vhost-user and the kernel
used sockperf over sockets, so only the added cost is comparable, not the
totals. The OVS-DPDK 1400-byte figure is at 5% load, the others at 1%.
Encrypting 64-byte packets with two PMDs sharing one outbound SA, with
and without patch 11, in a later run with a different measurement, so
compare these only with each other:
before after
1 PMD 6.4 Mpps 6.6 Mpps
2 PMDs 9.5 Mpps 11.2 Mpps
That is about 1.7 times one PMD; the sequence number stays shared, and
pktgen tops out near 12.7 Mpps. Decryption with several PMDs and many
SAs have not been measured. Raw ESP has no port numbers, so I expect
one tunnel to land on one receive queue.
Testing
=======
On this series, based on daafbd0a2:
make check passed on the first and last patches,
10 skipped
each patch builds with --enable-Werror, and passes the
tunnel and ESP tests
check-system-userspace 166 passed, 56 skipped
check-system-tso 147 passed, 49 skipped
check-kernel 173 passed, 52 skipped (ESP tests skip)
check-dpdk 155 passed, 68 skipped
check-valgrind ESP tests only: no leaks or invalid accesses
build variants IPsec-MB and OpenSSL; OpenSSL only; no crypto
other builds each patch with sparse, clang and 32-bit
make distcheck passed
sanitizers clang address and undefined: make check and the
ESP system tests passed, with no reports
clang analyzer no new reports against main
The system-test counts are from just before the last build-only fixes.
GitHub Actions passes on the final patches: all jobs, including macOS,
32-bit, sanitizers and system tests, and FreeBSD:
https://github.com/trajs/ovs/actions/runs/41384870082
By hand on E810 ports, against Linux kernel IPsec with manual keys and
with a second strongSwan: traffic, rekeys under traffic, ESN, several
CHILD_SAs, and restarts of ovs-vswitchd and charon.
A few tests are intermittent on my machine. 'conntrack - limit by zone'
and 'tunnel - table version' also fail occasionally on clean main.
'flow resume with geneve tun_metadata' failed once in check-dpdk and did
not recur in later runs on this series or on main. The system tests
were run in a separate network namespace.
Known limitations
=================
- Decrypted packets are not checked against the traffic selectors that
IKE negotiated, which RFC 4301 section 5.2 requires. OpenFlow rules
must enforce them. See question 1.
- Keys are wiped from the SA, the command's own copies and unixctl's
copies, but not from the jsonrpc receive buffer, the per-thread
OpenSSL contexts, or the IPsec-MB context on the stack.
- AES-GCM only. No fragmentation or reassembly of ESP packets.
- No MOBIKE or NAT mapping changes (the plugin's update_sa is not
implemented).
- With UDP encapsulation, IKE cannot share the port on that address.
- One ovs-vswitchd serves one kernel-ovs charon, which flushes the
IKE-keyed SAs when it starts.
- IPsec-MB is not added to the packaging or CI.
Questions
=========
1. Traffic selectors: is "OpenFlow is the policy" acceptable if it is
documented, or should the datapath enforce selectors, at the cost of
a lookup per packet?
2. Is the Intel IPsec Multi-Buffer library acceptable as an optional
dependency? It gains up to 61% over OpenSSL at small packets.
3. How should this relate to ovs-monitor-ipsec? Should it learn to
drive these tunnels, or do they stay separate?
4. Is unixctl the right interface for an IKE daemon? The commands and
secret arguments are new, and the keys cross a local socket.
5. Should push_header() returning an error (patch 1) go in on its own?
Parts of this series were written with an AI assistant, as the
Assisted-by tags show. I have reviewed it and take responsibility for it.
Thilak Raj Surendra Babu (12):
netdev: Allow tunnel push_header() to fail.
esp: Add IPsec ESP security associations with AES-GCM.
userspace: Add IPsec ESP tunnel support.
esp: Add per-SA statistics and the 'esp/show' command.
netdev: Segment TSO packets in software for ESP tunnels.
userspace: Add UDP encapsulation of ESP for NAT traversal.
unixctl: Allow commands with secret arguments.
esp: Let an IKE daemon install the security associations.
Documentation: Describe esp tunnels keyed by IKE.
esp: Use the Intel IPsec Multi-Buffer library for AES-GCM.
esp: Count packets per thread, to scale across PMDs.
esp: Report missing security associations.
Documentation/faq/configuration.rst | 59 ++
Documentation/faq/releases.rst | 1 +
Documentation/intro/install/general.rst | 5 +
NEWS | 13 +
configure.ac | 1 +
include/linux/openvswitch.h | 1 +
lib/automake.mk | 5 +
lib/dp-packet-gso.c | 7 +-
lib/dpif-netlink-rtnl.c | 4 +
lib/dpif-netlink.c | 5 +
lib/esp-ctl.c | 607 ++++++++++++++
lib/esp-ctl.h | 34 +
lib/esp.c | 1013 +++++++++++++++++++++++
lib/esp.h | 129 +++
lib/netdev-native-tnl.c | 221 ++++-
lib/netdev-native-tnl.h | 21 +-
lib/netdev-provider.h | 13 +-
lib/netdev-vport-private.h | 7 +
lib/netdev-vport.c | 515 +++++++++++-
lib/netdev-vport.h | 7 +
lib/netdev.c | 29 +-
lib/netdev.h | 10 +
lib/odp-util.c | 43 +
lib/packets.h | 4 +-
lib/tnl-ports.c | 11 +-
lib/unixctl.c | 37 +-
lib/unixctl.h | 3 +
lib/util.c | 12 +
lib/util.h | 1 +
m4/openvswitch.m4 | 33 +
ofproto/ofproto-dpif-xlate.c | 21 +-
ofproto/ofproto-tnl-unixctl.man | 31 +
python/ovs/flow/odp.py | 7 +
python/ovs/tests/test_odp.py | 15 +
tests/automake.mk | 1 +
tests/library.at | 6 +
tests/odp.at | 5 +
tests/system-kmod-macros.at | 8 +
tests/system-traffic.at | 108 +++
tests/system-userspace-macros.at | 9 +
tests/test-esp.c | 420 ++++++++++
tests/tunnel-push-pop-ipv6.at | 54 ++
tests/tunnel-push-pop.at | 506 +++++++++++
tests/tunnel.at | 84 ++
vswitchd/vswitch.xml | 186 ++++-
45 files changed, 4266 insertions(+), 46 deletions(-)
create mode 100644 lib/esp-ctl.c
create mode 100644 lib/esp-ctl.h
create mode 100644 lib/esp.c
create mode 100644 lib/esp.h
create mode 100644 tests/test-esp.c
--
2.43.0
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
