RE: [RFC TLS Offload Support 00/15] cover letter

[Boris Pismenny]

> >> TLS Tx crypto offload is a new feature of network devices. It enables
> >> the kernel TLS socket to skip encryption and authentication
> >> operations on the transmit side of the data path, delegating those to
> >> the NIC. In turn, the NIC encrypts packets that belong to an
> >> offloaded TLS socket on the fly. The NIC does not modify any packet
> >> headers. It expects to receive fully framed TCP packets with TLS
> >> records as payload. The NIC replaces plaintext with ciphertext and
> >> fills the authentication tag. The NIC does not hold any state beyond
> >> the context needed to encrypt the next expected packet, i.e. expected
> >> TCP sequence number and crypto state.
> >
> > It seems like, since you do the TLS framing in TCP and the card is
> > expecting to fill in certain aspects, there is a requirement that the
> > packet contents aren't mangled between the TLS framing code and when
> > the SKB hits the card.
> >
> > Is this right?
> >
> > For example, what happens if netfilter splits a TLS Tx offloaded frame
> > into two TCP segments?
We maintain the crypto context by tracking TCP sequence numbers, splitting
TCP segments is not a problem. Even if reordering is introduced anywhere
between TCP and the driver, we can identify it according to the TCP
sequence number and handle it gracefully – see mlx_tls_tx_handler.

> 
> Furthermore, it doesn't seem to work with bonding or any other virtual
> interface, which could move the skb's to be processed on another NIC, as the
> context is put onto the NIC. Even a redirect can not be processed anymore
> (seems like those patches try to stick the connection to an interface anyway).
> 
> Wouldn't it be possible to keep the state in software and push down a
> security context per skb, which get applied during sending? If not possible 
> via
> hw, slowpath can encrypt packet in sw.
We do have all the state required to encrypt a TLS packet in software. But,
pushing down the state for each skb is too expansive, because the state
depends on all data in the TLS record, essentially it requires to resend the
record up to that skb. This is accomplished for OOO packets in the
“handle_ooo” function in mlx_tls.

Maybe we could use that functionality to handle bonding, but at first it would
be easier to prevent it.

The slowpath you’ve mentioned is tricky, because you need to decide in
advance that a TLS record will use the slowpath, because after the plaintext
TLS record is pushed into TCP it is difficult to fallback to software crypto.
> 
> Also sticking connections to outgoing interfaces might work for TX, but you
> can't force the interface where packets come in.
Right. This RFC handles only TX offload.
> 
> Bye,
> Hannes

Thanks,
Boris

Re: [RFC TLS Offload Support 00/15] cover letter

[Hannes Frederic Sowa]

Hello,

On 29.03.2017 19:41, David Miller wrote:
> From: Aviad Yehezkel 
> Date: Tue, 28 Mar 2017 16:26:17 +0300
> 
>> TLS Tx crypto offload is a new feature of network devices. It
>> enables the kernel TLS socket to skip encryption and authentication
>> operations on the transmit side of the data path, delegating those
>> to the NIC. In turn, the NIC encrypts packets that belong to an
>> offloaded TLS socket on the fly. The NIC does not modify any packet
>> headers. It expects to receive fully framed TCP packets with TLS
>> records as payload. The NIC replaces plaintext with ciphertext and
>> fills the authentication tag. The NIC does not hold any state beyond
>> the context needed to encrypt the next expected packet,
>> i.e. expected TCP sequence number and crypto state.
> 
> It seems like, since you do the TLS framing in TCP and the card is
> expecting to fill in certain aspects, there is a requirement that the
> packet contents aren't mangled between the TLS framing code and when
> the SKB hits the card.
> 
> Is this right?
> 
> For example, what happens if netfilter splits a TLS Tx offloaded frame
> into two TCP segments?

Furthermore, it doesn't seem to work with bonding or any other virtual
interface, which could move the skb's to be processed on another NIC, as
the context is put onto the NIC. Even a redirect can not be processed
anymore (seems like those patches try to stick the connection to an
interface anyway).

Wouldn't it be possible to keep the state in software and push down a
security context per skb, which get applied during sending? If not
possible via hw, slowpath can encrypt packet in sw.

Also sticking connections to outgoing interfaces might work for TX, but
you can't force the interface where packets come in.

Bye,
Hannes

Re: [RFC TLS Offload Support 00/15] cover letter

[David Miller]

From: Aviad Yehezkel 
Date: Tue, 28 Mar 2017 16:26:17 +0300

> TLS Tx crypto offload is a new feature of network devices. It
> enables the kernel TLS socket to skip encryption and authentication
> operations on the transmit side of the data path, delegating those
> to the NIC. In turn, the NIC encrypts packets that belong to an
> offloaded TLS socket on the fly. The NIC does not modify any packet
> headers. It expects to receive fully framed TCP packets with TLS
> records as payload. The NIC replaces plaintext with ciphertext and
> fills the authentication tag. The NIC does not hold any state beyond
> the context needed to encrypt the next expected packet,
> i.e. expected TCP sequence number and crypto state.

It seems like, since you do the TLS framing in TCP and the card is
expecting to fill in certain aspects, there is a requirement that the
packet contents aren't mangled between the TLS framing code and when
the SKB hits the card.

Is this right?

For example, what happens if netfilter splits a TLS Tx offloaded frame
into two TCP segments?

[RFC TLS Offload Support 00/15] cover letter

[Aviad Yehezkel]

Overview

A kernel TLS Tx only socket option for TCP sockets.
Similarly to the kernel TLS socket(https://lwn.net/Articles/665602),
only symmetric crypto is done in the kernel, as well as TLS record framing.
The handshake remains in userspace, and the negotiated cipher keys/iv are 
provided to the TCP socket.

Today, userspace TLS must perform 2 passes over the data. First, it has to 
encrypt the data. Second, the data is copied to the TCP socket in the kernel.
Kernel TLS avoids one pass over the data by encrypting the data from userspace 
pages into kernelspace buffers.

Non application-data TLS records must be encrypted using the latest crypto 
state available in the kernel. It is possible to get the crypto context from 
the kernel and encrypt such recrods in user-space. But we choose to encrypt 
such TLS records in the kernel by setting the MSG_OOB flag and providing the 
record type with the data.

TLS Tx crypto offload is a new feature of network devices. It enables the 
kernel TLS socket to skip encryption and authentication operations on the 
transmit side of the data path, delegating those to the NIC. In turn, the NIC 
encrypts packets that belong to an offloaded TLS socket on the fly. The NIC 
does not modify any packet headers. It expects to receive fully framed TCP 
packets with TLS records as payload. The NIC replaces plaintext with ciphertext 
and fills the authentication tag. The NIC does not hold any state beyond the 
context needed to encrypt the next expected packet, i.e. expected TCP sequence 
number and crypto state.

There are 2 flows for TLS Tx offload, a fast path and a slow path.
Fast path: packet matches the expected TCP sequence number in the context.
Slow path: packet does not match the expected TCP sequence number in the 
context. For example: TCP retransmissions. For a packet in the slow path, we 
need to resynchronize the crypto context of the NIC by providing the TLS record 
data for that packet before it could be encrypted and transmitted by the NIC.

Motivation
==
1) Performance: The CPU overhead of encryption in the data path is high, at 
least 4x for netperf over TLS between 2 machines connected back-to-back.
Our single stream performance tests show that using crypto offload for TLS 
sockets achieves the same throughput as plain TCP traffic while increasing CPU 
utilization by only
 x1.4.

2) Flexibility: The protocol stack is implemented entirely on the host CPU.
Compared to solutions based on TCP offload, this approach offloads only 
encryption. Keeping memory management, congestion control, etc. in the host CPU.

Notes
=
1) New paths:
o net/tls - TLS layer in kernel
o drivers/net/ethernet/mellanox/accelerator/* - NIC driver support, 
currently implemented as seperated modules.
  In the future this code will go into the mlx5 driver. We attached to this 
patch only the module that integrated with TLS layer.
  The complete NIC sample driver is available at 
https://github.com/Mellanox/tls-offload/tree/tx_rfc_v5

2) We implemented support for this API in OpenSSL 1.1.0, the code is available 
at https://github.com/Mellanox/tls-openssl/tree/master

3) TLS crypto offload was presented during netdevconf1.2, more details could be 
found in the presentation and paper:
   https://netdevconf.org/1.2/session.html?boris-pismenny

4) These RFC patches are based on kernel 4.9-rc7.

Aviad Yehezkel (5):
  tcp: export do_tcp_sendpages function
  tcp: export tcp_rate_check_app_limited function
  tcp: Add TLS socket options for TCP sockets
  tls: tls offload support
  mlx/tls: Enable MLX5_CORE_QP_SIM mode for tls

Dave Watson (2):
  crypto: Add gcm template for rfc5288
  crypto: rfc5288 aesni optimized intel routines

Ilya Lesokhin (8):
  tcp: Add clean acked data hook
  net: Add TLS offload netdevice and socket support
  mlx/mlx5_core: Allow sending multiple packets
  mlx/tls: Hardware interface
  mlx/tls: Sysfs configuration interface Configure the driver/hardware
interface via sysfs.
  mlx/tls: Add mlx_accel offload driver for TLS
  mlx/tls: TLS offload driver Add the main module entrypoints and tie
the module into the build system
  net/tls: Add software offload

 MAINTAINERS|  14 +
 arch/x86/crypto/aesni-intel_asm.S  |   6 +
 arch/x86/crypto/aesni-intel_avx-x86_64.S   |   4 +
 arch/x86/crypto/aesni-intel_glue.c | 105 ++-
 crypto/gcm.c   | 122 
 crypto/tcrypt.c|  14 +-
 crypto/testmgr.c   |  16 +
 crypto/testmgr.h   |  47 ++
 drivers/net/ethernet/mellanox/Kconfig  |   1 +
 drivers/net/ethernet/mellanox/Makefile |   1 +
 .../net/ethernet/mellanox/accelerator/tls/Kconfig  |  11 +
 .../net/ethernet/mellanox/accelerator/tls/Makefile |   4 +
 .../net/ethernet/mellanox/accelerator/tls/tls.c| 658