Hello all, I would like to introduce you to the new kernel API for TLS transmit-side data-path, and open a discussion regarding its support in OpenSSL.
This is currently a V2 patch series in Linux net-next, and it is stabilizing. Dave has been working on this for a while [1][2], and Aviad, Ilya and myself have also been working on TLS transmit-side crypto offload to a NIC [3][4]. Similar work is also being done in FreeBSD by Netflix [5][6]. Placing the data-path of TLS in the kernel has several advantages. First, an efficient TLS sendfile system call implementation is possible with such a socket. Second, data from userspace could be encrypted when passing the user-kernel boundary, reducing the overhead of existing implementations that require at least encrypting the data and then copying from user to kernel. Third, TLS data-path crypto offload to hardware requires a kernel infrastructure for TLS. To sum up, a new kernel socket option is introduced to add support for the data-path of TLS and it will soon be common in operating systems. For anyone who is interested, here is the kernel code: https://github.com/Mellanox/tls-offload - branch tx_rfc_v10 And our openssl development branch: https://github.com/Mellanox/tls-openssl - branch tls_tx_v2 The user API is as follows: (also see [7]) ========================================== Creating a TLS connection ------------------------- First create a new TCP socket and set the TLS ULP. sock = socket(AF_INET, SOCK_STREAM, 0); setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls")); Setting the TLS ULP allows us to set/get TLS socket options. Currently only the symmetric encryption is handled in the kernel. Patch 2 sets the TLS ULP in BIO_new_socket. All the parameters required to move the data-path to the kernel are available after processing the change cipher state message during the handshake. Patch 4 adds code similar to the one below in tls1_change_cipher_state. There is a separate socket option for moving the transmit and the receive into the kernel. Current code, supports only the transmit side, TLS1.2 and AES-GCM128. In the future, this will be extended. Below is the code that sets the socket option which moves the transmit data-path into the kernel. Patch 1 adds a temporary netinet/tcp.h file which includes these structures. Do you know if netinet/tcp.h is the right place for these? /* From linux/tls.h */ struct tls_crypto_info { unsigned short version; unsigned short cipher_type; }; struct tls12_crypto_info_aes_gcm_128 { struct tls_crypto_info info; unsigned char iv[TLS_CIPHER_AES_GCM_128_IV_SIZE]; unsigned char key[TLS_CIPHER_AES_GCM_128_KEY_SIZE]; unsigned char salt[TLS_CIPHER_AES_GCM_128_SALT_SIZE]; unsigned char rec_seq[TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE]; }; struct tls12_crypto_info_aes_gcm_128 crypto_info; crypto_info.info.version = TLS_1_2_VERSION; crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_128; memcpy(crypto_info.iv, iv_write, TLS_CIPHER_AES_GCM_128_IV_SIZE); memcpy(crypto_info.rec_seq, seq_number_write, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE); memcpy(crypto_info.key, cipher_key_write, TLS_CIPHER_AES_GCM_128_KEY_SIZE); memcpy(crypto_info.salt, implicit_iv_write, TLS_CIPHER_AES_GCM_128_SALT_SIZE); setsockopt(sock, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info)); Sending TLS application data ---------------------------- After setting the TLS_TX socket option all application data sent over this socket is encrypted using TLS and the parameters provided in the socket option. For example, an application can send an encrypted hello world record as follows: const char *msg = "hello world\n"; send(sock, msg, strlen(msg)); send() data is directly encrypted from the userspace buffer provided to the encrypted kernel send buffer if possible. The sendfile system call will send the file's data over TLS records of maximum length (2^14). file = open(filename, O_RDONLY); fstat(file, &stat); sendfile(sock, file, &offset, stat.st_size); TLS records are created and sent after each send() call, unless MSG_MORE is passed. MSG_MORE will delay creation of a record until MSG_MORE is not passed, or the maximum record size is reached or an alert record needs to be sent. The kernel will need to allocate a buffer for the encrypted data. This buffer is allocated at the time send() is called, such that either the entire send() call will return -ENOMEM (or block waiting for memory), or the encryption will always succeed. If send() returns -ENOMEM and some data was left on the socket buffer from a previous call using MSG_MORE, the MSG_MORE data is left on the socket buffer. Send TLS control messages ------------------------- Other than application data, TLS has control messages such as alert messages (record type 21) and handshake messages (record type 22), etc. These messages can be sent over the socket by providing the TLS record type via a CMSG. Patch 2 adds the following function (in bss_sock.c) that sends @data of @length bytes using a record of type @record_type. /* send TLS control message using record_type */ static int klts_send_ctrl_message(int sock, unsigned char record_type, void *data, size_t length) { struct msghdr msg = {0}; int cmsg_len = sizeof(record_type); struct cmsghdr *cmsg; char buf[CMSG_SPACE(cmsg_len)]; struct iovec msg_iov; /* Vector of data to send/receive into. */ msg.msg_control = buf; msg.msg_controllen = sizeof(buf); cmsg = CMSG_FIRSTHDR(&msg); cmsg->cmsg_level = SOL_TLS; cmsg->cmsg_type = TLS_SET_RECORD_TYPE; cmsg->cmsg_len = CMSG_LEN(cmsg_len); *CMSG_DATA(cmsg) = record_type; msg.msg_controllen = cmsg->cmsg_len; msg_iov.iov_base = data; msg_iov.iov_len = length; msg.msg_iov = &msg_iov; msg.msg_iovlen = 1; return sendmsg(sock, &msg, 0); } Control message data should be provided unencrypted, and will be encrypted by the kernel. At a high level, the kernel TLS ULP is a replacement for the record layer of a userspace TLS library. References: [1] - https://netdevconf.org/1.2/session.html?dave-watson [2] - https://www.spinics.net/lists/linux-crypto/msg25953.html [3] - https://netdevconf.org/1.2/session.html?boris-pismenny [4] - https://www.spinics.net/lists/netdev/msg427122.html [5] - https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf [6] - https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf [7] - https://www.spinics.net/lists/linux-crypto/msg25957.html Boris Pismenny (4): e_os: hack! TLS offload bio: Linux TLS Offload evp/e_aes: Expose GCM IV ssl: Linux TLS Tx Offload crypto/bio/bss_sock.c | 93 +++++++++++++- crypto/evp/e_aes.c | 8 ++ e_os.h | 2 +- include/netinet/tcp.h | 305 ++++++++++++++++++++++++++++++++++++++++++++++ include/openssl/bio.h | 32 +++++ include/openssl/evp.h | 1 + ssl/record/rec_layer_s3.c | 95 +++++++++++---- ssl/t1_enc.c | 69 +++++++++++ 8 files changed, 578 insertions(+), 27 deletions(-) create mode 100644 include/netinet/tcp.h -- 1.8.3.1 -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev