Hi,
I tested a simple configuration where HAProxy connects to a backend
server
using TLS 1.3 with ktls and splice options enabled.
haproxy.cfg
```
global
expose-experimental-directives
nbthread 1
defaults
mode tcp
timeout connect 1s
timeout client 1s
timeout server 1s
option splice-request
option splice-response
frontend tcp_in
bind *:8080
default_backend tls_out
backend tls_out
server server 127.0.0.1:4043 ssl verify none ktls on force-tlsv13
```
After the initial handshake phase, the server sends two
new_session_ticket
handshake messages.
HAProxy tries to receive those messages with the splice syscall which
results
in the EINVAL error.
There is a comment in the HAProxy source (in the raw_sock_to_pipe
function)
regarding such situation:
```
* EINVAL, however, is a bit special.
* If we're trying to splice from a KTLS
* socket, the kernel may return EINVAL
* to signal that the current TLS record
* is not application data, and that we
* have to call recvmsg() to get it.
* This is not really an error, and doesn't
* mean we won't be able to splice later.
* Choosing EINVAL there is a bit unfortunate,
* because it can mean many things, but we
* should not get it for any other reason.
*/
```
Using strace I've seen only repeated splice calls returning EINVAL,
without
any recvmsg call to handle the failed splice call.
```
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=429214144}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=429392356}) =
0
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=429572280}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=429753718}) =
0
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=429931308}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=430114691}) =
0
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=430520235}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=430703514}) =
0
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=430879870}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=431062900}) =
0
epoll_wait(4, [], 200, 0) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=431239849}) =
0
splice(9, NULL, 11, NULL, 1073741824, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) =
-1 EINVAL (Invalid argument)
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=431422559}) =
0
epoll_wait(4, [], 200, 0) = 0
```
I've tested versions 3.4.2 and git master, both with OpenSSL 3.5.6 and
AWS-LC 5.1.0.
I've added a simple hack in the raw_sock_to_pipe function:
```
diff --git a/src/raw_sock.c b/src/raw_sock.c
index 33fb8f5c3..1552e44bc 100644
--- a/src/raw_sock.c
+++ b/src/raw_sock.c
@@ -137,6 +137,21 @@ int raw_sock_to_pipe(struct connection *conn, void
*xprt_ctx, struct pipe *pipe,
*/
if (errno != EINVAL)
retval = -1;
+ else
+ {
+ char dbuff[1024];
+ char cbuff[64];
+ struct msghdr msg;
+ struct iovec iov;
+ memset(&msg, 0, sizeof(msg));
+ iov.iov_base = dbuff;
+ iov.iov_len = sizeof(dbuff);
+ msg.msg_control = cbuff;
+ msg.msg_controllen =
sizeof(cbuff);
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ (void)recvmsg(conn->handle.fd,
&msg, 0);
+ }
goto leave;
}
else if (errno == EINTR) {
```
And it seems to work (HAProxy receives the data from the server and
forwards
it to the client).
Of course it's not a proper solution but unfortunately I'm not familiar
enough
with the HAProxy architecture to make a proper fix.
Best regards,
Karol
