Some tests, like tc_tunnel or tc_edt, sporadically fail in CI with the
following logs:
(network_helpers.c:309: errno: Operation now in progress) \
Failed to connect to server
send_and_test_data:FAIL:connect to server unexpected error: -115
This is due to SO_RCVTIMEO and SO_SNDTIMEO being set on the client
socket (see settimeo() in client_socket()), allowing connect() to return
an error and to set errno to EINPROGRESS instead of ETIMEDOUT.
Increasing the timeout value for those tests is likely not a good
solution (and it has already been done by commit 2790db208b44
("selftests/bpf: Improve tc_tunnel test reliability")): some tests
expect some data transfer to fail, and so the timeout value would
increase overall test execution duration again (not only the connection,
but any socket operation).
Another solution is to allocate a timeout budget specific to the
connection: we can apply a larger timeout only for connections, and once
the connection is established, set back the timeout configured through
opts->timeout_ms; this would allow connection to succeed under heavy CI
load, while keeping timeout reasonable for the rest of the test traffic.
Set a larger SO_SNDTIMEO/SO_RCVTIMEO for the connection step, and reset
it back to the timeout configured by the test once the connection has
succeeded.
Fixes: 99126abec5e5 ("bpf: selftests: A few improvements to network_helpers.c")
Signed-off-by: Alexis Lothoré (eBPF Foundation) <[email protected]>
---
Hello,
this is the v4 of the series aiming to reduce the flakyness of
tc_tunnel/tc_edt tests in CI. This revision takes a step back, based on
Ihor's tests, and drops the poll loop in favor of a bare, larger
timeout value applied only for the connection step. The main downside
of this new mechanism is a slight increase of the duration for tests
expecting a connection failure. In my testing setup (x86-based Qemu on
my work laptop), I observed a ~10s increase (on a ~5m18 base for the
whole test_progs set).
---
Changes in v4:
- dropped the polling loop in favor of a larger, connect-specific timeout
- drop timeout configuration from tc_edt test
- Link to v3:
https://patch.msgid.link/[email protected]
Changes in v3:
- set errno before logging errors
- respect time budget set by
- respect opts->timeout_ms when polling: only poll for the remaining
time not already consume by connect()
- keep polling if poll returns with EINTR
- reorder early returns and add intermediate variables to clarify code
flow
- Link to v2:
https://patch.msgid.link/[email protected]
Changes in v2:
- drop unneeded initialization
- add back error message for immediate connection failure, and slightly
reword the async connection failure error message
- Link to v1:
https://patch.msgid.link/[email protected]
To: Alexei Starovoitov <[email protected]>
To: Daniel Borkmann <[email protected]>
To: Andrii Nakryiko <[email protected]>
To: Eduard Zingerman <[email protected]>
To: Kumar Kartikeya Dwivedi <[email protected]>
To: Martin KaFai Lau <[email protected]>
To: Song Liu <[email protected]>
To: Yonghong Song <[email protected]>
To: Jiri Olsa <[email protected]>
To: Emil Tsalapatis <[email protected]>
To: Ihor Solodrai <[email protected]>
To: Shuah Khan <[email protected]>
Cc: [email protected]
Cc: Bastien Curutchet <[email protected]>
Cc: Thomas Petazzoni <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
tools/testing/selftests/bpf/network_helpers.c | 34 ++++++++++++++++++++++++---
1 file changed, 31 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/network_helpers.c
b/tools/testing/selftests/bpf/network_helpers.c
index db935a9d9fc1..89336201572f 100644
--- a/tools/testing/selftests/bpf/network_helpers.c
+++ b/tools/testing/selftests/bpf/network_helpers.c
@@ -49,6 +49,8 @@
errno = __save; \
})
+#define CONNECT_MIN_TIMEOUT_MS 5000
+
struct ipv4_packet pkt_v4 = {
.eth.h_proto = __bpf_constant_htons(ETH_P_IP),
.iph.ihl = 5,
@@ -291,6 +293,12 @@ int client_socket(int family, int type,
return -1;
}
+static int connect_timeout_ms(const struct network_helper_opts *opts)
+{
+ /* Enforce a minimum connect timeout value */
+ return MAX(opts->timeout_ms, CONNECT_MIN_TIMEOUT_MS);
+}
+
int connect_to_addr(int type, const struct sockaddr_storage *addr, socklen_t
addrlen,
const struct network_helper_opts *opts)
{
@@ -305,13 +313,33 @@ int connect_to_addr(int type, const struct
sockaddr_storage *addr, socklen_t add
return -1;
}
+ /* Override timeout configuration with a larger value for the
+ * connection
+ */
+ if (settimeo(fd, connect_timeout_ms(opts))) {
+ log_err("Failed to set connect timeout");
+ goto close;
+ }
+
if (connect(fd, (const struct sockaddr *)addr, addrlen)) {
- log_err("Failed to connect to server");
- save_errno_close(fd);
- return -1;
+ log_err("Failed to connect");
+ goto close;
+ }
+
+ /* If the timeout configured by the test is different from the
+ * connect timeout, restore it
+ */
+ if (opts->timeout_ms != CONNECT_MIN_TIMEOUT_MS &&
+ settimeo(fd, opts->timeout_ms)) {
+ log_err("Failed to set timeout for connected socket");
+ goto close;
}
return fd;
+
+close:
+ save_errno_close(fd);
+ return -1;
}
int connect_to_addr_str(int family, int type, const char *addr_str, __u16 port,
---
base-commit: 673b1f0272d7dedeec78dee2b47040481e4332ba
change-id: 20260710-tc_tunnel_flaky-27e9a191bd03
Best regards,
--
Alexis Lothoré (eBPF Foundation) <[email protected]>
