casaroli opened a new pull request, #22131: URL: https://github.com/apache/nuttx/pull/22131
## Summary With `CONFIG_NET_SEND_BUFSIZE > 0`, a TCP `send()` that finds the send buffer full waits on `conn->snd_sem`. It compares the queued bytes with the limit while it holds the device and connection locks; then `conn_dev_sem_timedwait()` releases the locks and waits. `tcp_sendbuffer_notify()` posted the semaphore only when a thread already waited on it (count < 0). When the ACKs that empty the write queue are handled between the release of the locks and the wait, the post is lost. This happens when the work queue of a network driver waits for the device lock while the sender holds it (1 CPU), or when it runs on another CPU (SMP). If no other ACK comes, the sender waits until its `SO_SNDTIMEO` time-out, or for ever. A test-only log (not in this PR) in `tcp_sendbuffer_notify()` shows the lost post: with the sender between the release of the locks and the wait, the count is 0 and the notify does nothing. At the end of the 30 s wait of the sender, the write queue and the unacked queue are empty, nothing is unacked, the peer window is open, and the count is 0. This PR posts also when the count is 0. A count that no sender waited for only makes a sender check the limit once more. UDP (`udp_sendbuffer_notify()`) has the same code and the same window (the sender holds the connection lock only), and gets the same change in a second commit. ## Impact TCP and UDP with `CONFIG_NET_SEND_BUFSIZE > 0` (or a send buffer size from `SO_SNDBUF`): a send that waits on the send limit is woken when the buffer drains. With `CONFIG_NET_SEND_BUFSIZE` 0 (the default) this code is not compiled. No API, ABI or Kconfig change. Build modes and architectures: common network code, the same in flat, protected and kernel builds, with and without SMP. Not changed: the packet socket (`pkt_sendbuffer_notify()`) and CAN (`can_sendbuffer_notify()`) have the same `val < 0` check. ## Testing qemu-armv8a:knsh under HVF (Apple M-series), 96 IOBs of 196 bytes, `IOB_THROTTLE` 8, `CONFIG_NET_TCP_WRITE_BUFFERS`, `CONFIG_NET_SEND_BUFSIZE=2156` and `CONFIG_NET_RECV_BUFSIZE=2156`, master 5e4ecd1d603. The programs run from `/system/bin`. - `tcpsndwake`: rounds of "send 16 KiB (or 64 KiB, 4 KiB) over 127.0.0.1, wait for a 1-byte reply from a thread that reads it" for 30 s, 5 s time-outs; FAIL when a round waits. - `tcpthru send`: 4 MiB to a host program over the QEMU user network, 8 times in one boot; `SO_SNDTIMEO` 30 s. - `test_tcp_send_limit_wakeup`, a new case of `cmocka_net_tcp` (apache/nuttx-apps#APPS): the same rounds as `tcpsndwake` for 10 s, 2 s limit per round. The `cmocka_net_tcp` runs have #22025 on both sides: without it, the connections in TIME_WAIT of the earlier cases can use all devif callbacks (see below). | Test | master | this PR | |---|---|---| | `tcpsndwake`, 4 CPUs, 4 runs (16, 64, 4, 16 KiB) | FAIL in 3 of 4: a round waits 5 s, 10 s, 15 s | PASS 4 of 4 (2.6 million rounds) | | `tcpsndwake`, 1 CPU, 3 runs | PASS | | | `tcpthru send` to the host, 1 CPU, 8 uploads | 2 of 8 take 30 s (the time-out) | 0 of 8 | | `test_tcp_send_limit_wakeup`, 4 CPUs, 5 boots | FAIL 5 of 5: a round waits 10 s or 15 s | PASS 5 of 5 | | `test_tcp_send_limit_wakeup`, 1 CPU | | PASS | | `test_tcp_send_limit_wakeup`, flat qemu-armv8a:nsh, 4 CPUs | PASS (1 boot) | PASS | | `test_tcp_send_limit_wakeup`, 4 CPUs, `CONFIG_NET_SEND_BUFSIZE` 0 | | PASS | | `ostest`, 1 and 4 CPUs, flat nsh | | status 0 | With the test-only log, the 1-CPU uploads on master waited 30 s in 1 and in 4 of 8 runs (two boots; the log makes the window longer). The `ENOMEM` of `send()` that we saw after several transfers with `CONFIG_NET_SEND_BUFSIZE=8192` is not in this code: connections in TIME_WAIT keep three devif callbacks each for two minutes (the close callback, the disabled send callback and the monitor), and with the default pool of 16 a few closed connections use all of them. #22025 fixes that (its commit "Free the send callback and the monitor of a closed connection"); with #22025 the same sequence passes (8 transfers in one boot, `CONFIG_NET_SEND_BUFSIZE=8192`: master `ENOMEM` after the fifth, master with #22025 all pass). Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and this PR: all OK, 0 warnings. Text size (bytes): | Configuration | master | this PR | |---|---|---| | qemu-armv8a:netnsh (no TCP write buffers, no send limit) | 554188 | 554172 (-16) | | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS` (no send limit) | 557164 | 557148 (-16) | | the same + `NET_UDP_WRITE_BUFFERS` + `NET_SEND_BUFSIZE=8192` (TCP and UDP code built) | 559432 | 559424 (-8) | | qemu-armv8a:netnsh_smp + write buffers | 532864 | 532864 (+0) | | sim:tcpblaster | 997922 | 997922 (+0) | The first two rows do not build the changed code: all symbols have the same sizes, the 16 bytes are in unnamed read-only data. `tools/checkpatch.sh -c -u -m -g` passes. Documentation: no page describes this wait; no change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
