casaroli opened a new pull request, #22129:
URL: https://github.com/apache/nuttx/pull/22129

   ## Summary
   
   `iob_allocwait()` counts a waiter before it waits: `g_throttle_wait++` for a 
throttled allocation, `g_iob_count--` for the others. When the wait ends 
without an IOB (the timeout of `iob_timedalloc()`, for example a TCP `send()` 
with `SO_SNDTIMEO` that waits for a write buffer), the count stays. 
`iob_free()` then puts the next freed IOB on the committed list for a waiter 
that is gone and posts the semaphore. That IOB is no longer in the free count. 
Only a later allocation that has to wait can take it.
   
   Each such timeout keeps one IOB. With a few senders whose peers do not read, 
the stale counts grow fast and the committed list takes almost the whole pool. 
Then the free count stays below `CONFIG_IOB_THROTTLE`, and `tcp_datahandler()` 
drops all received TCP data that no `recv()` waits for, on every connection of 
the node, after all sockets of the senders are closed.
   
   A test-only line in `/proc/iobinfo` (the counters and the lengths of the 
free and committed lists; not in this PR) shows the pool on master, 
qemu-armv8a:knsh, 96 IOBs, `IOB_THROTTLE` 8 (24 IOBs are in the RX ring of 
virtio-net, so 72 are free at boot):
   
   - 4 TCP pairs over 127.0.0.1 whose readers do not read, senders with 
`SO_SNDTIMEO` 1 s, for 30 s: 154 throttled waiters are counted, 5 of them 
really wait.
   - After all sockets closed and the program ended: 3 IOBs free, 68 IOBs on 
the committed list with 68 posts on the throttle semaphore, 86 waiters still 
counted, no thread waits. 3 + 68 + 25 (virtio-net) = 96.
   - `ifdown eth0` gives back 1 IOB; the 68 stay.
   
   This PR undoes the count of a wait that fails, under `g_iob_lock`, unless 
`iob_free()` already counted the waiter out. In that case the IOB committed to 
this waiter is taken: its post comes at once.
   
   ## Impact
   
   Every IOB user with a timed wait (`iob_timedalloc()`, 
`conn_dev_iobtimedalloc()`: TCP and UDP write buffers, packet sockets, the 
netdev RX/TX buffer). A wait that ends with an IOB does not change. With 
`CONFIG_IOB_THROTTLE` 0 only the `g_iob_count` path applies. No API, ABI or 
Kconfig change.
   
   Build modes and architectures: common code under the existing spinlock; the 
same in flat, protected and kernel builds, with and without SMP.
   
   ## Testing
   
   qemu-armv8a under HVF, 96 IOBs of 196 bytes, `IOB_THROTTLE` 8, 
`CONFIG_NET_TCP_WRITE_BUFFERS`, master debb89e1aa9. The programs run from 
`/system/bin` (kernel build).
   
   - `test_tcp_sndtimeo_iob`, a new case of `cmocka_net_tcp` 
(apache/nuttx-apps#APPS): a loopback pair whose reader does not read; 10 sends 
with `SO_SNDTIMEO` 100 ms time out; after the close, the free count must come 
back.
   - `iobwait N`: the same with N timeouts of 300 ms, in a separate program.
   - `tcpstuck`: the scenario above (4 pairs for 30 s, an echo server on eth0 
that a host program uses at the same time), then `/proc/iobinfo` after the 
program ended, and after `ifdown eth0`.
   
   | Test | master | this PR |
   |---|---|---|
   | `test_tcp_sndtimeo_iob`, knsh, 1 CPU | FAIL: 61 free, 72 before | PASS |
   | `iobwait 10`, then `iobwait 10`, `iobwait 40` | 72 -> 61 -> 61 -> 61 | 72 
-> 72 -> 72 -> 72 |
   | `tcpstuck`, free IOBs after the program ended | 3 | 72 |
   | the same after `ifdown eth0` | 4 | 72 |
   | `iobwait 10`, knsh, 4 CPUs | 72 -> 61 | 72 -> 72 |
   | `test_tcp_sndtimeo_iob`, knsh, 4 CPUs, after the `iobwait` above | PASS 
(see below) | PASS |
   | `test_tcp_sndtimeo_iob`, flat qemu-armv8a:nsh (96 free at boot) | FAIL: 85 
free, 96 before | PASS |
   | `ostest` (each boot) | status 0 | status 0 |
   
   `test_tcp_connect_ipv4` of `cmocka_net_tcp` fails in this configuration on 
master and with this PR (`ECONNREFUSED` from the group server). On master the 
new case passes when an earlier timeout already left IOBs on the committed 
list: their posts serve the next waits, so the loss does not grow.
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and 
this PR: all OK, 0 warnings. Text size (bytes):
   
   | Configuration | Build mode | `IOB_THROTTLE` | master | this PR |
   |---|---|---|---|---|
   | qemu-armv8a:netnsh | flat | 8 | 554184 | 554272 (+88) |
   | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS` | flat | 8 | 557160 | 557248 
(+88) |
   | the same, CMake | flat | 8 | 557944 | 558032 (+88) |
   | the same + `CONFIG_IOB_THROTTLE=0` | flat | 0 | 556908 | 556960 (+52) |
   | qemu-armv8a:netnsh_smp + write buffers | flat, SMP | 8 | 532876 | 532976 
(+100) |
   | rv-virt:pnsh64 + TCP, write buffers | protected | 8 | 184672 | 184928 
(+256) |
   | rv-virt:knetnsh64 | kernel | 2 | 229097 | 229175 (+78) |
   | rv-virt:knetnsh64 + write buffers, dynamic pools | kernel | 2 | 231477 | 
231555 (+78) |
   | lc823450-xgevk:krndis | protected | 0 | 193476 | 193524 (+48) |
   | qemu-intel64:nsh_pci + TCP, write buffers | flat | 8 | 8760473 | 8760562 
(+89) |
   | sim:tcpblaster | sim | 16 | 997922 | 998082 (+160) |
   | lm3s6432-s2e:nsh (only `iob_tryalloc()` is linked) | flat | 0 | 63496 | 
63496 (+0) |
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: 
`Documentation/os/memory/iob.rst` does not describe the waiter counts; no 
change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] mm/iob: Undo the waiter ... casaroli (via GitHub) via commits

Reply via email to