yangsong8-a1 opened a new pull request, #22155:
URL: https://github.com/apache/nuttx/pull/22155
## Summary
Fix the `CONFIG_CONSOLE_NETDEV` hang on SMP reported in #21746, and the
other problems the stress test found on the same path.
#21746 replaced `enter_critical_section()` in the serial upper half with
`uart_spinlock()`. That lock is owned by a CPU, not by a task, so a task that
blocks or is switched out while holding it can resume on another CPU and spin
on the lock it holds itself. The netdev console blocked inside the locked
callbacks (`psock_send()`, `psock_recv()`, `psock_socket()`, ...), and the
upper half could switch a lock holder out from inside `uart_open()`,
`uart_close()` and `uart_readv()`.
- `drivers/serial: read the ring index once in the DMA helpers`: same race
as a6393fa3201 / a3d4802ff85, in `uart_xmitchars_dma()` and
`uart_recvchars_dma()`. Seen as `psock_send()` with a length of 65408.
- `drivers/serial: lock the scheduler while uart_open/close/readv hold the
lock`: these used `uart_spinlock(dev, false)` and call code that ends in
`sched_unlock()` (`uart_poll_notify()`, `work_queue()`), which can switch the
holder out.
- `serial/netdev_console: do not block under the uart spinlock`: send and
receive run on LPWORK; `send()` only queues into the xmit ring.
- `serial/netdev_console: open and close the socket from a work`:
`attach()`/`detach()` only queue a work that opens or closes the socket; this
also fixes `detach()` closing the socket under a running work.
- `serial/netdev_console: add an option for a dedicated work queue`:
`CONFIG_CONSOLE_NETDEV_WORKQUEUE`, default n.
## Impact
- All serial drivers: `uart_open()`, `uart_close()`, `uart_readv()` and the
interrupt/idle write path now also lock the scheduler while they hold the uart
lock (no effect in interrupt context). The DMA helpers read the ring index
once.
- Netdev console: `open()` no longer fails when the socket cannot be
created. The driver keeps the error in `priv->sockerr` and retries every
second; until the socket opens, output stays in the xmit buffer as when the
network is down (a blocking writer waits once it is full). The driver no
longer logs socket errors, because the log may go to this console and the work
thread would wait on itself. In a panic, output written through `send()` is
dropped.
- New Kconfig `CONFIG_CONSOLE_NETDEV_WORKQUEUE` (default n), with
`CONFIG_CONSOLE_NETDEV_WORKPRIORITY` and `CONFIG_CONSOLE_NETDEV_WORKSTACKSIZE`.
Existing defconfigs are unchanged.
## Testing
Ubuntu 22.04, gcc (Ubuntu 13.1.0-8ubuntu1~22.04) 13.1.0,
aarch64-none-elf-gcc (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 13.2.1
20231009, QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.31) (KVM).
Builds, each commit on its own: `qemu-intel64:eth_console` with
`CONFIG_SMP=y`, no new warnings. Final branch: `qemu-intel64:eth_console` (SMP
and non-SMP, with and without `CONFIG_CONSOLE_NETDEV_WORKQUEUE`),
`qemu-intel64:can_console` (SMP and non-SMP, with and without the option),
`qemu-intel64:nsh` (SMP and non-SMP), `qemu-armv8a:nsh_smp`, `sim:nsh`.
`checkpatch.sh -c -u -m -g` clean.
Runtime, qemu-intel64 (KVM, e1000, slirp; console target 10.0.2.2:55000):
| Test | master | this PR |
|-|-|-|
| Echo: type `help\n` 30 times, one byte per datagram (SMP=2) | 3/3 hang
(162, 162, 894 bytes, then nothing) | 3/3 pass, 29354 bytes |
| RX flood: one datagram/ms for 15 s, then a marker command (SMP=2) | 3/3
hang, marker missing | 3/3 pass |
| Open/close: NSH over telnet, two scripts loop `echo ... > /dev/console`,
host floods the console port, 30 s (SMP=2) | 3/3 hang | 15/15 pass |
| Same, `CONFIG_CONSOLE_NETDEV_WORKQUEUE=y` | — | 5/5 pass |
| Echo / RX flood / open-close, SMP=n | — | 3/3, 3/3, 3/3 pass |
| `ostest`, SMP=2 and SMP=n | — | `Exiting with status 0` |
| No UDP connections (`NET_UDP_PREALLOC_CONNS=0`), `dd` 51200 bytes to
/dev/console in the background | — | `dd` waits on `xmitsem`, NSH over telnet
keeps working, `sockerr = -12`, ctrlwork requeued 1 s later |
master, open/close test, gdb (`lock` is `dev->lock`: owner CPU0, which is
idle):
```
Thread 2 (Thread 1.2 (CPU#1 [running])):
#0 rspin_lock (lock=0x100a0c180 <g_netdev_console+128>) at
/home/yangsong8/project/nuttx_github/nuttx/.worktrees/netdev-console-smp/include/nuttx/spinlock.h:729
#1 0x000000010092965d in rspin_lock_irqsave (lock=0x100a0c180
<g_netdev_console+128>) at
/home/yangsong8/project/nuttx_github/nuttx/.worktrees/netdev-console-smp/include/nuttx/spinlock.h:759
#2 0x0000000100929686 in rspin_lock_irqsave_nopreempt (lock=0x100a0c180
<g_netdev_console+128>) at
/home/yangsong8/project/nuttx_github/nuttx/.worktrees/netdev-console-smp/include/nuttx/spinlock.h:769
#3 0x000000010092c3a3 in uart_spinlock (dev=0x100a0c100 <g_netdev_console>,
nopreempt=true) at serial/serial.c:2530
#4 0x0000000100929a5a in uart_poll_notify (dev=0x100a0c100
<g_netdev_console>, min=0, max=4, eventset=4) at serial/serial.c:245
#5 0x000000010092c5f9 in uart_datasent (dev=0x100a0c100 <g_netdev_console>)
at serial/serial.c:2677
#6 0x000000010092cb7e in uart_xmitchars_done (dev=0x100a0c100
<g_netdev_console>) at serial/serial_dma.c:199
#7 0x0000000100928280 in netdev_console_dmasend (dev=0x100a0c100
<g_netdev_console>) at serial/netdev_console.c:378
#8 0x000000010092caa1 in uart_xmitchars_dma (dev=0x100a0c100
<g_netdev_console>) at serial/serial_dma.c:155
#9 0x0000000100928563 in netdev_console_dmatxavail (dev=0x100a0c100
<g_netdev_console>) at serial/netdev_console.c:468
#10 0x000000010092b91e in uart_writev (filep=0x101c3a0b0, uio=0x102c3a250)
at serial/serial.c:1916
lock {{val = 65537, local_val = 65537}}
```
Other drivers after the serial.c / serial_dma.c changes: `qemu-intel64:nsh`
(16550, SMP=2 and SMP=n) typed 20 lines, all echoed; `qemu-armv8a:nsh_smp`
(pl011) 20 lines echoed and `ostest` status 0, same as master; `sim:nsh` (TX/RX
DMA) 20 lines echoed and `ostest` status 0.
Not tested: CAN (ISO-TP) console at runtime (build only); a socket that
fails to open and then recovers; `O_NONBLOCK` writes while the socket is not
open; protected and kernel builds; real hardware; hardware DMA UART drivers on
SMP. A regression test in nuttx-apps is not possible here because the test
needs a host peer on the network console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]