casaroli opened a new pull request, #21793: URL: https://github.com/apache/nuttx/pull/21793
depends-on: apache/nuttx/pull/21778 ## Summary With `CONFIG_LIBC_ATOMIC_IRQ`, the atomic functions of `libs/libc/machine/arch_atomic.c` disable the interrupts with `up_irq_save()`. The 8-byte atomics of `libs/libc/machine/arch_atomic64.c` disable them with `spin_lock_irqsave()` with every atomics option. The compiler calls these functions on the 32-bit targets with no 8-byte atomic instructions (RISC-V, Xtensa, ARM M-profile), for example for `_Atomic long long` or `std::atomic<int64_t>`. The user space libc of a PROTECTED or KERNEL build has these functions too. The semaphore and mutex fast paths, the condition variables, the temporary buffers and the C11 and C++ atomics of the programs use them. User code cannot disable the interrupts: RISC-V traps on `csrrc mstatus`/`sstatus`, Xtensa traps on `rsil`, and ARM M-profile ignores `cpsid` and the `BASEPRI` write in unprivileged mode, so the operation is not atomic. On `esp32c3-legacy-devkit:knsh`, `nxsem_wait()` in `nuttx_user.elf` calls `atomic_compare_exchange_4()`, which runs `csrrc a5,mstatus,a5` in U-mode. On `rv-virt:knsh` (32-bit), a program that adds to a `uint64_t` with `__atomic_fetch_add()` is killed at the first call: `__atomic_fetch_add_8()` runs `csrrc a3,sstatus,a3` in U-mode. - `libs/libc/machine`: in user space, `atomic_lock()` locks the scheduler instead. As with `up_irq_save()`, no other thread runs on this CPU during the operation. The 8-byte atomics keep their spinlock for the other CPUs of an SMP build. They take it with `spin_trylock_notrace()` in a loop, because `spin_lock()` can call kernel functions (`CONFIG_SPINLOCK_INLINE=n`, the spinlock instrumentation). - `libs/libc/semaphore`: in user space with `CONFIG_LIBC_ATOMIC_IRQ`, `nxsem_wait()`, `nxsem_trywait()` and `nxsem_post()` skip the fast path. One fast path try is four system calls now (`sched_lock()` and `sched_unlock()` around the read and the compare and exchange); the slow path does the whole operation in the kernel with one system call. The new macro `NXSEM_FASTPATH` is the initial value of the existing `fastpath` flag. ## Impact User code of PROTECTED and KERNEL builds. The 4-byte atomics: with `CONFIG_LIBC_ATOMIC_IRQ` (in-tree: `esp32c3-legacy-devkit:knsh`). The 8-byte atomics: on every 32-bit target with no 8-byte atomic instructions (in-tree for example `rv-virt:knsh`, `mps2-an500:knsh`, `esp32-devkitc:knsh`, `esp32s3-devkit:knsh`). A user space atomic there is now two system calls; before, it trapped or was not atomic. The build mode test is the one that already selects the kernel-only code in `include/nuttx/semaphore.h` (`NXSEM_SET_UNINTERRUPTIBLE`) and `mm/mm_heap/mm_free.c`. Flat builds and the kernel do not change: the code is the same before and after (see the disassembly table below). In the user space libc of the other atomics options (`LIBC_ATOMIC_TOOLCHAIN`, `LIBC_ATOMIC_ARCH`), only `arch_atomic64.o` changes. ARMv7-A, ARMv7-R and the 64-bit targets have 8-byte atomic instructions: the compiler does not call these functions there. `CONFIG_LIBC_ATOMIC_HWSPINLOCK` does not change: its lock is kernel data, which user code cannot use. No ABI change. No documentation change: `CONFIG_PTHREAD_SPINLOCKS` still needs the toolchain or arch atomics outside a flat build, as `Documentation/standards/posix.rst` says. ## Testing Test program `uatomtest.c`: four `SCHED_RR` threads of the same priority each take a lock N times and increment a plain counter inside it (`sem_wait()`/`sem_post()`, `pthread_mutex_lock()`/`unlock()`), increment an `atomic_t` with a compare and exchange loop, or add `0x100000001` to a `uint64_t` with `__atomic_fetch_add()`. Where the compiler has 8-byte atomic instructions, the 8-byte test calls the libc `__atomic_fetch_add_8()` by its symbol. A lost count, two threads in the lock at once, two different halves of the `uint64_t`, or a semaphore count other than 1 is a FAIL. QEMU, in-tree configurations (`mps2-an500:knsh` with `CONFIG_RR_INTERVAL=10`): | Configuration | master | this PR | |---|---|---| | `rv-virt:knsh` (KERNEL, RV32, U-mode) | uatomtest atomic64: `hello` killed at the first call: Illegal instruction, MTVAL `0x1006b6f3` (`csrrc a3,sstatus,a3`) | uatomtest PASS (sem, mutex, atomic, atomic64); ostest status 0 | | `mps2-an500:knsh` (PROTECTED, ARMv7-M) | uatomtest atomic64 PASS, also with 5000000 adds per thread (5.2 s) | uatomtest PASS (sem, mutex, atomic, atomic64); ostest status 0; 5000000 adds per thread PASS (170.9 s) | | `qemu-armv7a:knsh_smp` (KERNEL, SMP, 4 CPUs) | uatomtest atomic64 PASS | uatomtest PASS (sem, mutex, atomic, atomic64 through the SMP spinlock) | On ARMv7-M, master runs `up_irq_save()` (`msr BASEPRI`) in user mode, which the CPU ignores. The test does not show the race in QEMU: the code between the load and the store has no branch, and QEMU takes interrupts only between translated blocks. On the PR, each 8-byte user atomic is two system calls (33 times slower in this test). ostest on `qemu-armv7a:knsh_smp` stops with a data abort in the signal handler tests on master and on this PR (not related: the compiler inlines the 8-byte atomics on ARMv7-A). `CONFIG_LIBC_ATOMIC_IRQ` forced on in QEMU (no board here has an IRQ atomics chip in a protected build): | Configuration | master | this PR | |---|---|---| | `rv-virt:knsh64` (KERNEL, U-mode) | `/system/bin/init` killed at boot: Illegal instruction, MTVAL `0x1007b7f3` (`csrrc a5,sstatus,a5`) | uatomtest PASS (sem, mutex, atomic, atomic64); ostest status 0 | | `mps2-an500:knsh` (PROTECTED, ARMv7-M) | uatomtest PASS (the ignored `cpsid`/`BASEPRI` write races in a few instructions only); ostest status 0 | uatomtest PASS; ostest status 0 | `-Werror` builds: `rv-virt:knsh`, `rv-virt:knsh64`, `mps2-an500:knsh`, `qemu-armv7a:knsh`, `qemu-armv7a:knsh_smp`, `esp32c3-legacy-devkit:knsh`, `esp32-devkitc:knsh`, `esp32s3-devkit:knsh`; flat: `rv-virt:nsh`, `qemu-armv7a:nsh`, `raspberrypi-pico:nsh`. clang 22: `arch_atomic64.c` with `-Werror` for user space and for the kernel of `mps2-an500:knsh` (thumbv7m), `rv-virt:knsh` (rv32imac) and `qemu-armv7a:knsh_smp` (ARMv7-A, SMP, `CONFIG_SPINLOCK`). Disassembly (`objdump -d`) of master and this PR, built in two trees: | Configuration | `libc.a` | `libkc.a` | `nuttx` | `nuttx_user.elf` | |---|---|---|---|---| | `rv-virt:nsh`, `qemu-armv7a:nsh`, `raspberrypi-pico:nsh` (flat) | same | - | same | - | | `mps2-an500:knsh` | `arch_atomic64.o` changes | same | same | same | | `rv-virt:knsh`, `qemu-armv7a:knsh` | `arch_atomic64.o` changes | same | same instructions; `.rodata` addresses differ (the apps path of the two trees) | - | | `esp32c3-legacy-devkit:knsh` | `arch_atomic.o`, `arch_atomic64.o`, `sem_post.o`, `sem_trywait.o` and `sem_wait.o` change | same | same | changes | On `esp32c3-legacy-devkit:knsh`, `nuttx_user.elf` has no `mstatus` access in the atomics any more (130 CSR instructions before, 2 after, both in `mm_lock_irq()`, which user code does not call). `nxsem_wait()`, `nxsem_trywait()` and `nxsem_post()` jump to the slow path. The user code grows by 1066 bytes (the atomics call `sched_lock()` and `sched_unlock()`); the kernel does not change. `tools/checkpatch.sh -c -u -m -g` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
