CV-Bowen opened a new pull request, #22098:
URL: https://github.com/apache/nuttx/pull/22098
## Summary
Priority handling of the rpmsg rx workers and the tx done interrupt of
rpmsg_virtio:
- `openamp: add the priority field to the rpmsg virtio config`: new
open-amp patch 0025. It puts a `priority` byte into
`struct fw_rsc_config`, taken from the reserved space so the structure
keeps its size, and adds the feature bit `VIRTIO_RPMSG_F_PRIORITY`.
- `take the rx worker priority from the virtio config`:
`rpmsg_init_wqueues()` gets the priority of the device's work queues.
rpmsg_virtio reads it from the resource table when the device offers
`VIRTIO_RPMSG_F_PRIORITY`, else it stays
`CONFIG_RPMSG_WQUEUE_PRIORITY`.
- `enable the tx done interrupt only while waiting`: the tx virtqueue
callback was enabled once at start-up, so the peer interrupted the
sender for every returned tx buffer. Now the first sender that waits
for a buffer in `rpmsg_virtio_notify_wait()` enables it and the last
one disables it again (as Linux virtio_rpmsg_bus does with
`rpmsg_upref_sleepers()`/`rpmsg_downref_sleepers()`), with a recheck
of the virtqueue under the device lock so a buffer returned just
before the callback was enabled is not missed.
- `give each rx worker the priority of the work it runs`: the work
queues were created at `priority + i` and `rpmsg_queue_work()` chose
one by the distance of the endpoint priority from
`RPMSG_PRIO_DEFAULT`. Now all queues start at the device priority and
get a priority on demand with `work_queue_setpriority_wq()`: the
endpoint priority is mapped around the device priority, an existing
queue at that priority is reused, else a free queue is assigned, else
the closest one is used. With `CONFIG_RPMSG_WQUEUE_GLOBAL` the shared
queues are assigned under a spinlock.
- `keep the rx worker priority in range`: the mapped priority was stored
in a `uint8_t` and wrapped. A name service message inherited the
priority of the endpoint it announces, so the announcement of an RT
endpoint (255) mapped to 255 - 127 + 224 = 352 and ran at 96. Name
service messages no longer inherit `RPMSG_PRIO_RT`, and the result is
clamped to the scheduler range.
Depends on apache/nuttx#22058 (its commits are the first 10 of this
branch; review the last 5).
## Impact
- New open-amp patch 0025: `struct fw_rsc_config` gets `priority`, the
former `reserved[6]` becomes `reserved1[3]` and `reserved2[5]` (same
size, no in-tree user of the field). New feature bit
`VIRTIO_RPMSG_F_PRIORITY` (5). No in-tree resource table sets it yet,
so the worker priority stays `CONFIG_RPMSG_WQUEUE_PRIORITY` unless a
platform fills the field.
- With `CONFIG_RPMSG_WQUEUE_NUMBER` > 1 the extra workers no longer run
at fixed `CONFIG_RPMSG_WQUEUE_PRIORITY + i`; their priority follows the
endpoints that use them. With the default of 1 nothing changes.
- The tx done interrupt of rpmsg_virtio is only enabled while a sender
waits for a buffer. With `CONFIG_RPMSG_VIRTIO_PM=y` and
`CONFIG_RPMSG_VIRTIO_PM_AUTORELAX=n` the PM wakelock taken on send is
released from that callback, so it now stays held until the next wait
(`PM_AUTORELAX` defaults to y and releases it from a timer instead; no
in-tree config enables `RPMSG_VIRTIO_PM`).
- `rpmsg_init_wqueues()` takes a priority, `rpmsg_queue_work()` takes
the priority as `int`; both are internal to drivers/rpmsg.
## Testing
Host: Ubuntu 22.04.5 LTS x86_64, QEMU version 10.2.1, sim: cc (Ubuntu
13.1.0-8ubuntu1~22.04) 13.1.0, qemu-armv8a: aarch64-none-elf-gcc (Arm GNU
Toolchain 12.2.Rel1 (Build arm-12.24)) 12.2.1 20221205
Every commit builds sim:rpserver, sim:rpproxy, qemu-armv8a:rpserver and
qemu-armv8a:rpproxy with 0 warnings (open-amp re-extracted for each
commit, so patch 0025 is applied from scratch). sim:rpserver/rpproxy
also build with `CONFIG_RPMSG_WQUEUE_GLOBAL=y` with 0 warnings.
**Worker priorities**, sim:rpserver + sim:rpproxy, `ps` on both nodes
after boot and after a 200 x 2048 byte ping in each direction (2 runs
each, same result). "before" is the base of this PR (#22058 head).
"config 150" adds a test-only change (not in this PR) to
`arch/sim/src/sim/sim_rptun.c` that sets `VIRTIO_RPMSG_F_PRIORITY` in
the rpmsg vdev `dfeatures` and `rpmsg0_config.priority = 150`.
| `CONFIG_RPMSG_WQUEUE_NUMBER` | before | this PR | this PR, config 150 |
|---|---|---|---|
| 1 | server 224, proxy 224 | (same as before) | server 150, proxy 150 |
| 3 | 224 / 225 / 226 on both | server 224 / 223 / 224, proxy 224 / 224 /
224 | server 150 / 149 / 150, proxy 150 / 150 / 150 |
With 3 queues the server queue at 223 serves the rpmsg-syslog server
endpoint (endpoint priority 126); the others serve endpoints at the
default priority. Without the last commit the same run showed a queue
at 96 (22 with config 150) on both nodes: the name service message of
`rpmsg-ping-rt`.
**Tx buffer exhaustion**: `rpmsg ping all 1000 2048 0 0` (no ack, back
to back, so senders wait in `rpmsg_virtio_notify_wait()`) on both nodes,
then a normal ping, 2 runs per target.
| | before | this PR |
|---|---|---|
| sim, 1000 pings each way | pass, pass | pass, pass |
| qemu-armv8a ivshmem (rpmsg_virtio) channel | pass in every run that
reached it | pass in every run |
| qemu-armv8a rpmsg_port_uart channel | stalls with all 8 tx buffers in use
in 3 of 4 directions | stalls in 1 of 4 |
The rpmsg_port_uart stall happens before and after this PR and that
transport is not touched here.
**Ping latency**, 200 x 2048 byte with ack (test.sh extra cases):
| average | before | this PR |
|---|---|---|
| sim server / proxy | 9.998 / 9.998 ms | 9.998 / 9.998 ms |
| qemu-armv8a ivshmem, server / proxy | 202 / 222 us | 203 / 203 us |
Not tested: the number of tx done interrupts (no interrupt counter in
these configs), a platform that fills `fw_rsc_config.priority` in its
resource table, `CONFIG_RPMSG_VIRTIO_PM`, and endpoints with priorities
other than the defaults present in these configs.
No regression in rpmsg: sim:rpserver/rpproxy and
qemu-armv8a:rpserver/rpproxy boot and pass the fixed rpmsg cases (dump, ping,
rpmsgfs; 22/22). qemu-armv8a was built with LIBC_ENVPATH and BINFMT_CHECKEXEC
disabled, which current master needs for nxinit to start nsh there; unrelated
to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]