On Thu Apr 2, 2026 at 12:41 PM CEST, Kevin Traynor via dev wrote:
On 4/1/26 1:03 PM, Eelco Chaudron via dev wrote:
On 1 Apr 2026, at 13:57, Eelco Chaudron via dev wrote:
This patch adds support for specific PMD thread initialization,
deinitialization, and a callback execution to perform work as
part of the PMD thread loop. This allows hardware offload
providers to handle any specific asynchronous or batching work.
This patch also adds cycle statistics for the provider-specific
callbacks to the 'ovs-appctl dpif-netdev/pmd-perf-show' command.
Bringing back the discussion on the earlier patch between Ilya
and Gaetan to this revision :)
Ilya:
Hi, Eelco. As we talked before, this infrastructure
resembles the async
work infra that was proposed in the past for the use case of
async vhost
processing. And I don't see any real use case proposed for it here nor
in the RFC, where the question was asked, but not replied.
Gaetan:
Hi Gaetan,
A few questions below. I'm not so clear on the DOCA threading
requirements, so questions may be broad.
Hi Ilya, Eelco,
Thanks for the patch and for the review.
The use-case on our side is distributed data-structures in DOCA that
requires each participating threads to do maintenance work
periodically.
Specifically, offload threads will insert offload objects.
Those will reserve entries in a map that can be resized. The DOCA
implementation requires any thread that owns an entry to perform the
work of moving it to the new bucket / space after resize is initiated.
This is a pervasive design choice in DOCA, they write most of
their APIs
assuming participating threads are periodically calling into these
maintenance functions.
What is a "particpating thread" ? IIUC, the pmd thread passes down the
flow pattern/action and the offload thread inserts the offload
into the NIC.
In that case, is it the offload thread that owns the entry ?
Participating threads are any threads that registered to DOCA-flow as
offloading threads. In our case, it means:
* The main thread
--> When probing a port, starting it requires installing
DOCA offloads to execute RSS in particular, and a few other
'admin' offloads (optional rate-limiting on VF to avoid
noisy-neighbors, etc).
* The offload thread(s) (in the OVS sense)
A thread in OVS managing dp-flow offloads asynchronously.
* The polling thread(s)
CT-offload is much simpler and faster than dp-flow offload.
Executing offload insertion synchronously from the fastpath
is beneficial.
In our case, 'participating threads' are any thread owning an offload
queue in DOCA-flow.
We have a few exceptions for the main thread, mainly that we force all
offload operations to be fully synchronous there: we do not want to
publish a new netdev if its 'admin' offloads have not yet been received
and successfully acknowledged by the hardware, so we force waiting
operations for it: it does not need to do regular upkeep etc.
Some of such work is also time-sensitive, for example the current
implementation requires a CT offload thread to receive
completions after
some hardware initialization. Until this completion is done, the CT
offload entry is not fully usable (cannot be queried for activity /
counters). We cannot leave batches of CT offload entry waiting for
completion, assuming that at some later point, we will eventually
re-execute something in our offload provider: it leaves a few stranded
connection objects incomplete.
This has the result of having hardware execution of a flow with CT
actions, but no activity counters: the software datapath then deletes
the connection and/or flow due to inactivity.
Can this periodic work be done by the offload thread ? If it is fast
enough for inserting the offload, then maybe it is fast enough for this.
The PMD thread owns the offload queue. If another thread has to execute
its upkeep work, it means sharing the queue between threads.
Some DPDK PMDs use alarms for periodic maintenance work, could they be
used inside DOCA for this?
Those upkeep functions are exposed by DOCA and part of the DOCA-flow
API. DOCA does not expose an event framework to schedule this kind of
work, it requires DOCA applications to explicitly call those functions.
If it needs to be on the PMD thread, is the work significant (i.e. more
than a few % cpu) and how variable is it ? Could it be added inside the
call to rte_eth_rx_burst polling ?
It can be significant.
The work is anything requiring the use of the offload queue owned by
this thread. The principle is that the owning thread must execute it.
Currently, with CT offloads we have:
* offload queue polling for HW completion (requests have been
executed: add / mod / del were executed)
* CT-del: A conn was offloaded by PMD 1. The connection either expired
or another PMD 2 closed it: ct-clean or PMD-2 send a CT-del
request to PMD-1: PMD-1 must poll for CT-del requests and
execute them locally.
* Offload flush: when a port is deleted, all owning threads must
process a blocking flush request from the main thread. The main
thread only proceeds once all participating threads have completed
their flush.
Completion is a very lightweight work, but we must execute it.
Generally we do only completion polling as needed: we only clear enough
room in the offload queue for the current batch of requests we want to
enqueue, but we have an issue on idle: some stray completion can
be left in the queue and won't be processed if we rely only on activity.
Currently DOCA-flow does not support leaving the completions until the
port is deleted: they need to be processed.
CT-del can be significant in some cases. We have a 'rolling-window' case
of constant open + close of short connections, and in this worst case,
CT-del takes ~30% (both local and distant). Some portion of it comes from
CT-del messages, in particular in case of multiple PMDs.
Offload flush is generally quick, but we must answer the flush message
quickly to block the main thread as little as possible.
Some of the messages must be handled even if there is no RX-burst: a PMD
that is waiting for reload will need to execute a flush message that it
has received.
