On Tue Sep 1, 2026 at 1:28 PM CEST, Eelco Chaudron wrote: > > > On 21 Aug 2026, at 17:17, Kevin Traynor wrote: > > > On 7/7/26 20:08, Gaetan Rivet wrote: > >> On Tue Jun 23, 2026 at 7:04 PM CEST, Aaron Conole wrote: > >>> Kevin Traynor via dev <[email protected]> writes: > >>> > >>>> On 6/2/26 5:24 PM, Gaetan Rivet wrote: > >>>>> On Fri May 29, 2026 at 6:26 PM CEST, Kevin Traynor wrote: > >>>>>> On 5/28/26 10:29 AM, Eelco Chaudron wrote: > >>>>>>> > >>>>>>> > >>>>>>> On 27 May 2026, at 16:37, Gaetan Rivet wrote: > >>>>>>> > >>>>>>>> On Thu Apr 2, 2026 at 12:41 PM CEST, Kevin Traynor via dev wrote: > >>>>>>>>> On 4/1/26 1:03 PM, Eelco Chaudron via dev wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 1 Apr 2026, at 13:57, Eelco Chaudron via dev wrote: > >>>>>>>>>> > >>>>>>>>>>> This patch adds support for specific PMD thread initialization, > >>>>>>>>>>> deinitialization, and a callback execution to perform work as > >>>>>>>>>>> part of the PMD thread loop. This allows hardware offload > >>>>>>>>>>> providers to handle any specific asynchronous or batching work. > >>>>>>>>>>> > >>>>>>>>>>> This patch also adds cycle statistics for the provider-specific > >>>>>>>>>>> callbacks to the 'ovs-appctl dpif-netdev/pmd-perf-show' command. > >>>>>>>>>> > >>>>>>>>>> Bringing back the discussion on the earlier patch between Ilya > >>>>>>>>>> and Gaetan to this revision :) > >>>>>>>>>> > >>>>>>>>>> Ilya: > >>>>>>>>>> Hi, Eelco. As we talked before, this infrastructure > >>>>>>>>>> resembles the async > >>>>>>>>>> work infra that was proposed in the past for the use case of > >>>>>>>>>> async vhost > >>>>>>>>>> processing. And I don't see any real use case proposed for it > >>>>>>>>>> here nor > >>>>>>>>>> in the RFC, where the question was asked, but not replied. > >>>>>>>>>> > >>>>>>>>>> Gaetan: > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Hi Gaetan, > >>>>>>>>> > >>>>>>>>> A few questions below. I'm not so clear on the DOCA threading > >>>>>>>>> requirements, so questions may be broad. > >>>>>>>>> > >>>>>>>>>> Hi Ilya, Eelco, > >>>>>>>>>> > >>>>>>>>>> Thanks for the patch and for the review. > >>>>>>>>>> > >>>>>>>>>> The use-case on our side is distributed data-structures in DOCA > >>>>>>>>>> that > >>>>>>>>>> requires each participating threads to do maintenance work > >>>>>>>>>> periodically. > >>>>>>>>>> > >>>>>>>>>> Specifically, offload threads will insert offload objects. > >>>>>>>>>> Those will reserve entries in a map that can be resized. The > >>>>>>>>>> DOCA > >>>>>>>>>> implementation requires any thread that owns an entry to > >>>>>>>>>> perform the > >>>>>>>>>> work of moving it to the new bucket / space after resize is > >>>>>>>>>> initiated. > >>>>>>>>>> > >>>>>>>>>> This is a pervasive design choice in DOCA, they write most of > >>>>>>>>>> their APIs > >>>>>>>>>> assuming participating threads are periodically calling into > >>>>>>>>>> these > >>>>>>>>>> maintenance functions. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> What is a "particpating thread" ? IIUC, the pmd thread passes down > >>>>>>>>> the > >>>>>>>>> flow pattern/action and the offload thread inserts the offload > >>>>>>>>> into the NIC. > >>>>>>>>> > >>>>>>>>> In that case, is it the offload thread that owns the entry ? > >>>>>>>>> > >>>>>>>> > >>>>>>>> Participating threads are any threads that registered to DOCA-flow as > >>>>>>>> offloading threads. In our case, it means: > >>>>>>>> > >>>>>>>> * The main thread > >>>>>>>> --> When probing a port, starting it requires installing > >>>>>>>> DOCA offloads to execute RSS in particular, and a few > >>>>>>>> other > >>>>>>>> 'admin' offloads (optional rate-limiting on VF to avoid > >>>>>>>> noisy-neighbors, etc). > >>>>>>>> > >>>>>>>> * The offload thread(s) (in the OVS sense) > >>>>>>>> A thread in OVS managing dp-flow offloads asynchronously. > >>>>>>>> > >>>>>>>> * The polling thread(s) > >>>>>>>> CT-offload is much simpler and faster than dp-flow offload. > >>>>>>>> Executing offload insertion synchronously from the fastpath > >>>>>>>> is beneficial. > >>>>>>>> > >>>>>>>> In our case, 'participating threads' are any thread owning an offload > >>>>>>>> queue in DOCA-flow. > >>>>>>>> > >>>>>>>> We have a few exceptions for the main thread, mainly that we force > >>>>>>>> all > >>>>>>>> offload operations to be fully synchronous there: we do not want to > >>>>>>>> publish a new netdev if its 'admin' offloads have not yet been > >>>>>>>> received > >>>>>>>> and successfully acknowledged by the hardware, so we force waiting > >>>>>>>> operations for it: it does not need to do regular upkeep etc. > >>>>>>>> > >>>>>>>>>> Some of such work is also time-sensitive, for example the > >>>>>>>>>> current > >>>>>>>>>> implementation requires a CT offload thread to receive > >>>>>>>>>> completions after > >>>>>>>>>> some hardware initialization. Until this completion is done, > >>>>>>>>>> the CT > >>>>>>>>>> offload entry is not fully usable (cannot be queried for > >>>>>>>>>> activity / > >>>>>>>>>> counters). We cannot leave batches of CT offload entry waiting > >>>>>>>>>> for > >>>>>>>>>> completion, assuming that at some later point, we will > >>>>>>>>>> eventually > >>>>>>>>>> re-execute something in our offload provider: it leaves a few > >>>>>>>>>> stranded > >>>>>>>>>> connection objects incomplete. > >>>>>>>>>> > >>>>>>>>>> This has the result of having hardware execution of a flow with > >>>>>>>>>> CT > >>>>>>>>>> actions, but no activity counters: the software datapath then > >>>>>>>>>> deletes > >>>>>>>>>> the connection and/or flow due to inactivity. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Can this periodic work be done by the offload thread ? If it is fast > >>>>>>>>> enough for inserting the offload, then maybe it is fast enough for > >>>>>>>>> this. > >>>>>>>>> > >>>>>>>> > >>>>>>>> The PMD thread owns the offload queue. If another thread has to > >>>>>>>> execute > >>>>>>>> its upkeep work, it means sharing the queue between threads. > >>>>>>>> > >>>>>>>>> Some DPDK PMDs use alarms for periodic maintenance work, could they > >>>>>>>>> be > >>>>>>>>> used inside DOCA for this? > >>>>>>>>> > >>>>>>>> > >>>>>>>> Those upkeep functions are exposed by DOCA and part of the DOCA-flow > >>>>>>>> API. DOCA does not expose an event framework to schedule this kind of > >>>>>>>> work, it requires DOCA applications to explicitly call those > >>>>>>>> functions. > >>>>>>>> > >>>>>>>>> If it needs to be on the PMD thread, is the work significant (i.e. > >>>>>>>>> more > >>>>>>>>> than a few % cpu) and how variable is it ? Could it be added inside > >>>>>>>>> the > >>>>>>>>> call to rte_eth_rx_burst polling ? > >>>>>>>>> > >>>>>>>> > >>>>>>>> It can be significant. > >>>>>>>> The work is anything requiring the use of the offload queue owned by > >>>>>>>> this thread. The principle is that the owning thread must execute it. > >>>>>>>> > >>>>>>>> Currently, with CT offloads we have: > >>>>>>>> > >>>>>>>> * offload queue polling for HW completion (requests have been > >>>>>>>> executed: add / mod / del were executed) > >>>>>>>> > >>>>>>>> * CT-del: A conn was offloaded by PMD 1. The connection either > >>>>>>>> expired > >>>>>>>> or another PMD 2 closed it: ct-clean or PMD-2 send a CT-del > >>>>>>>> request to PMD-1: PMD-1 must poll for CT-del requests and > >>>>>>>> execute them locally. > >>>>>>>> > >>>>>>>> * Offload flush: when a port is deleted, all owning threads must > >>>>>>>> process a blocking flush request from the main thread. The main > >>>>>>>> thread only proceeds once all participating threads have > >>>>>>>> completed > >>>>>>>> their flush. > >>>>>>>> > >>>>>>>> Completion is a very lightweight work, but we must execute it. > >>>>>>>> Generally we do only completion polling as needed: we only clear > >>>>>>>> enough > >>>>>>>> room in the offload queue for the current batch of requests we want > >>>>>>>> to > >>>>>>>> enqueue, but we have an issue on idle: some stray completion can > >>>>>>>> be left in the queue and won't be processed if we rely only on > >>>>>>>> activity. > >>>>>>>> Currently DOCA-flow does not support leaving the completions until > >>>>>>>> the > >>>>>>>> port is deleted: they need to be processed. > >>>>>>>> > >>>>>>>> CT-del can be significant in some cases. We have a 'rolling-window' > >>>>>>>> case > >>>>>>>> of constant open + close of short connections, and in this worst > >>>>>>>> case, > >>>>>>>> CT-del takes ~30% (both local and distant). Some portion of it comes > >>>>>>>> from > >>>>>>>> CT-del messages, in particular in case of multiple PMDs. > >>>>>>>> > >>>>>>>> Offload flush is generally quick, but we must answer the flush > >>>>>>>> message > >>>>>>>> quickly to block the main thread as little as possible. > >>>>>>>> > >>>>>>>> Some of the messages must be handled even if there is no RX-burst: a > >>>>>>>> PMD > >>>>>>>> that is waiting for reload will need to execute a flush message that > >>>>>>>> it > >>>>>>>> has received. > >>>>>>> > >>>>>>> Hi Gaetan, > >>>>>>> > >>>>>>> I guess Kevin is suggesting to hide this work in > >>>>>>> netdev_doca_rxq_recv(), > >>>>>>> as it will always be called as long as DOCA ports are present on the > >>>>>>> PMD. Or are there cases where this is not the case? > >>>>>>> > >>>>>>> dp_netdev_process_rxq_port() > >>>>>>> netdev_rxq_recv() > >>>>>>> netdev_doca_rxq_recv() > >>>>>>> > >>>>>>> Kevin, please confirm. > >>>>>> > >>>>>> Yes, that's what I was suggesting. The work is rxq specific and we > >>>>>> already have an rxq specific call that is called in a loop so why not > >>>>>> do > >>>>>> it there and include the cycles needed for the maintenance work in the > >>>>>> measured cycles needed for that rxq. > >>>>>> > >>>>>>> > >>>>>>>> I think completions and flushes would be the main issues with the > >>>>>>>> rx-burst approach. > >>>>> > >>>>> Hi, > >>>> > >>>> Hi Gaetan, > >>>> > >>>> Thanks for explaining further. > >>>> > >>>>> > >>>>> We had an issue with this kind of approach with flush commands. > >>>>> A PMD can be registered as a DOCA offload thread, in which case it > >>>>> will receive a blocking flush request on port deletion. > >>>>> This happens even if that port is not scheduled on that PMD. > >>>>> > >>>>> The issue arises when the PMD has no netdev-doca rxq scheduled: it > >>>>> is registered as a DOCA offload thread, but will never process its flush > >>>>> requests. A typical example might be on multi-NUMA, where by default 1 > >>>>> PMD is created per NUMA, and ports are configured with 1 rxq. With a > >>>>> single NIC, its rxq is configured on the closest PMD, leaving the other > >>>>> one idle. The idle PMD is still registered as a DOCA offload thread, as > >>>>> nothing forbids the user from adding a port on its NUMA at a future > >>>>> time. > >>>>> > >>>> > >>>> iiuc, the same issue will be present with the approach in this patch as > >>>> the PMD thread will block if there are no rxqs to poll. > >>>> > >>>> Another issue is that even if there are rxq's being polled with sleep > >>>> settings then there could be a delay in the flush which means blocking. > >>>> > >>>>> In this case, the idle PMD would never enter the right rxq-burst command > >>>>> to process its offload messages. > >>>>> > >>>>> All other cases would seem fine however, I think it almost works. > >>>>> I just don't have a solid approach for this flush issue. > >>>>> > >>>> > >>>> Waiting for PMD threads to flush that aren't doing anything meaningful > >>>> related to the offload or rxqs from the device is not ideal and creates > >>>> a few headaches. > >>>> > >>>> Maybe you could dynamically register/unregister them as needed or find a > >>>> way to not require a flush from ones which aren't actively involved but > >>>> I'm just thinking out loud. > >>> > >>> Ping - it doesn't seem like this discussion is resolved (unless I'm > >>> misreading it). There are some open questions - Kevin, Gaetan, Eelco? > >>> Did I misunderstand it? > >> > >> Indeed, sorry I thought I had answered. > >> > >> I agree that we should not require unrelated PMDs to process flush > >> commands that are irrelevant for them. This can be refined. > >> > >> However I don't see a way around requiring processing offload flush > >> commands from all participating threads, and it does not seem possible > >> currently if offload messages are only received when processing netdev > >> queues. > >> > >> The PMD thread loses that access at the beginning of a 'do_del_port'. > >> At this point, the netdev provider is not yet made aware of the > >> impending deletion, so we cannot prepare a flush message while > >> PMDs still have access to the queues. > >> > >> If we want to have the PMDs only call into queue bursts, then we would > >> need the main thread to signal to the netdev that it will soon be > >> deleted before removing the queues from the PMDs. > >> > > > > After re-reading through the thread and thinking again I can see the > > benefit of an async call for offload work tasks over embedding it in rxq > > calls. > > > > There are still some questions though around usage and coordination between > > threads, these ones come to mind > > > > - Idle PMDs (no rxqs to poll) will be sleeping and need to be woken > > > > - PMDs polling rxqs with little or no traffic may also be sleeping if > > pms-sleep-max is used and there may be a delay in doing async work, > > presumably it will be short enough to not impact > > > > - It's not clear to me how the cycles used for this async work will be > > incorporated into calculations wrt rebalancing rxqs. It is assumed that the > > PMD is reserved and available for rxq (minus a small amount of overhead). > > If async work is taking significant PMD cycles in some cases, then this > > would need to be re-thought out. > > Hi all, > > After going over the discussions, here is my proposal for moving forward. > > The main concerns raised are: > > - Ilya's concern about the work callback API being too generic and potentially > abused for unrelated things outside the offload scope. > > - Kevin's cycle accounting and rebalancing issues introduced by tracking > offload cycles separately. > > - Gaetan's issue with idle PMDs (no rxqs scheduled) not executing the receive > callback, causing a potential flush deadlock on port deletion, etc., etc. > > My proposal is: > > 1. Drop the dpif_offload_pmd_thread_do_work() callback entirely. The async > maintenance work (HW completion polling, CT-del processing, etc.) should > be handled inside the driver's receive callback, i.e., for DOCA this > would be netdev_doca_rxq_recv(). This avoids the cycle accounting > complexity and addresses Ilya's concern, as there is no longer a generic > "do work" hook that could be repurposed for other things. > > 2. Keep dpif_offload_pmd_thread_reload() with its context. This is still > needed to allow the provider to do initialization, cleanup, and flush > work when PMD threads are reconfigured or exit. The lifecycle management > (init/reinit/exit) is scoped purely to resource management, which avoids > API abuse. > > This approach should match all the requirements for DOCA, with minimal > changes to the infrastructure. > > Gaetan, what do you think? Does it match all your requirements? If so I > can send out a v3 with this approach. > > Thanks, > > Eelco
Hello Eelco, I think it works. I am not sure about the amount of context we will need in the reload callback, but I think the general idea is sound. Thanks, _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
