I have limited understanding in rcu or workqueues, but my two cents.
On 4/29/26 10:48 PM, Paul E. McKenney wrote:
On Wed, Apr 29, 2026 at 07:08:23PM +0200, Vasily Gorbik wrote:
On Wed, Apr 29, 2026 at 08:30:38PM +0530, Srikar Dronamraju wrote:
* Tejun Heo <[email protected]> [2026-04-10 08:53:30]:
Hello,
Seems that we (mostly Paul) have our own trick to track whether a CPU
has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also
used it in his fix [1]. And I think it won't be that hard to copy it
into workqueue and let queue_work_on() use it so that if the user queues
a work on a never-onlined CPU, it can detect it (with a warning?) and do
something?
The easiest way to do this is just creating the initial workers for all
possible pools. Please see below. However, the downside is that it's going
to create all workers for all possible cpus. This isn't a problem for
anybody else but these IBM mainframes often come up with a lot of possible
but not-yet-or-ever-online CPUs for capacity management, so the cost may not
be negligible on some configurations.
IBM folks, is that okay?
Even on PowerPC LPARS, its not uncommon to have possible cpus != online cpus
at boot. However your approach will work.
And Samir has already tested the same too and reported here
https://lkml.kernel.org/r/[email protected]
From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init
Per-CPU worker pools are initialized for every possible CPU during early boot,
but workqueue_init() only creates initial workers for online CPUs. On systems
where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and
400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED
set but no workers. Any work item queued on such a CPU hangs indefinitely.
This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when
non-preemptible") which made SRCU schedule callbacks on all possible CPUs
during size transitions, triggering workqueue lockup warnings for all
never-onlined CPUs.
Create workers for all possible CPUs during init, not just online ones. For
online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the
worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED
remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can
execute on any CPU. When the CPU later comes online, rebind_workers() handles
the transition to associated operation as usual.
With these patch, if a CPU has been onlined once, it's should be ok to queue
the work on that CPU even if its offline now.
That already seems to hold without this patch, what this patch newly
covers is queueing on CPUs that have never been online.
Do we actually need to create workers for every possible CPU at boot?
On the s390 LPAR in question (76 online / 400 possible) that's a few
hundred extra kthreads kept around for the life of the system.
That's probably the same on PowerPC.
Wouldn't Paul's SRCU-side fix [1] alone be enough here for PowerPC
as well? I retested it on s390 (76/400) and on x86 KVM with
--smp 16,maxcpus=255 and the lockup didn't reproduce in either case.
[1]
https://lore.kernel.org/rcu/ed1fa6cd-7343-4ca3-8b9d-d699ca496f83@paulmck-laptop/
Just to emphasize that SRCU really was buggy before my fix. The
queue_work_on() kernel-doc header clearly states the rules. The bug
is even more embarrassing given just who it was that wrote those two
sentences. ;-)
That mask = ~0 is really looks uncomfortable to me. What does it mean?
It might end up even sending to non possible CPUs without proper checks.
It should use either cpumask_setall? or use cpu_online_mask?
Your current patch rcu_cpu_beenfullyonline indicates that code around
srcu_schedule_cbs_sdp handles hotplug already right?
in that case, just setting mask = cpu_online_mask would work?
Thanx, Paul
/**
* queue_work_on - queue work on specific cpu
* @cpu: CPU number to execute work on
* @wq: workqueue to use
* @work: work to queue
*
* We queue the work to a specific CPU, the caller must ensure it
* can't go away. Callers that fail to ensure that the specified
* CPU cannot go away will execute on a randomly chosen CPU.
* But note well that callers specifying a CPU that never has been
* online will get a splat.
*
* Return: %false if @work was already on a queue, %true otherwise.
*/
In that case, making offline CPUs have a unbound workqueue is wrong. no?
It might encourage more users to abuse queue_work_on interface to
send to offline CPUs without any checks and onus now falls onto
workqueue to disaptch to unbound wqs.
So I think it is better to put the guardrails in SRCU instead of any change in
workqueue.
