What you're observing isn't a bug, but rather a result of how the Linux
scheduler interacts with the isolcpus boot parameter and the default
scheduling policy (SCHED_OTHER).
When you use isolcpus=1-3, those CPUs are removed from the kernel’s
load-balancing domains. As a result, under SCHED_OTHER, the scheduler
does not actively migrate threads between those isolated CPUs. So when a
multi-threaded program is launched with taskset -c 1-3, all threads
typically stay on the CPU where they were first scheduled (in your case
CPU 3), leading to the imbalance you observed.
In contrast, real-time scheduling policies like SCHED_FIFO (used with
chrt -f) don’t rely on load balancing at all. Threads are immediately
scheduled on any available CPU within their affinity mask, leading to
the even distribution you see when using chrt.
** Changed in: linux (Ubuntu)
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2116749
Title:
Thread affinity broken under isolcpus
Status in linux package in Ubuntu:
Invalid
Bug description:
I'd like to report a bug with unexpected scheduling of programs run
with `taskset` when target cores have been reserved via `isolcpus`
setting (e.g. for benchmark stability purposes).
When I try to run multi-threaded program on reserved cores:
```
$ taskset -c 1-3 ./a.out
```
OS will pin all of them to single core for the whole duration of lifetime.
I would instead expect them to be distributed evenly across allowed
cores (1-3 in this case). Threads are distributed correctly if program
is run under `chrt -f 1`.
The attached small test program illustrates issues with isolcpus and
taskset (see below). Results were obtained for Ubuntu 24.04 running on
4-core VirtualBox VM (I've also reproduced them on real PC with Ubuntu
22.04). Bug has also been reported previously (by another user) in
https://serverfault.com/questions/573025/taskset-not-working-over-a-
range-of-cores-in-isolcpus
Isolcpus is very useful for obtaining stable CPU benchmark results and
effective lack of support for multi-threaded apps is unfortunate.
Everything works as expected when isolcpus is not set:
1) without `taskset` threads run on all cores:
```
yugr@yugr-VirtualBox:~/tasks/isolcpus$ gcc -O2 test.c -pthread
yugr@yugr-VirtualBox:~/tasks/isolcpus$ ./a.out
Thread 7 was scheduled on 1 2
Thread 5 was scheduled on 2 3
Thread 11 was scheduled on 0 2 3
Thread 1 was scheduled on 1 2 3
Thread 2 was scheduled on 0 1 3
Thread 8 was scheduled on 0 1
Thread 6 was scheduled on 0 1 2
Thread 9 was scheduled on 0 1 2 3
Thread 4 was scheduled on 1 2 3
Thread 10 was scheduled on 0 2 3
Thread 0 was scheduled on 0 1 3
Thread 3 was scheduled on 0 1 2 3
```
2) with taskset threads run according to the mask:
```
yugr@yugr-VirtualBox:~/tasks/isolcpus$ taskset -c 2-3 ./a.out
Thread 3 was scheduled on 2
Thread 4 was scheduled on 2
Thread 0 was scheduled on 2
Thread 5 was scheduled on 2 3
Thread 2 was scheduled on 2 3
Thread 10 was scheduled on 2 3
Thread 11 was scheduled on 2 3
Thread 1 was scheduled on 2 3
Thread 9 was scheduled on 2 3
Thread 8 was scheduled on 2 3
Thread 6 was scheduled on 3
Thread 7 was scheduled on 2 3
```
And here are results after adding `isolcpus=1-3` to
`GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` and doing `sudo
update-grub`.
1) without taskset results are expected (everything goes to core 0):
```
yugr@yugr-VirtualBox:~/tasks/isolcpus$ ./a.out
Thread 11 was scheduled on 0
Thread 3 was scheduled on 0
Thread 0 was scheduled on 0
Thread 1 was scheduled on 0
Thread 6 was scheduled on 0
Thread 8 was scheduled on 0
Thread 4 was scheduled on 0
Thread 7 was scheduled on 0
Thread 5 was scheduled on 0
Thread 10 was scheduled on Thread 9 was scheduled on 0
Thread 2 was scheduled on 0
0
```
2) but with taskset for some reason everything goes to single core (rather
than three, 1-3) as well:
```
yugr@yugr-VirtualBox:~/tasks/isolcpus$ taskset -c 1-3 ./a.out
Thread 3 was scheduled on 3
Thread 2 was scheduled on 3
Thread 0 was scheduled on 3
Thread 9 was scheduled on 3
Thread 8 was scheduled on 3
Thread 1 was scheduled on Thread 6 was scheduled on Thread 5 was scheduled on
3
Thread 4 was scheduled on 3
3
3
Thread 11 was scheduled on Thread 7 was scheduled on Thread 10 was scheduled
on 3
3
3
```
Everything fixed with `SCHED_FIFO` policy:
```
yugr@yugr-VirtualBox:~/tasks/isolcpus$ chrt -f 1 taskset -c 1-3 ./a.out
Thread 2 was scheduled on 1
Thread 1 was scheduled on 2
Thread 0 was scheduled on 3
Thread 5 was scheduled on 3
Thread 4 was scheduled on 2
Thread 3 was scheduled on 1
Thread 7 was scheduled on 2
Thread 6 was scheduled on 3
Thread 8 was scheduled on 1
Thread 9 was scheduled on 2
Thread 10 was scheduled on 3
Thread 11 was scheduled on 1
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2116749/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
