Hi CWills,
I have been reading through the information you provided, and it is very
interesting. It really is incredible how far LLMs have come.
I read the patches in the patches/ directory, and at first glance, they all seem
quite reasonable and plausible.
Now, I did some testing, against the current HEAD of Linus's tree, which is at
7.3-rc3 9b87fdc9af2fbfcdb5c24a64139685ef80f6573f at the time of writing.
The first build is as the mainline kernel is, and the second has all 5 patches
applied. I had to adjust the context in patch 3, but it was exactly what was
mentioned in your comment about changing to mm/folio.c, and everything compiles
cleanly.
I used the AWS config from the current stonking 7.2.0-1001-aws kernel.
With no patches, things reproduce, as expected.
Unfortunately, when running my reproducer against the patched kernel, the issue
still occurs, and there doesn't seem to be an improvement.
root@ip-172-31-27-241:/home/ubuntu/reproducer# uname -rv
7.3.0-rc3+test2patches+ #2 SMP PREEMPT Wed Sep 16 06:22:19 UTC 2026
[*] PID 1 entered D state at 06:40:05...
[+] PID 1 recovered. Total duration in D state: 273.383s
[*] PID 1 entered D state at 06:50:02...
Failed to start transient scope unit: Unit repro-workers.scope was already
loaded or has a fragment file.
[+] PID 1 recovered. Total duration in D state: 194.861s
[*] PID 1 entered D state at 07:00:03...
[+] PID 1 recovered. Total duration in D state: 199.260s
[*] PID 1 entered D state at 07:40:05...
[+] PID 1 recovered. Total duration in D state: 203.533s
Looking at the dmesg output, the same call traces are present. Attached
below.
Some runs, I never managed to get it to D state, but upon closer inspection,
NUMA migration never occurred, even though it was explicitly asked for:
root@ip-172-31-27-241:/home/ubuntu# numastat -c anchor_alloc || true
Per-node process memory usage (in MBs) for PID 71714 (anchor_alloc)
Node 0 Node 1 Total
------ ------ -----
Huge 0 0 0
Heap 0 0 0
Stack 0 0 0
Private 1 51200 51202
------- ------ ------ -----
Total 1 51200 51202
root@ip-172-31-27-241:/home/ubuntu# systemctl set-property ${SLICE}
AllowedMemoryNodes=0
root@ip-172-31-27-241:/home/ubuntu# numastat -c anchor_alloc || true
Per-node process memory usage (in MBs) for PID 71714 (anchor_alloc)
Node 0 Node 1 Total
------ ------ -----
Huge 0 0 0
Heap 0 0 0
Stack 0 0 0
Private 1 51200 51202
------- ------ ------ -----
Total 1 51200 51202
root@ip-172-31-27-241:/home/ubuntu# numastat -c anchor_alloc || true
Per-node process memory usage (in MBs) for PID 71714 (anchor_alloc)
Node 0 Node 1 Total
------ ------ -----
Huge 0 0 0
Heap 0 0 0
Stack 0 0 0
Private 1 51200 51202
------- ------ ------ -----
Total 1 51200 51202
I do wonder what is causing NUMA migration to no longer work.
Regardless, unfortunately the patches don't seem to work as intended, and this
is going to need some deeper research. I will write back once I have looked into
this more.
Thanks,
Matthew
kernel: INFO: task systemd:1 blocked for more than 61 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025498 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 00000000000012d9 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380254f0
kernel: RBP: 00007ffc38025730 R08: 00007ffc38025550 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025550 R14: 00007ffc38025840 R15: 00007ffc38025ad0
kernel: </TASK>
kernel: INFO: task systemd:1 blocked for more than 122 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025498 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 00000000000012d9 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380254f0
kernel: RBP: 00007ffc38025730 R08: 00007ffc38025550 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025550 R14: 00007ffc38025840 R15: 00007ffc38025ad0
kernel: </TASK>
kernel: INFO: task systemd:1 blocked for more than 184 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025498 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 00000000000012d9 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380254f0
kernel: RBP: 00007ffc38025730 R08: 00007ffc38025550 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025550 R14: 00007ffc38025840 R15: 00007ffc38025ad0
kernel: </TASK>
kernel: INFO: task systemd:1 blocked for more than 245 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025498 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 00000000000012d9 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380254f0
kernel: RBP: 00007ffc38025730 R08: 00007ffc38025550 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025550 R14: 00007ffc38025840 R15: 00007ffc38025ad0
kernel: </TASK>
kernel: INFO: task systemd:1 blocked for more than 61 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? mmap_region+0x9a/0x110
kernel: ? __get_unmapped_area+0xbc/0x1b0
kernel: ? do_mmap+0x531/0x700
kernel: ? vm_mmap_pgoff+0x164/0x210
kernel: ? ksys_mmap_pgoff+0x44/0x280
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __x64_sys_mmap+0x36/0x70
kernel: ? do_syscall_64+0x207/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __x64_sys_prctl+0x21/0x40
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025558 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 000000000000c240 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380255b0
kernel: RBP: 00007ffc380257f0 R08: 00007ffc38025610 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025610 R14: 00007ffc38025900 R15: 00007ffc38025b90
kernel: </TASK>
kernel: INFO: task systemd:1 blocked for more than 122 seconds.
kernel: Not tainted 7.3.0-rc3+test2patches+ #2
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd state:D stack:0 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x0>
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x40f/0x1ae0
kernel: ? schedule_timeout+0xcf/0x110
kernel: schedule+0x28/0xb0
kernel: schedule_timeout+0xcf/0x110
kernel: wait_for_completion+0x7c/0x140
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? mmap_region+0x9a/0x110
kernel: ? __get_unmapped_area+0xbc/0x1b0
kernel: ? do_mmap+0x531/0x700
kernel: ? vm_mmap_pgoff+0x164/0x210
kernel: ? ksys_mmap_pgoff+0x44/0x280
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __x64_sys_mmap+0x36/0x70
kernel: ? do_syscall_64+0x207/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: __flush_workqueue+0x14e/0x430
kernel: flush_migrate_mm_task_workfn+0x19/0x30
kernel: task_work_run+0x62/0x90
kernel: exit_to_user_mode_loop+0x1f4/0x5b0
kernel: do_syscall_64+0x251/0x550
kernel: ? mmap_region+0x9a/0x110
kernel: ? __get_unmapped_area+0xbc/0x1b0
kernel: ? do_mmap+0x531/0x700
kernel: ? vm_mmap_pgoff+0x164/0x210
kernel: ? ksys_mmap_pgoff+0x44/0x280
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __x64_sys_mmap+0x36/0x70
kernel: ? do_syscall_64+0x207/0x550
kernel: ? do_syscall_64+0x207/0x550
kernel: ? __do_sys_prctl+0x3f/0xde0
kernel: ? __ct_user_enter+0x2c/0x100
kernel: ? __x64_sys_prctl+0x21/0x40
kernel: ? __ct_user_exit+0x1a/0xf0
kernel: ? do_syscall_64+0x33/0x550
kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x75c98b73f54d
kernel: RSP: 002b:00007ffc38025558 EFLAGS: 00000202 ORIG_RAX: 00000000000001b3
kernel: RAX: 000000000000c240 RBX: 0000000000009000 RCX: 000075c98b73f54d
kernel: RDX: 000075c98b723c70 RSI: 0000000000000058 RDI: 00007ffc380255b0
kernel: RBP: 00007ffc380257f0 R08: 00007ffc38025610 R09: 0000000000000000
kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 000075c98b828000
kernel: R13: 00007ffc38025610 R14: 00007ffc38025900 R15: 00007ffc38025b90
kernel: </TASK>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165410
Title:
Ubuntu LTS 26.04 linux-aws: systemd enters D state and blocks SSH
during cpuset migration on nohz_full CPUs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2165410/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
