06 Mar 2026 17:57:26 Matthieu Baerts <[email protected]>:
(...)
> After having sent this email, I re-checked on my side, and I was able to
> reproduce this issue with the technique described above: using the
> docker image with "build" argument, then max 50 boot iterations with "vm
> auto normal" argument. I then used 'git bisect' between v6.18 and
> v6.19-rc1 to find the guilty commit, and got:
>
> 653fda7ae73d ("sched/mmcid: Switch over to the new mechanism")
>
> Reverting it on top of v6.19-rc1 fixes the issue.
>
> Unfortunatelly, reverting it on top of Linus' tree causes some
> conflicts. I did my best to resolve them, and with this patch attached
> below -- also available in [1] -- I no longer have the issue. I don't
> know if it is correct -- some quick tests don't show any issues -- nor
> if Jiri should test it. I guess the final fix will be different from
> this simple revert.
As probably expected, even if this revert fixed the boot issues, it
caused a regression during the tests execution, see below.
[ 493.608357][ C3] rcu: 3-....: (26000 ticks this GP)
idle=d54c/1/0x4000000000000000 softirq=214151/214154 fqs=6500
[ 493.609867][ C3] rcu: (t=26003 jiffies g=392697 q=1127 ncpus=4)
[ 493.610566][ C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G
N 7.0.0-rc2+ #1 PREEMPT(full)
[ 493.610575][ C3] Tainted: [N]=TEST
[ 493.610577][ C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 493.610583][ C3] RIP: 0010:sched_mm_cid_after_execve
(kernel/sched/sched.h:4026 (discriminator 2))
[ 493.610596][ C3] Code: 05 00 00 4c 89 e8 48 c1 f8 06 4c 89 4c 24 08 49 8d
bc c1 10 0b 00 00 e8 99 4a 90 00 4c 8b 4c 24 08 f0 4d 0f ab a9 10 0b 00 00 <0f>
82 b7 00 00 00 48 8b 54 24 10 44 8b 44 24 1c 48 b8 00 00 00 00
All code
========
0: 05 00 00 4c 89 add $0x894c0000,%eax
5: e8 48 c1 f8 06 call 0x6f8c152
a: 4c 89 4c 24 08 mov %r9,0x8(%rsp)
f: 49 8d bc c1 10 0b 00 lea 0xb10(%r9,%rax,8),%rdi
16: 00
17: e8 99 4a 90 00 call 0x904ab5
1c: 4c 8b 4c 24 08 mov 0x8(%rsp),%r9
21: f0 4d 0f ab a9 10 0b lock bts %r13,0xb10(%r9)
28: 00 00
2a:* 0f 82 b7 00 00 00 jb 0xe7 <-- trapping instruction
30: 48 8b 54 24 10 mov 0x10(%rsp),%rdx
35: 44 8b 44 24 1c mov 0x1c(%rsp),%r8d
3a: 48 rex.W
3b: b8 00 00 00 00 mov $0x0,%eax
Code starting with the faulting instruction
===========================================
0: 0f 82 b7 00 00 00 jb 0xbd
6: 48 8b 54 24 10 mov 0x10(%rsp),%rdx
b: 44 8b 44 24 1c mov 0x1c(%rsp),%r8d
10: 48 rex.W
11: b8 00 00 00 00 mov $0x0,%eax
[ 493.610600][ C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000247
[ 493.610605][ C3] RAX: 0000000000000001 RBX: 0000000000000001 RCX:
0000000000000001
[ 493.610609][ C3] RDX: 0000000000000001 RSI: 0000000000000008 RDI:
ffff8881000aba10
[ 493.610612][ C3] RBP: dffffc0000000000 R08: ffffffff8185fe97 R09:
ffff8881000aaf00
[ 493.610615][ C3] R10: ffffed1020015743 R11: 0000000000000000 R12:
ffffed1021dda7a8
[ 493.610618][ C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15:
ffff88810eed3800
[ 493.610663][ C3] FS: 0000000000000000(0000) GS:ffff8881cc110000(0000)
knlGS:0000000000000000
[ 493.610680][ C3] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 493.610684][ C3] CR2: 00007ffef3a7dd89 CR3: 00000001204a1005 CR4:
0000000000370ef0
[ 493.610688][ C3] Call Trace:
[ 493.610691][ C3] <TASK>
[ 493.610703][ C3] bprm_execve (include/linux/rseq.h:140)
[ 493.610717][ C3] do_execveat_common.isra.0 (fs/exec.c:1846)
[ 493.610731][ C3] __x64_sys_execve (include/linux/fs.h:2539)
[ 493.610740][ C3] do_syscall_64 (arch/x86/entry/syscall_64.c:63
(discriminator 1))
[ 493.610750][ C3] ? exc_page_fault (arch/x86/mm/fault.c:1480
(discriminator 3))
[ 493.610760][ C3] entry_SYSCALL_64_after_hwframe
(arch/x86/entry/entry_64.S:130)
[ 493.610767][ C3] RIP: 0033:0x7fb401448140
[ 493.610780][ C3] Code: Unable to access opcode bytes at 0x7fb401448116.
Code starting with the faulting instruction
===========================================
[ 493.610782][ C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX:
000000000000003b
[ 493.610787][ C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
0000000000000000
[ 493.610790][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 493.610793][ C3] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 493.610795][ C3] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 493.610796][ C3] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 493.610815][ C3] </TASK>
main_loop_s: timed out
[ 503.574195][ T17] rcu: INFO: rcu_preempt detected expedited stalls on
CPUs/tasks: { 3-.... } 35898 jiffies s: 8761 root: 0x8/.
[ 503.575291][ T17] rcu: blocking rcu_node structures (internal RCU debug):
[ 503.576282][ T17] Sending NMI from CPU 1 to CPUs 3:
[ 503.577258][ C3] NMI backtrace for cpu 3
[ 503.577269][ C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G
N 7.0.0-rc2+ #1 PREEMPT(full)
[ 503.577277][ C3] Tainted: [N]=TEST
[ 503.577279][ C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 503.577282][ C3] RIP: 0010:sched_mm_cid_after_execve
(kernel/sched/sched.h:4022)
[ 503.577295][ C3] Code: 34 2e 40 38 f0 7c 09 40 84 f6 0f 85 6e 01 00 00 8b
35 d6 72 d5 02 49 8d be 10 0b 00 00 e8 b6 cc f9 00 49 89 c5 41 80 3c 24 00 <0f>
85 5f 01 00 00 4d 8b b7 40 05 00 00 41 39 dd 0f 83 1e fd ff ff
All code
========
0: 34 2e xor $0x2e,%al
2: 40 38 f0 cmp %sil,%al
5: 7c 09 jl 0x10
7: 40 84 f6 test %sil,%sil
a: 0f 85 6e 01 00 00 jne 0x17e
10: 8b 35 d6 72 d5 02 mov 0x2d572d6(%rip),%esi # 0x2d572ec
16: 49 8d be 10 0b 00 00 lea 0xb10(%r14),%rdi
1d: e8 b6 cc f9 00 call 0xf9ccd8
22: 49 89 c5 mov %rax,%r13
25: 41 80 3c 24 00 cmpb $0x0,(%r12)
2a:* 0f 85 5f 01 00 00 jne 0x18f <-- trapping instruction
30: 4d 8b b7 40 05 00 00 mov 0x540(%r15),%r14
37: 41 39 dd cmp %ebx,%r13d
3a: 0f 83 1e fd ff ff jae 0xfffffffffffffd5e
Code starting with the faulting instruction
===========================================
0: 0f 85 5f 01 00 00 jne 0x165
6: 4d 8b b7 40 05 00 00 mov 0x540(%r15),%r14
d: 41 39 dd cmp %ebx,%r13d
10: 0f 83 1e fd ff ff jae 0xfffffffffffffd34
[ 503.577300][ C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000246
[ 503.577305][ C3] RAX: 0000000000000001 RBX: 0000000000000001 RCX:
0000000000000001
[ 503.577308][ C3] RDX: 0000000000000001 RSI: dffffc0000000000 RDI:
ffff8881000aba10
[ 503.577311][ C3] RBP: dffffc0000000000 R08: ffffffff8185fe97 R09:
ffff8881000aaf00
[ 503.577314][ C3] R10: ffffed1020015743 R11: 0000000000000000 R12:
ffffed1021dda7a8
[ 503.577316][ C3] R13: 0000000000000001 R14: ffff8881000aaf00 R15:
ffff88810eed3800
[ 503.577335][ C3] FS: 0000000000000000(0000) GS:ffff8881cc110000(0000)
knlGS:0000000000000000
[ 503.577350][ C3] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 503.577353][ C3] CR2: 00007fb401448116 CR3: 00000001204a1005 CR4:
0000000000370ef0
[ 503.577356][ C3] Call Trace:
[ 503.577360][ C3] <TASK>
[ 503.577368][ C3] bprm_execve (include/linux/rseq.h:140)
[ 503.577377][ C3] do_execveat_common.isra.0 (fs/exec.c:1846)
[ 503.577385][ C3] __x64_sys_execve (include/linux/fs.h:2539)
[ 503.577392][ C3] do_syscall_64 (arch/x86/entry/syscall_64.c:63
(discriminator 1))
[ 503.577401][ C3] ? exc_page_fault (arch/x86/mm/fault.c:1480
(discriminator 3))
[ 503.577408][ C3] entry_SYSCALL_64_after_hwframe
(arch/x86/entry/entry_64.S:130)
[ 503.577415][ C3] RIP: 0033:0x7fb401448140
[ 503.577427][ C3] Code: Unable to access opcode bytes at 0x7fb401448116.
Code starting with the faulting instruction
===========================================
[ 503.577430][ C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX:
000000000000003b
[ 503.577435][ C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
0000000000000000
[ 503.577437][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 503.577439][ C3] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 503.577442][ C3] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 503.577444][ C3] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 503.577453][ C3] </TASK>
[ 530.927646][ C3] watchdog: BUG: soft lockup - CPU#3 stuck for 60s!
[sleep:4961]
[ 530.927660][ C3] Modules linked in: nft_tproxy nf_tproxy_ipv6
nf_tproxy_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 nf_tables sch_netem
tcp_diag mptcp_diag inet_diag mptcp_token_test mptcp_crypto_test kunit
[ 530.927693][ C3] irq event stamp: 141062
[ 530.927696][ C3] hardirqs last enabled at (141061): irqentry_exit
(kernel/entry/common.c:243)
[ 530.927711][ C3] hardirqs last disabled at (141062):
sysvec_apic_timer_interrupt (arch/x86/include/asm/hardirq.h:81)
[ 530.927715][ C3] softirqs last enabled at (140962): handle_softirqs
(kernel/softirq.c:469 (discriminator 2))
[ 530.927720][ C3] softirqs last disabled at (140949): __irq_exit_rcu
(kernel/softirq.c:657)
[ 530.927727][ C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G
N 7.0.0-rc2+ #1 PREEMPT(full)
[ 530.927732][ C3] Tainted: [N]=TEST
[ 530.927734][ C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 530.927736][ C3] RIP: 0010:sched_mm_cid_after_execve
(kernel/sched/sched.h:4045 (discriminator 6))
[ 530.927742][ C3] Code: 02 83 04 85 c0 0f 84 6b 02 00 00 48 83 c4 30 5b 5d
41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 c7 c7 40 f5 cd 83 e8 bd df 19 02 <49>
8d be 00 01 00 00 48 89 f8 48 c1 e8 03 80 3c 28 00 0f 85 65 02
All code
========
0: 02 83 04 85 c0 0f add 0xfc08504(%rbx),%al
6: 84 6b 02 test %ch,0x2(%rbx)
9: 00 00 add %al,(%rax)
b: 48 83 c4 30 add $0x30,%rsp
f: 5b pop %rbx
10: 5d pop %rbp
11: 41 5c pop %r12
13: 41 5d pop %r13
15: 41 5e pop %r14
17: 41 5f pop %r15
19: c3 ret
1a: cc int3
1b: cc int3
1c: cc int3
1d: cc int3
1e: 48 c7 c7 40 f5 cd 83 mov $0xffffffff83cdf540,%rdi
25: e8 bd df 19 02 call 0x219dfe7
2a:* 49 8d be 00 01 00 00 lea 0x100(%r14),%rdi <-- trapping instruction
31: 48 89 f8 mov %rdi,%rax
34: 48 c1 e8 03 shr $0x3,%rax
38: 80 3c 28 00 cmpb $0x0,(%rax,%rbp,1)
3c: 0f .byte 0xf
3d: 85 65 02 test %esp,0x2(%rbp)
Code starting with the faulting instruction
===========================================
0: 49 8d be 00 01 00 00 lea 0x100(%r14),%rdi
7: 48 89 f8 mov %rdi,%rax
a: 48 c1 e8 03 shr $0x3,%rax
e: 80 3c 28 00 cmpb $0x0,(%rax,%rbp,1)
12: 0f .byte 0xf
13: 85 65 02 test %esp,0x2(%rbp)
[ 530.927744][ C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000202
[ 530.927747][ C3] RAX: 0000000000000003 RBX: 0000000000000001 RCX:
0000000000000001
[ 530.927749][ C3] RDX: 0000000000000001 RSI: ffffffff83cdf540 RDI:
ffffffff83eec920
[ 530.927751][ C3] RBP: dffffc0000000000 R08: ffffffff8185fd4a R09:
0000000000000000
[ 530.927752][ C3] R10: 0000000000000003 R11: 0000000000000000 R12:
ffffed1021dda7a8
[ 530.927753][ C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15:
ffff88810eed3800
[ 530.927766][ C3] FS: 0000000000000000(0000) GS:ffff8881cc110000(0000)
knlGS:0000000000000000
[ 530.927776][ C3] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 530.927778][ C3] CR2: 00007fb401448116 CR3: 00000001204a1005 CR4:
0000000000370ef0
[ 530.927780][ C3] Call Trace:
[ 530.927784][ C3] <TASK>
[ 530.927792][ C3] bprm_execve (include/linux/rseq.h:140)
[ 530.927801][ C3] do_execveat_common.isra.0 (fs/exec.c:1846)
[ 530.927807][ C3] __x64_sys_execve (include/linux/fs.h:2539)
[ 530.927812][ C3] do_syscall_64 (arch/x86/entry/syscall_64.c:63
(discriminator 1))
[ 530.927817][ C3] ? exc_page_fault (arch/x86/mm/fault.c:1480
(discriminator 3))
[ 530.927821][ C3] entry_SYSCALL_64_after_hwframe
(arch/x86/entry/entry_64.S:130)
[ 530.927826][ C3] RIP: 0033:0x7fb401448140
[ 530.927835][ C3] Code: Unable to access opcode bytes at 0x7fb401448116.
Code starting with the faulting instruction
===========================================
[ 530.927837][ C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX:
000000000000003b
[ 530.927839][ C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
0000000000000000
[ 530.927840][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 530.927842][ C3] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 530.927843][ C3] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 530.927844][ C3] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 530.927854][ C3] </TASK>
[ 530.927857][ C3] Kernel panic - not syncing: softlockup: hung tasks
[ 530.958801][ C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G
L N 7.0.0-rc2+ #1 PREEMPT(full)
[ 530.960019][ C3] Tainted: [L]=SOFTLOCKUP, [N]=TEST
[ 530.960652][ C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 530.961437][ C3] Call Trace:
[ 530.961931][ C3] <IRQ>
[ 530.962208][ C3] dump_stack_lvl (lib/dump_stack.c:122)
[ 530.962740][ C3] vpanic (kernel/panic.c:651)
[ 530.963120][ C3] panic (kernel/panic.c:787)
[ 530.963502][ C3] ? __pfx_panic (kernel/panic.c:783)
[ 530.964024][ C3] ? add_taint (arch/x86/include/asm/bitops.h:60)
[ 530.964459][ C3] watchdog_timer_fn.cold (kernel/watchdog.c:871)
[ 530.965081][ C3] ? __pfx_watchdog_timer_fn (kernel/watchdog.c:774)
[ 530.965629][ C3] __run_hrtimer (kernel/time/hrtimer.c:1785)
[ 530.966417][ C3] __hrtimer_run_queues (include/linux/timerqueue.h:25)
[ 530.966930][ C3] ? __pfx___hrtimer_run_queues
(kernel/time/hrtimer.c:1819)
[ 530.967470][ C3] ? ktime_get_update_offsets_now
(kernel/time/timekeeping.c:381)
[ 530.968251][ C3] hrtimer_interrupt (kernel/time/hrtimer.c:1914)
[ 530.968990][ C3] __sysvec_apic_timer_interrupt
(arch/x86/include/asm/jump_label.h:37)
[ 530.969783][ C3] sysvec_apic_timer_interrupt
(arch/x86/kernel/apic/apic.c:1056 (discriminator 47))
[ 530.970343][ C3] </IRQ>
[ 530.970723][ C3] <TASK>
[ 530.971036][ C3] asm_sysvec_apic_timer_interrupt
(arch/x86/include/asm/idtentry.h:697)
[ 530.971947][ C3] RIP: 0010:sched_mm_cid_after_execve
(kernel/sched/sched.h:4045 (discriminator 6))
[ 530.972671][ C3] Code: 02 83 04 85 c0 0f 84 6b 02 00 00 48 83 c4 30 5b 5d
41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 c7 c7 40 f5 cd 83 e8 bd df 19 02 <49>
8d be 00 01 00 00 48 89 f8 48 c1 e8 03 80 3c 28 00 0f 85 65 02
All code
========
0: 02 83 04 85 c0 0f add 0xfc08504(%rbx),%al
6: 84 6b 02 test %ch,0x2(%rbx)
9: 00 00 add %al,(%rax)
b: 48 83 c4 30 add $0x30,%rsp
f: 5b pop %rbx
10: 5d pop %rbp
11: 41 5c pop %r12
13: 41 5d pop %r13
15: 41 5e pop %r14
17: 41 5f pop %r15
19: c3 ret
1a: cc int3
1b: cc int3
1c: cc int3
1d: cc int3
1e: 48 c7 c7 40 f5 cd 83 mov $0xffffffff83cdf540,%rdi
25: e8 bd df 19 02 call 0x219dfe7
2a:* 49 8d be 00 01 00 00 lea 0x100(%r14),%rdi <-- trapping instruction
31: 48 89 f8 mov %rdi,%rax
34: 48 c1 e8 03 shr $0x3,%rax
38: 80 3c 28 00 cmpb $0x0,(%rax,%rbp,1)
3c: 0f .byte 0xf
3d: 85 65 02 test %esp,0x2(%rbp)
Code starting with the faulting instruction
===========================================
0: 49 8d be 00 01 00 00 lea 0x100(%r14),%rdi
7: 48 89 f8 mov %rdi,%rax
a: 48 c1 e8 03 shr $0x3,%rax
e: 80 3c 28 00 cmpb $0x0,(%rax,%rbp,1)
12: 0f .byte 0xf
13: 85 65 02 test %esp,0x2(%rbp)
[ 530.974394][ C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000202
[ 530.975308][ C3] RAX: 0000000000000003 RBX: 0000000000000001 RCX:
0000000000000001
[ 530.976147][ C3] RDX: 0000000000000001 RSI: ffffffff83cdf540 RDI:
ffffffff83eec920
[ 530.977030][ C3] RBP: dffffc0000000000 R08: ffffffff8185fd4a R09:
0000000000000000
[ 530.977910][ C3] R10: 0000000000000003 R11: 0000000000000000 R12:
ffffed1021dda7a8
[ 530.978926][ C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15:
ffff88810eed3800
[ 530.979595][ C3] ? sched_mm_cid_after_execve
(arch/x86/include/asm/bitops.h:136 (discriminator 1))
[ 530.980252][ C3] bprm_execve (include/linux/rseq.h:140)
[ 530.980941][ C3] do_execveat_common.isra.0 (fs/exec.c:1846)
[ 530.981424][ C3] __x64_sys_execve (include/linux/fs.h:2539)
[ 530.981941][ C3] do_syscall_64 (arch/x86/entry/syscall_64.c:63
(discriminator 1))
[ 530.982406][ C3] ? exc_page_fault (arch/x86/mm/fault.c:1480
(discriminator 3))
[ 530.983097][ C3] entry_SYSCALL_64_after_hwframe
(arch/x86/entry/entry_64.S:130)
[ 530.983809][ C3] RIP: 0033:0x7fb401448140
[ 530.984409][ C3] Code: Unable to access opcode bytes at 0x7fb401448116.
Code starting with the faulting instruction
===========================================
[ 530.985240][ C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX:
000000000000003b
[ 530.986149][ C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
0000000000000000
[ 530.986945][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 530.987807][ C3] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 530.988490][ C3] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 530.989334][ C3] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 530.990257][ C3] </TASK>
[ 530.991655][ C3] Kernel Offset: disabled
More details:
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/22775627304#summary-66068034344
Cheers,
Matt
