+rth (maintainer: accel/tcg/tcg-accel-ops-rr.c)
Hi,
On 11/02/2026 09:12, Yodel Eldar wrote:
+Ilya (author of mentioned commit)
Hi, all
On 10/02/2026 05:14, Akihiko Odaki wrote:
On 2026/02/10 20:04, Jim MacArthur wrote:
On 2/9/26 12:03, Thomas Huth wrote:
From: Thomas Huth <[email protected]>
This reverts commit 55d98e3edeeb17dd8445db27605d2b34f4c3ba85.
The commit introduced a regression in the replay functional test
on alpha (tests/functional/alpha/test_replay.py), that causes CI
failures regularly. Thus revert this change until someone has
figured out what is going wrong here.
Buglink: https://gitlab.com/qemu-project/qemu/-/issues/3197
Signed-off-by: Thomas Huth <[email protected]>
---
I just hit this again in the CI:
https://gitlab.com/thuth/qemu/-/jobs/13018713844
... it's annoying, so I'd like to suggest to revert the offending
patch 'til someone fixed this.
Hi Thomas,
I am actively investigating this problem, but haven't made any
important progress so far. It's looking like a timing bug which is
annoying to track down. All I know for certain at the moment is that
faking some of the values returned by qemu_clock_get_ns to be
deterministic improves the failure rate, but does not remove it
entirely. I'm fine with this being reverted for now; we can add it
back in once we figure out what the true cause is.
Thanks for investigating this. Let's have the commit reverted until
there is some progress:
Reviewed-by: Akihiko Odaki <[email protected]>
After the quiescence commit, timeouts were the only regression, until
f098c32db48 ("target/s390x: Fix infinite loop during replay") when the
replay test began failing with error, too; maybe this serves as a
diagnostic trigger?
Yodel
Seems like the force_rcu notifier in rr tcg became incompatible with
record/replay after Akihiko's patch; preventing registration of the
notifier when in record/replay mode appears to fix the problem:
diff --git a/accel/tcg/tcg-accel-ops-rr.c b/accel/tcg/tcg-accel-ops-rr.c
index f84342e044..2073b1b13e 100644
--- a/accel/tcg/tcg-accel-ops-rr.c
+++ b/accel/tcg/tcg-accel-ops-rr.c
@@ -179,13 +179,15 @@ static int rr_cpu_count(void)
static void *rr_cpu_thread_fn(void *arg)
{
- Notifier force_rcu;
CPUState *cpu = arg;
assert(tcg_enabled());
rcu_register_thread();
- force_rcu.notify = rr_force_rcu;
- rcu_add_force_rcu_notifier(&force_rcu);
+ if (replay_mode == REPLAY_MODE_NONE) {
+ Notifier force_rcu;
+ force_rcu.notify = rr_force_rcu;
+ rcu_add_force_rcu_notifier(&force_rcu);
+ }
tcg_register_thread();
bql_lock();
as confirmed by running:
meson test --setup thorough --no-rebuild -t 0 \
--num-processes 1 --repeat 100 func-alpha-replay
I'd like to do some additional testing/analysis to make sure this
doesn't just trade one bug for another as I continue to study the
internals of RCU and replay/record, but any comments regarding the above
fix would be much appreciated.
E.g., could we continue using a force_rcu notifier with record/replay if
we added an async replay event for it, or by some other means?
Should we?
Furthermore, mttcg also uses a force_rcu notifier, but record/replay
isn't supported with multiple CPUs, so are we really in the clear there?
Also, the issue that firstly motivated rr_force_rcu():
https://gitlab.com/qemu-project/qemu/-/issues/650
doesn't appear at first glance to occur in record/replay mode when the
notifier isn't registered during record/replay; is that thanks to
Akihiko's patch or was that never a problem in record/replay mode?
Side note: eliminating the force_rcu notifier for non-record/replay mode
does cause the same regression as in issue #650; i.e., rr_force_rcu is
still seemingly necessary outside of replay/record mode.
Here's the backtrace of qemu-system-alpha as it hangs in replay mode:
------------------------------------------------------------------------
Thread 4 (Thread 0x7f2b841ff6c0 (LWP 7653) "qemu-system-alp"):
#0 0x00007f2b88c0906d in syscall () at /usr/lib/libc.so.6
#1 0x000055692672e554 in qemu_futex_wait (f=0x556926ca6670
<rcu_call_ready_event>, val=0xffffffff) at /qemu/include/qemu/futex.h:47
#2 0x000055692672e768 in qemu_event_wait (ev=0x556926ca6670
<rcu_call_ready_event>) at ../util/event.c:162
#3 0x000055692673a87e in call_rcu_thread (opaque=0x0) at ../util/rcu.c:304
#4 0x000055692672d032 in qemu_thread_start (args=0x556965cbd7b0) at
../util/qemu-thread-posix.c:393
#5 0x00007f2b88b8797a in ??? () at /usr/lib/libc.so.6
#6 0x00007f2b88c0b2bc in ??? () at /usr/lib/libc.so.6
Thread 3 (Thread 0x7f2b831fe6c0 (LWP 7654) "qemu-system-alp"):
#0 0x00007f2b88b8ff32 in ??? () at /usr/lib/libc.so.6
#1 0x00007f2b88b8439c in ??? () at /usr/lib/libc.so.6
#2 0x00007f2b88b843e4 in ??? () at /usr/lib/libc.so.6
#3 0x00007f2b88bfe2f6 in ppoll () at /usr/lib/libc.so.6
#4 0x00007f2b8a690784 in ??? () at /usr/lib/libglib-2.0.so.0
#5 0x00007f2b8a690a17 in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#6 0x0000556926565683 in iothread_run (opaque=0x556965d17c70) at
../iothread.c:70
#7 0x000055692672d032 in qemu_thread_start (args=0x556965cf46e0) at
../util/qemu-thread-posix.c:393
#8 0x00007f2b88b8797a in ??? () at /usr/lib/libc.so.6
#9 0x00007f2b88c0b2bc in ??? () at /usr/lib/libc.so.6
Thread 2 (Thread 0x7f2b821fd6c0 (LWP 7655) "qemu-system-alp"):
#0 0x00007f2b88b8ff32 in ??? () at /usr/lib/libc.so.6
#1 0x00007f2b88b8439c in ??? () at /usr/lib/libc.so.6
#2 0x00007f2b88b8468c in ??? () at /usr/lib/libc.so.6
#3 0x00007f2b88b86e5e in pthread_cond_wait () at /usr/lib/libc.so.6
#4 0x000055692672c9ee in qemu_cond_wait_impl (cond=0x556965e6df40,
mutex=0x556926c821e0 <bql>, file=0x55692683204f "../system/cpus.c",
line=0x24e) at ../util/qemu-thread-posix.c:225
#5 0x000055692630728b in qemu_cond_wait_bql (cond=0x556965e6df40) at
../system/cpus.c:590
#6 0x000055692643b864 in rr_wait_io_event () at
../accel/tcg/tcg-accel-ops-rr.c:114
#7 0x000055692643bcc5 in rr_cpu_thread_fn (arg=0x556965e69aa0) at
../accel/tcg/tcg-accel-ops-rr.c:235
#8 0x000055692672d032 in qemu_thread_start (args=0x556965cf2ea0) at
../util/qemu-thread-posix.c:393
#9 0x00007f2b88b8797a in ??? () at /usr/lib/libc.so.6
#10 0x00007f2b88c0b2bc in ??? () at /usr/lib/libc.so.6
Thread 1 (Thread 0x7f2b8690f180 (LWP 7652) "qemu-system-alp"):
#0 0x00007f2b88b8ff32 in ??? () at /usr/lib/libc.so.6
#1 0x00007f2b88b8439c in ??? () at /usr/lib/libc.so.6
#2 0x00007f2b88b843e4 in ??? () at /usr/lib/libc.so.6
#3 0x00007f2b88bfe2f6 in ppoll () at /usr/lib/libc.so.6
#4 0x000055692674d35a in qemu_poll_ns (fds=0x5569667017a0, nfds=0x5,
timeout=0x3b597370) at ../util/qemu-timer.c:343
#5 0x00005569267485ee in os_host_main_loop_wait (timeout=0x3b597370) at
../util/main-loop.c:308
#6 0x0000556926748740 in main_loop_wait (nonblocking=0x0) at
../util/main-loop.c:592
#7 0x0000556926335226 in qemu_main_loop () at ../system/runstate.c:903
#8 0x000055692665e8cf in qemu_default_main (opaque=0x0) at
../system/main.c:50
#9 0x000055692665e989 in main (argc=0x19, argv=0x7ffdb6f6f898) at
../system/main.c:93
------------------------------------------------------------------------
Interestingly, the `shift` value used in the alpha replay test's
`-icount` option somewhat sneakily complicated debugging: When `shift`
is greater than eight (it's currently nine), both TIMEOUTs and ERRORs
would occur in `meson test ... func-alpha-replay` (after f098c32db48),
but when `shift` is lowered, only TIMEOUTs occur. With my fix, neither
occurs regardless of the `shift` value, but the kernel still seems to
mind as seen when given an initrd [1]:
./qemu-system-alpha -initrd ./initrd.gz \
-machine clipper \
-kernel ./vmlinuz \
-append 'console=ttyS0' \
-icount shift=9,rr=record,rrfile=replay.bin \
-nographic -net none
yielding this printk:
BUG: soft lockup - CPU#0 stuck for 61s! [udevd:219]
Is there an important reason why we set `shift=9` in the test?
If not, maybe we should lower it, too, while we're at it?
Please let me know if there's any other information that would be
helpful with resolving this bug in the best way possible once and for
all; I did quite a bit of testing/debugging over the last couple of days
and would be happy to share more of the results/methodology.
Thomas and Akihiko, if it's alright with you, could we delay the PR of
the revert at least until we sort this out? I realize the erratic
timeouts are gumming up the CI, but maybe we could reduce the deltas of
the git history in short order if it's not too much trouble? I can
submit the patch in one to two days; I just want to make sure it's done
correctly.
[1]
https://archive.debian.org/debian/dists/lenny/main/installer-alpha/20090123lenny10/images/cdrom/initrd.gz
Thanks,
Yodel
