The per-CPU TB jump cache has held 4096 entries since it was introduced.
That is too small for guests running large programs: an emulated compiler
misses often enough that the fallback qht lookup shows up prominently in
a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The
compile performs 34.2 billion TB executions, of which 8.4 billion take
the indirect dispatch path.
Sizing curve, on top of the preceding patch, instructions retired and
wall clock:
12 bits ( 64 KiB): 1,562,204,796,597 132.58s
14 bits ( 256 KiB): 1,493,318,515,396 -4.41% 124.67s -5.97%
16 bits ( 1 MiB): 1,469,772,951,575 -5.92% 121.04s -8.71%
18 bits ( 4 MiB): 1,462,309,832,762 -6.39% 119.82s -9.62%
16 bits is the knee. 18 buys another 0.47% of instructions for four times
the memory. It does show a further 1.01% of wall clock, which is outside
the 0.70% run-to-run spread at 16 bits, so the effect is probably real --
but paying four times the memory for it is a poor trade, and instructions
retired does not account for the data cache pressure of a 4 MiB table.
In a perf profile the mechanism is visible directly: tb_htable_lookup(),
which is where qht_lookup_custom() lands once it is inlined in an LTO
build, falls from 6.10% of samples to 1.66%.
The cost is memory: the cache grows from 64 KiB to 1 MiB, once per
CPUState. In linux-user that is per guest thread rather than per process,
so a threaded guest pays it as many times as it has threads, exactly as
system emulation pays it per vCPU. The allocation is g_new0(), so the
pages are faulted in as the cache is touched and a thread that runs a
small amount of code touches a small part of it, but the address space is
committed either way.
So this may still want to be tunable, or scaled from the number of CPUs,
rather than raised unconditionally. I do not have a threaded workload where
the smaller cache is the better trade, and would welcome one.
v4: Fix the claim that a linux-user process is a single vCPU. The cache is
per CPUState, and linux-user creates one per guest thread. Pointed out
by Richard Henderson.
Signed-off-by: Matt Turner <[email protected]>
---
accel/tcg/tb-jmp-cache.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h
index c3a505e394..268dacd7ba 100644
--- ./accel/tcg/tb-jmp-cache.h
+++ ./accel/tcg/tb-jmp-cache.h
@@ -12,7 +12,7 @@
#include "qemu/rcu.h"
#include "exec/cpu-common.h"
-#define TB_JMP_CACHE_BITS 12
+#define TB_JMP_CACHE_BITS 16
#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)
/*
--
2.54.0
