Every indirect branch that cannot use goto_tb ends in a dispatch that calls
helper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper
calls in a single translation unit: 24.6% of all TB exits take this path,
because jsr/ret/jmp have a register destination and because goto_tb is
restricted to same-page targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead, for the callers that have migrated to
tcg_gen_goto_jc_*(). Those supply what it needs: the destination PC is in a
TCG temp, and the flags, cflags and cs_base the destination must match are
constants at translation time. The fast path is therefore a hash, four
guarded loads and a goto_ptr. Only a miss calls the helper, which still owns
filling the cache.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. Breakpoints are handled by CF_NO_GOTO_JC, added by the previous
patch: while one is set, blocks are translated with a cflags that both keeps
them off the inline path and keeps them unreachable from blocks already on
it. What is left is one_insn_per_tb and -d nochain; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the preceding patches, against an unmodified LTO build,
1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%. The
emulated compiler produces byte-identical output throughout.
A follow-up worth having: the probe is emitted entirely out of generic TCG
ops, and several backends can do much better than the result. x86_64 and
s390x have memory-operand comparisons; aarch64 can form env + off + h * 16
with a shift-add, load (tb, pc) and (cs_base, flags) with two ldp, and halve
the branches with ccmp. That wants a backend expansion of a dedicated
opcode, which is a separate series.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline on the old cflags until it exits for some
other reason. This is the same window goto_tb chaining already has, since
a chained direct jump consults nothing either, but it is worth saying out
loud.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that use goto_jc are
built and boot-tested only.
v4: Split out of the patch that also changed the
tcg_gen_lookup_and_goto_ptr() API and introduced tb_jmp_cache_probe,
which are now the two preceding patches. Requested by Richard
Henderson.
v4: Emit the softmmu form of tb_jmp_cache_hash_func() under
CONFIG_SOFTMMU rather than the user-only form everywhere. v3 emitted
the user-only hash unconditionally, which was wrong for system mode
and was only not a correctness bug because a wrong index simply
misses. Caught by Richard Henderson. tcg-op.c is compiled once per
build rather than once per target, but CONFIG_SOFTMMU is set for it,
and TARGET_PAGE_BITS -- a load from target_page here -- is fixed long
before any translation happens.
v4: Compare the pc before testing tb for NULL. On a hash miss the pc is
the field most likely to differ, and an unused entry has a zero pc
that only pc 0 can match, so the tb test buys nothing ahead of it.
Suggested by Richard Henderson.
v4: Assert that offsetof(TranslationBlock, flags) is 8-byte aligned, since
folding the flags and cflags guards into one 64-bit load relies on it
and nothing else does. Requested by Richard Henderson.
v4: Zero-extend a 32-bit guest PC instead of falling back to the helper.
Suggested by Richard Henderson. The high half then folds to a compare
against zero.
v4: Describe cs_base in the probe as a second word of target-specific
flags rather than by name. Suggested by Richard Henderson.
v5: Build the folded flags/cflags constant with deposit64() rather than
under #if HOST_BIG_ENDIAN, so both arms compile on every host.
Requested by Richard Henderson.
v5: Read cpu->tb_jmp_cache directly, and honor CF_NO_GOTO_JC rather than a
poisoned base pointer, which is no longer how breakpoints are handled.
A separate base pointer comes back later in the series for pending
exits.
v5: Note the backend expansion this wants as a follow-up. Suggested by
Richard Henderson, whose list it is.
Signed-off-by: Matt Turner <[email protected]>
---
include/tcg/tcg-op-common.h | 3 +-
tcg/tcg-op.c | 110 +++++++++++++++++++++++++++++++++++-
2 files changed, 111 insertions(+), 2 deletions(-)
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 4f334faaaa..f41f3ee58f 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -91,7 +91,8 @@ void tcg_gen_lookup_and_goto_ptr(void);
* @pc: temp holding the destination guest PC
*
* As tcg_gen_lookup_and_goto_ptr(), but the caller states where the
- * dispatch is going, which allows the lookup to be done inline.
+ * dispatch is going, so the TB jump cache is probed inline and only a miss
+ * reaches helper_lookup_tb_ptr().
*
* The contract is that when this runs, the CPU state must already be
* exactly the destination's: @pc must hold what get_tb_cpu_state() would
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index a2f35359fe..b10b2d66d5 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-hash.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2735,6 +2737,102 @@ void tcg_gen_lookup_and_goto_ptr(void)
gen_lookup_tb_ptr_and_goto();
}
+static void gen_jmp_cache_hash(TCGv_i64 h, TCGv_i64 pc)
+{
+#ifdef CONFIG_SOFTMMU
+ /*
+ * tb_jmp_cache_hash_func(), softmmu form. TARGET_PAGE_BITS is a load
+ * from target_page in this translation unit, but it is decided long
+ * before any translation happens, so it is a constant here.
+ */
+ int shift = TARGET_PAGE_BITS - TB_JMP_PAGE_BITS;
+ TCGv_i64 tmp = tcg_temp_ebb_new_i64();
+
+ tcg_gen_shri_i64(tmp, pc, shift);
+ tcg_gen_xor_i64(tmp, tmp, pc);
+ tcg_gen_shri_i64(h, tmp, shift);
+ tcg_gen_andi_i64(h, h, TB_JMP_PAGE_MASK);
+ tcg_gen_andi_i64(tmp, tmp, TB_JMP_ADDR_MASK);
+ tcg_gen_or_i64(h, h, tmp);
+ tcg_temp_free_i64(tmp);
+#else
+ /* tb_jmp_cache_hash_func(), user-only form. */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+#endif
+}
+
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
+ /* One 64-bit load has to cover both, so they must be adjacent... */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+ /* ...and aligned, which nothing else currently relies on. */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
+ gen_jmp_cache_hash(h, pc);
+ tcg_gen_shli_i64(h, h, 4);
+
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ /*
+ * The pc first: on a hash miss it is the field most likely to differ,
+ * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
+ */
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+ fpair = (HOST_BIG_ENDIAN
+ ? deposit64(tb->cflags, 32, 32, tb->flags)
+ : deposit64(tb->flags, 32, 32, tb->cflags));
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * cs_base is a second word of target-specific flags despite the name,
+ * and the pc alone does not imply it on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ gen_lookup_tb_ptr_and_goto();
+}
+
/*
* The common half of tcg_gen_goto_jc_i32() and tcg_gen_goto_jc_i64(). @pc
* is widened to i64 because the jump cache is keyed on a vaddr; for a
@@ -2761,7 +2859,17 @@ static void gen_goto_jc(TCGv_i64 pc)
tcg_constant_i64(tb->cs_base));
#endif
- gen_lookup_tb_ptr_and_goto();
+ /*
+ * A breakpoint is the one thing the probe cannot check for itself, so
+ * while one is set the flag is set too and every dispatch takes the
+ * helper, which does check. See tcg_update_cflags().
+ */
+ if (tb->cflags & CF_NO_GOTO_JC) {
+ gen_lookup_tb_ptr_and_goto();
+ return;
+ }
+
+ gen_jmp_cache_probe(pc, tb);
}
void tcg_gen_goto_jc_i64(TCGv_i64 pc)
--
2.54.0
