Every indirect branch that cannot use goto_tb ends in
tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
register destination and because goto_tb is restricted to same-page
targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead. The two preceding patches supply what it
needs: the destination PC is in a TCG temp, the flags, cflags and cs_base
the destination must match are constants at translation time, and
tb_jmp_cache_probe is a base pointer the main loop can poison. The fast
path is therefore a hash, four guarded loads and a goto_ptr. Only a miss
calls the helper, which still owns filling the cache.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. What is left is one_insn_per_tb and -d nochain, which the
monitor can toggle under a running vCPU without a flush; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the preceding patches, against an unmodified LTO build,
1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%. The
emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline with the old cflags until it exits for
some other reason. Poisoning the probe from tcg_update_all_cflags()
would close it.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that pass a PC are
built and boot-tested only.
v4: Split out of the patch that also changed the
tcg_gen_lookup_and_goto_ptr() API and introduced tb_jmp_cache_probe,
which are now the two preceding patches. Requested by Richard
Henderson.
v4: Emit the softmmu form of tb_jmp_cache_hash_func() under
CONFIG_SOFTMMU rather than the user-only form everywhere. v3 emitted
the user-only hash unconditionally, which was wrong for system mode
and was only not a correctness bug because a wrong index simply
misses. Caught by Richard Henderson. tcg-op.c is compiled once per
build rather than once per target, but CONFIG_SOFTMMU is set for it,
and TARGET_PAGE_BITS -- a load from target_page here -- is fixed long
before any translation happens.
v4: Compare the pc before testing tb for NULL. On a hash miss the pc is
the field most likely to differ, and an unused entry has a zero pc
that only pc 0 can match, so the tb test buys nothing ahead of it.
Suggested by Richard Henderson.
v4: Assert that offsetof(TranslationBlock, flags) is 8-byte aligned, since
folding the flags and cflags guards into one 64-bit load relies on it
and nothing else does. Requested by Richard Henderson.
v4: Zero-extend a 32-bit guest PC instead of falling back to the helper.
Suggested by Richard Henderson. The high half then folds to a compare
against zero.
v4: Describe cs_base in the probe as a second word of target-specific
flags rather than by name. Suggested by Richard Henderson.
Signed-off-by: Matt Turner <[email protected]>
---
include/tcg/tcg-op-common.h | 14 ++--
tcg/tcg-op.c | 126 ++++++++++++++++++++++++++++++++++++
2 files changed, 133 insertions(+), 7 deletions(-)
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 34102b3b7a..b8399229c9 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -81,13 +81,13 @@ void tcg_gen_goto_tb(unsigned idx);
*
* If the TB is not valid, jump to the epilogue.
*
- * The lookup is a call to helper_lookup_tb_ptr(). @pc and @tb describe the
- * destination for a faster lookup that a later patch adds, and neither is
- * used yet. When @pc is non-NULL it must hold exactly the value
- * get_tb_cpu_state() reports as the pc for the destination, and the
- * destination must match @tb's flags, cflags and cs_base. A target whose
- * pc is derived rather than being that key -- avr's word address, i386's
- * eip before segmentation -- must pass NULL.
+ * The lookup is normally a call to helper_lookup_tb_ptr(). If @pc is
+ * non-NULL the jump cache is probed inline instead, and only a miss reaches
+ * the helper. @pc must then hold exactly the value get_tb_cpu_state()
+ * reports as the pc for the destination; a target whose pc is derived
+ * (avr's word address, i386's eip before segmentation) must pass NULL. The
+ * destination is required to match @tb's flags, cflags and cs_base, which
+ * is what makes them constants in the probe.
*
* This operation is optional. If the TCG backend does not implement goto_ptr,
* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 2fda6e5c07..cf7b6882d8 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-hash.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2715,6 +2717,112 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
+static void gen_jmp_cache_hash(TCGv_i64 h, TCGv_i64 pc)
+{
+#ifdef CONFIG_SOFTMMU
+ /*
+ * tb_jmp_cache_hash_func(), softmmu form. TARGET_PAGE_BITS is a load
+ * from target_page in this translation unit, but it is decided long
+ * before any translation happens, so it is a constant here.
+ */
+ int shift = TARGET_PAGE_BITS - TB_JMP_PAGE_BITS;
+ TCGv_i64 tmp = tcg_temp_ebb_new_i64();
+
+ tcg_gen_shri_i64(tmp, pc, shift);
+ tcg_gen_xor_i64(tmp, tmp, pc);
+ tcg_gen_shri_i64(h, tmp, shift);
+ tcg_gen_andi_i64(h, h, TB_JMP_PAGE_MASK);
+ tcg_gen_andi_i64(tmp, tmp, TB_JMP_ADDR_MASK);
+ tcg_gen_or_i64(h, h, tmp);
+ tcg_temp_free_i64(tmp);
+#else
+ /* tb_jmp_cache_hash_func(), user-only form. */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+#endif
+}
+
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
+ /* One 64-bit load has to cover both, so they must be adjacent... */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+ /* ...and aligned, which nothing else currently relies on. */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
+ gen_jmp_cache_hash(h, pc);
+ tcg_gen_shli_i64(h, h, 4);
+
+ /*
+ * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
+ * loop can poison it, which is how conditions the probe cannot test for
+ * itself force every dispatch back into the helper. See
+ * tcg_cpu_sync_jmp_cache().
+ */
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ /*
+ * The pc first: on a hash miss it is the field most likely to differ,
+ * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
+ */
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+#if HOST_BIG_ENDIAN
+ fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
+#else
+ fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
+#endif
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * cs_base is a second word of target-specific flags despite the name,
+ * and the pc alone does not imply it on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ ptr = tcg_temp_ebb_new_ptr();
+ gen_helper_lookup_tb_ptr(ptr, tcg_env);
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+}
+
void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
{
TCGv_ptr ptr;
@@ -2726,6 +2834,24 @@ void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const
TranslationBlock *tb)
plugin_gen_disable_mem_helpers();
+ if (pc) {
+ TCGv_i64 pc64;
+
+ /*
+ * The jump cache is keyed on a vaddr, so a 32-bit guest PC is
+ * compared as its zero-extension. The high half then folds to a
+ * constant compare against zero.
+ */
+ if (pc->type == TCG_TYPE_I32) {
+ pc64 = tcg_temp_ebb_new_i64();
+ tcg_gen_extu_i32_i64(pc64, temp_tcgv_i32(pc));
+ } else {
+ pc64 = temp_tcgv_i64(pc);
+ }
+ gen_jmp_cache_probe(pc64, tb);
+ return;
+ }
+
ptr = tcg_temp_ebb_new_ptr();
gen_helper_lookup_tb_ptr(ptr, tcg_env);
tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
--
2.54.0
