Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Mon, 14 Jan 2019 at 01:08, Aleksandar Markovic wrote: > I do understand the definition of cluster_index in the sense > of this series. However, it looks to me that the term > "cluster" is generally overused in areas where we work. > This may lead to some confusion for future developers, and > let me suggest some other name, like "tcg_cluster_index" or > "tcg_group_id", or "translation_group_id". Note also that the cluster index is not purely a TCG concept -- it also (in master at the moment) affects the gdbstub interface. Different clusters appear as separate processes in gdb, whereas different CPUs in the same cluster are different threads in the same CPU. (And by default only the first cluster's CPUs will appear, unless you explicitly attach to the second cluster: this is a limitation of how gdb's UI handles multiple processes.) thanks -- PMM
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Mon, 14 Jan 2019 at 01:08, Aleksandar Markovic wrote: > > > > On Tuesday, January 8, 2019, Peter Maydell wrote: >> >> Include the cluster number in the hash we use to look >> up TBs. This is important because a TB that is valid >> for one cluster at a given physical address and set >> of CPU flags is not necessarily valid for another: >> the two clusters may have different views of physical >> memory, or may have different CPU features (eg FPU >> present or absent). > I do understand the definition of cluster_index in the sense of this series. > However, it looks to me that the term "cluster" is generally overused in > areas where we work. This may lead to some confusion for future developers, > and let me suggest some other name, like "tcg_cluster_index" or > "tcg_group_id", or "translation_group_id". Admitedly, they all sound ugly to > me too. But having the name that would clearly separate this id from too > generic "cluster_index" IMHO would save lots of time during potential related > future development. > > (Needled to say that, for example, we in MIPS, for multi-core sustems, group > cores in clusters, that actually do not have anything to do with clusters in > TCG sense...) Yeah, the term is a bit overloaded. Arm also has clusters that are used more in the NUMA sense. However, the term we have is what is in git master currently... thanks -- PMM
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Tuesday, January 8, 2019, Peter Maydell wrote:
> Include the cluster number in the hash we use to look
> up TBs. This is important because a TB that is valid
> for one cluster at a given physical address and set
> of CPU flags is not necessarily valid for another:
> the two clusters may have different views of physical
> memory, or may have different CPU features (eg FPU
> present or absent).
>
>
Hi, Peter.
I do understand the definition of cluster_index in the sense of this
series. However, it looks to me that the term "cluster" is generally
overused in areas where we work. This may lead to some confusion for future
developers, and let me suggest some other name, like "tcg_cluster_index" or
"tcg_group_id", or "translation_group_id". Admitedly, they all sound ugly
to me too. But having the name that would clearly separate this id from too
generic "cluster_index" IMHO would save lots of time during potential
related future development.
(Needled to say that, for example, we in MIPS, for multi-core sustems,
group cores in clusters, that actually do not have anything to do with
clusters in TCG sense...)
Sincerely,
Aleksandar
> We put the cluster number in the high 8 bits of the
> TB cflags. This gives us up to 256 clusters, which should
> be enough for anybody. If we ever need more, or need
> more bits in cflags for other purposes, we could make
> tb_hash_func() take more data (and expand qemu_xxhash7()
> to qemu_xxhash8()).
>
> Signed-off-by: Peter Maydell
> ---
> include/exec/exec-all.h | 4 +++-
> accel/tcg/cpu-exec.c | 4
> accel/tcg/translate-all.c | 3 +++
> 3 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 815e5b1e838..aa7b81aaf01 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -351,9 +351,11 @@ struct TranslationBlock {
> #define CF_USE_ICOUNT 0x0002
> #define CF_INVALID 0x0004 /* TB is stale. Set with @jmp_lock held
> */
> #define CF_PARALLEL0x0008 /* Generate code for a parallel context
> */
> +#define CF_CLUSTER_MASK 0xff00 /* Top 8 bits are cluster ID */
> +#define CF_CLUSTER_SHIFT 24
> /* cflags' mask for hashing/comparison */
> #define CF_HASH_MASK \
> -(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL)
> +(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL |
> CF_CLUSTER_MASK)
>
> /* Per-vCPU dynamic tracing state used to generate this TB */
> uint32_t trace_vcpu_dstate;
> diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
> index 870027d4359..e578a1a3aee 100644
> --- a/accel/tcg/cpu-exec.c
> +++ b/accel/tcg/cpu-exec.c
> @@ -336,6 +336,10 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu,
> target_ulong pc,
> return NULL;
> }
> desc.phys_page1 = phys_pc & TARGET_PAGE_MASK;
> +
> +cf_mask &= ~CF_CLUSTER_MASK;
> +cf_mask |= cpu->cluster_index << CF_CLUSTER_SHIFT;
> +
> h = tb_hash_func(phys_pc, pc, flags, cf_mask, *cpu->trace_dstate);
> return qht_lookup_custom(_ctx.htable, , h, tb_lookup_cmp);
> }
> diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
> index 639f0b27287..ba27f5acc8c 100644
> --- a/accel/tcg/translate-all.c
> +++ b/accel/tcg/translate-all.c
> @@ -1692,6 +1692,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
> cflags |= CF_NOCACHE | 1;
> }
>
> +cflags &= ~CF_CLUSTER_MASK;
> +cflags |= cpu->cluster_index << CF_CLUSTER_SHIFT;
> +
> buffer_overflow:
> tb = tb_alloc(pc);
> if (unlikely(!tb)) {
> --
> 2.19.2
>
>
>
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Friday, January 11, 2019, Peter Maydell wrote: > On Fri, 11 Jan 2019 at 12:49, Aleksandar Markovic > wrote: > > 1. What would be, in more detail, if possible in layman terms, > > the "bad case" that this series fixes? > > I describe this in the cover letter (which also has a link to > a tarball with a test case demonstrating it): > > TCG implicitly assumes that all CPUs are alike, because we have > > a single cache of generated TBs and we don't account for which > > CPU generated the code or is looking for the TB when adding or > > searching for generated TBs. This can go wrong in two situations: > > (1) two CPUs have different physical address spaces (eg CPU 1 > > has one lot of RAM/ROM, and CPU 2 has different RAM/ROM): the > > physical address alone is then not sufficient to distinguish > > what code to run > > (2) two CPUs have different features (eg FPU > > vs no FPU): since our TCG frontends bake assumptions into the > > generated code about the presence/absence of features, if a > > CPU with FPU picks up a TB for one generated without an FPU > > it will behave wrongly > > What happens is that CPU 1 picks up code that was generated > for CPU 2 and which is not correct for it, and thus does > not behave correctly. (In the test case, an instruction that > should UNDEF on the Cortex-R5F but not on the Cortex-A53 will > either UNDEF on the A53 or fail to UNDEF on the R5F, depending > on which CPU happened to get to the test code first.) > > Thanks, this example makes the intentions of the patch clearer to me. If you don't mind, I may take a closer look at MIPS' (and perhaps some other targets' a little) multi-core design details in few coming weeks, and see if we could improve feightfulness of our emulation, or maybe make it more flexible, or scalable. Thanks again, and happy holiday season to all!! Aleksandar > > 2. Let's suppose, hypothetically, and based on your example > > from one of commit messages from this series, that we want to > > support two multicore systems: > > A. Cluster 1: 1 core with FPU; cluster 2: 3 cores without FPU > > B. Cluster 1: 2 cores with FPU; cluster 2: 1 core without FPU > > Is there an apparatus that would allow the end user specify these > > and similar cpnfigurations through command line or acsimilar mean > > (so, without QEMU explicitely supporting such core organization, > > but supporting the single core in question, of course)? > > The QEMU definition of "cluster" requires that all the CPUs > in the cluster must share (a) the same features (eg FPU) > and (b) the same view of physical memory -- this is what > defines that they are in the same cluster and not different > ones. So you'd model this as four clusters (assuming that > A and B have different views of physical memory. Otherwise > you could put all the with-FPU cores in one cluster and > the without-FPU cores in a second.) > > Real hardware might choose to define what it calls a "cluster" > differently, but that doesn't matter. > > > 3. Is there a possibility to have two layer clustering sheme, > > instead of one layer? Cluster/subcluster/core instead of > > cluster/core? For MIPS, there is a need for such organization. > > It looks to me 8 bits for cluster id, and 3 bits for subcluster > > id would be sufficient. > > My view is that there is no need for the internal "cluster ID" > to match what the hardware happens to do with SMP CPU IDs > and NUMA architecture. What do you think we miss by this? > (Handling of NUMA architecture is a distinct bit of QEMU code, > unrelated to this.) > > thanks > -- PMM >
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Fri, 11 Jan 2019 at 12:49, Aleksandar Markovic wrote: > 1. What would be, in more detail, if possible in layman terms, > the "bad case" that this series fixes? I describe this in the cover letter (which also has a link to a tarball with a test case demonstrating it): > TCG implicitly assumes that all CPUs are alike, because we have > a single cache of generated TBs and we don't account for which > CPU generated the code or is looking for the TB when adding or > searching for generated TBs. This can go wrong in two situations: > (1) two CPUs have different physical address spaces (eg CPU 1 > has one lot of RAM/ROM, and CPU 2 has different RAM/ROM): the > physical address alone is then not sufficient to distinguish > what code to run > (2) two CPUs have different features (eg FPU > vs no FPU): since our TCG frontends bake assumptions into the > generated code about the presence/absence of features, if a > CPU with FPU picks up a TB for one generated without an FPU > it will behave wrongly What happens is that CPU 1 picks up code that was generated for CPU 2 and which is not correct for it, and thus does not behave correctly. (In the test case, an instruction that should UNDEF on the Cortex-R5F but not on the Cortex-A53 will either UNDEF on the A53 or fail to UNDEF on the R5F, depending on which CPU happened to get to the test code first.) > 2. Let's suppose, hypothetically, and based on your example > from one of commit messages from this series, that we want to > support two multicore systems: > A. Cluster 1: 1 core with FPU; cluster 2: 3 cores without FPU > B. Cluster 1: 2 cores with FPU; cluster 2: 1 core without FPU > Is there an apparatus that would allow the end user specify these > and similar cpnfigurations through command line or acsimilar mean > (so, without QEMU explicitely supporting such core organization, > but supporting the single core in question, of course)? The QEMU definition of "cluster" requires that all the CPUs in the cluster must share (a) the same features (eg FPU) and (b) the same view of physical memory -- this is what defines that they are in the same cluster and not different ones. So you'd model this as four clusters (assuming that A and B have different views of physical memory. Otherwise you could put all the with-FPU cores in one cluster and the without-FPU cores in a second.) Real hardware might choose to define what it calls a "cluster" differently, but that doesn't matter. > 3. Is there a possibility to have two layer clustering sheme, > instead of one layer? Cluster/subcluster/core instead of > cluster/core? For MIPS, there is a need for such organization. > It looks to me 8 bits for cluster id, and 3 bits for subcluster > id would be sufficient. My view is that there is no need for the internal "cluster ID" to match what the hardware happens to do with SMP CPU IDs and NUMA architecture. What do you think we miss by this? (Handling of NUMA architecture is a distinct bit of QEMU code, unrelated to this.) thanks -- PMM
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On Tue, 8 Jan 2019 at 16:30, Peter Maydell wrote:
>
> Include the cluster number in the hash we use to look
> up TBs. This is important because a TB that is valid
> for one cluster at a given physical address and set
> of CPU flags is not necessarily valid for another:
> the two clusters may have different views of physical
> memory, or may have different CPU features (eg FPU
> present or absent).
>
> We put the cluster number in the high 8 bits of the
> TB cflags. This gives us up to 256 clusters, which should
> be enough for anybody. If we ever need more, or need
> more bits in cflags for other purposes, we could make
> tb_hash_func() take more data (and expand qemu_xxhash7()
> to qemu_xxhash8()).
>
> Signed-off-by: Peter Maydell
> ---
> include/exec/exec-all.h | 4 +++-
> accel/tcg/cpu-exec.c | 4
> accel/tcg/translate-all.c | 3 +++
> 3 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 815e5b1e838..aa7b81aaf01 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -351,9 +351,11 @@ struct TranslationBlock {
> #define CF_USE_ICOUNT 0x0002
> #define CF_INVALID 0x0004 /* TB is stale. Set with @jmp_lock held */
> #define CF_PARALLEL0x0008 /* Generate code for a parallel context */
> +#define CF_CLUSTER_MASK 0xff00 /* Top 8 bits are cluster ID */
> +#define CF_CLUSTER_SHIFT 24
> /* cflags' mask for hashing/comparison */
> #define CF_HASH_MASK \
> -(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL)
> +(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL |
> CF_CLUSTER_MASK)
>
> /* Per-vCPU dynamic tracing state used to generate this TB */
> uint32_t trace_vcpu_dstate;
> diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
> index 870027d4359..e578a1a3aee 100644
> --- a/accel/tcg/cpu-exec.c
> +++ b/accel/tcg/cpu-exec.c
> @@ -336,6 +336,10 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu,
> target_ulong pc,
> return NULL;
> }
> desc.phys_page1 = phys_pc & TARGET_PAGE_MASK;
> +
> +cf_mask &= ~CF_CLUSTER_MASK;
> +cf_mask |= cpu->cluster_index << CF_CLUSTER_SHIFT;
> +
This hunk turns out not to be quite right -- it needs to move
to the top of the function, before the assignment
"desc.flags = flags;". Otherwise tb_lookup_cmp() will
spuriously fail, and execution becomes somewhat slower because
we have to keep retranslating TBs rather than reusing them.
(Surprisingly this is only noticeable in an ARM TFM image
I happen to have, not in Linux kernel boot...)
thanks
-- PMM
Re: [Qemu-devel] [PATCH 3/4] accel/tcg: Add cluster number to TCG TB hash
On 1/8/19 5:30 PM, Peter Maydell wrote:
> Include the cluster number in the hash we use to look
> up TBs. This is important because a TB that is valid
> for one cluster at a given physical address and set
> of CPU flags is not necessarily valid for another:
> the two clusters may have different views of physical
> memory, or may have different CPU features (eg FPU
> present or absent).
>
> We put the cluster number in the high 8 bits of the
> TB cflags. This gives us up to 256 clusters, which should
> be enough for anybody. If we ever need more, or need
> more bits in cflags for other purposes, we could make
> tb_hash_func() take more data (and expand qemu_xxhash7()
> to qemu_xxhash8()).
>
> Signed-off-by: Peter Maydell
Reviewed-by: Luc Michel
> ---
> include/exec/exec-all.h | 4 +++-
> accel/tcg/cpu-exec.c | 4
> accel/tcg/translate-all.c | 3 +++
> 3 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 815e5b1e838..aa7b81aaf01 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -351,9 +351,11 @@ struct TranslationBlock {
> #define CF_USE_ICOUNT 0x0002
> #define CF_INVALID 0x0004 /* TB is stale. Set with @jmp_lock held */
> #define CF_PARALLEL0x0008 /* Generate code for a parallel context */
> +#define CF_CLUSTER_MASK 0xff00 /* Top 8 bits are cluster ID */
> +#define CF_CLUSTER_SHIFT 24
> /* cflags' mask for hashing/comparison */
> #define CF_HASH_MASK \
> -(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL)
> +(CF_COUNT_MASK | CF_LAST_IO | CF_USE_ICOUNT | CF_PARALLEL |
> CF_CLUSTER_MASK)
>
> /* Per-vCPU dynamic tracing state used to generate this TB */
> uint32_t trace_vcpu_dstate;
> diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
> index 870027d4359..e578a1a3aee 100644
> --- a/accel/tcg/cpu-exec.c
> +++ b/accel/tcg/cpu-exec.c
> @@ -336,6 +336,10 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu,
> target_ulong pc,
> return NULL;
> }
> desc.phys_page1 = phys_pc & TARGET_PAGE_MASK;
> +
> +cf_mask &= ~CF_CLUSTER_MASK;
> +cf_mask |= cpu->cluster_index << CF_CLUSTER_SHIFT;
> +
> h = tb_hash_func(phys_pc, pc, flags, cf_mask, *cpu->trace_dstate);
> return qht_lookup_custom(_ctx.htable, , h, tb_lookup_cmp);
> }
> diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
> index 639f0b27287..ba27f5acc8c 100644
> --- a/accel/tcg/translate-all.c
> +++ b/accel/tcg/translate-all.c
> @@ -1692,6 +1692,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
> cflags |= CF_NOCACHE | 1;
> }
>
> +cflags &= ~CF_CLUSTER_MASK;
> +cflags |= cpu->cluster_index << CF_CLUSTER_SHIFT;
> +
> buffer_overflow:
> tb = tb_alloc(pc);
> if (unlikely(!tb)) {
>
