Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
Hi Sean,
On Friday 19 Feb 2021 at 10:32:58 (-0800), Sean Christopherson wrote:
> On Wed, Feb 03, 2021, Will Deacon wrote:
> > On Fri, Jan 08, 2021 at 12:15:14PM +, Quentin Perret wrote:
>
> ...
>
> > > +static inline unsigned long hyp_s1_pgtable_size(void)
> > > +{
>
> ...
>
> > > + res += nr_pages << PAGE_SHIFT;
> > > + }
> > > +
> > > + /* Allow 1 GiB for private mappings */
> > > + nr_pages = (1 << 30) >> PAGE_SHIFT;
> >
> > SZ_1G >> PAGE_SHIFT
>
> Where does the 1gb magic number come from?
Admittedly it is arbitrary. It needs to be enough to cover all the
so-called 'private' mappings that EL2 needs, and which can vary a little
depending on the hardware.
> IIUC, this is calculating the number
> of pages needed for the hypervisor's Stage-1 page tables.
Correct. The thing worth noting is that the hypervisor VA space is
essentially split in half. One half is reserved to map portions of
memory with a fixed offset, and the other half is used for a whole bunch
of other things: we have a vmemmap, the 'private' mappings and the idmap
page.
> The amount of memory
> needed for those page tables should be easily calculated
As mentioned above, that is true for pretty much everything in the hyp
VA space except the private mappings as that depends on e.g. the CPU
uarch and such.
> and assuming huge pages can be used, should be far less the 1gb.
Ack, though this is no supported for the EL2 mappings yet. Historically
the amount of contiguous portions of memory mapped at EL2 has been
rather small, so there wasn't really a need, but we might want to
revisit this at some point.
> > > + nr_pages = __hyp_pgtable_max_pages(nr_pages);
> > > + res += nr_pages << PAGE_SHIFT;
> > > +
> > > + return res;
>
> ...
>
> > > +void __init kvm_hyp_reserve(void)
> > > +{
> > > + u64 nr_pages, prev;
> > > +
> > > + if (!is_hyp_mode_available() || is_kernel_in_hyp_mode())
> > > + return;
> > > +
> > > + if (kvm_get_mode() != KVM_MODE_PROTECTED)
> > > + return;
> > > +
> > > + if (kvm_nvhe_sym(hyp_memblock_nr) < 0) {
> > > + kvm_err("Failed to register hyp memblocks\n");
> > > + return;
> > > + }
> > > +
> > > + sort_memblock_regions();
> > > +
> > > + /*
> > > + * We don't know the number of possible CPUs yet, so allocate for the
> > > + * worst case.
> > > + */
> > > + hyp_mem_size += NR_CPUS << PAGE_SHIFT;
>
> Is this for per-cpu stack?
Correct.
> If so, what guarantees a single page is sufficient? Mostly a curiosity
> question,
> since it looks like this is an existing assumption by init_hyp_mode().
> Shouldn't
> the required stack size be defined in bytes and converted to pages, or is
> there a
> guarantee that 64kb pages will be used?
Nope, we have no such guarantees, but 4K has been more than enough for
EL2 so far. The hyp code doesn't use recursion much (I think the only
occurence we have is Will's pgtable code, and that is architecturally
limited to 4 levels of recursion for obvious reasons) and doesn't have
use stack allocations.
It's on my todo list to remap the stack pages in the 'private' range, to
surround them with guard pages so we can at least run-time check this
assumption, so stay tuned :)
> > There was a recent patch bumping NR_CPUs to 512, so this would be 32MB
> > with 64k pages. Is it possible to return memory to the host later on once
> > we have a better handle on the number of CPUs in the system?
>
> Does kvm_hyp_reserve() really need to be called during bootmem_init()? What
> prevents doing the reservation during init_hyp_mode()? If the problem is that
> pKVM needs a single contiguous chunk of memory, then it might be worth solving
> _that_ problem, e.g. letting the host donate memory in N-byte chunks instead
> of
> requiring a single huge blob of memory.
Right, I've been thinking about this over the weekend and that might
actually be fairly straightforward for stack pages. I'll try to move this
allocation to init_hyp_mode() where it belongs (or better, re-use the
existing one) in the nest version.
Thanks,
Quentin
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Wed, Feb 03, 2021, Will Deacon wrote:
> On Fri, Jan 08, 2021 at 12:15:14PM +, Quentin Perret wrote:
...
> > +static inline unsigned long hyp_s1_pgtable_size(void)
> > +{
...
> > + res += nr_pages << PAGE_SHIFT;
> > + }
> > +
> > + /* Allow 1 GiB for private mappings */
> > + nr_pages = (1 << 30) >> PAGE_SHIFT;
>
> SZ_1G >> PAGE_SHIFT
Where does the 1gb magic number come from? IIUC, this is calculating the number
of pages needed for the hypervisor's Stage-1 page tables. The amount of memory
needed for those page tables should be easily calculated, and assuming huge
pages can be used, should be far less the 1gb.
> > + nr_pages = __hyp_pgtable_max_pages(nr_pages);
> > + res += nr_pages << PAGE_SHIFT;
> > +
> > + return res;
...
> > +void __init kvm_hyp_reserve(void)
> > +{
> > + u64 nr_pages, prev;
> > +
> > + if (!is_hyp_mode_available() || is_kernel_in_hyp_mode())
> > + return;
> > +
> > + if (kvm_get_mode() != KVM_MODE_PROTECTED)
> > + return;
> > +
> > + if (kvm_nvhe_sym(hyp_memblock_nr) < 0) {
> > + kvm_err("Failed to register hyp memblocks\n");
> > + return;
> > + }
> > +
> > + sort_memblock_regions();
> > +
> > + /*
> > +* We don't know the number of possible CPUs yet, so allocate for the
> > +* worst case.
> > +*/
> > + hyp_mem_size += NR_CPUS << PAGE_SHIFT;
Is this for per-cpu stack?
If so, what guarantees a single page is sufficient? Mostly a curiosity
question,
since it looks like this is an existing assumption by init_hyp_mode().
Shouldn't
the required stack size be defined in bytes and converted to pages, or is there
a
guarantee that 64kb pages will be used?
> There was a recent patch bumping NR_CPUs to 512, so this would be 32MB
> with 64k pages. Is it possible to return memory to the host later on once
> we have a better handle on the number of CPUs in the system?
Does kvm_hyp_reserve() really need to be called during bootmem_init()? What
prevents doing the reservation during init_hyp_mode()? If the problem is that
pKVM needs a single contiguous chunk of memory, then it might be worth solving
_that_ problem, e.g. letting the host donate memory in N-byte chunks instead of
requiring a single huge blob of memory.
> > + hyp_mem_size += hyp_s1_pgtable_size();
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Thursday 04 Feb 2021 at 10:47:08 (+), Quentin Perret wrote:
> On Wednesday 03 Feb 2021 at 14:37:10 (+), Will Deacon wrote:
> > > +static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
> > > +{
> > > + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1);
> > > + DECLARE_REG(unsigned long, size, host_ctxt, 2);
> > > + DECLARE_REG(unsigned long, nr_cpus, host_ctxt, 3);
> > > + DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 4);
> > > +
> > > + cpu_reg(host_ctxt, 1) = __pkvm_init(phys, size, nr_cpus, per_cpu_base);
> >
> > __pkvm_init() doesn't return, so I think this assignment back into host_ctxt
> > is confusing.
>
> Very good point, I'll get rid of this.
Actually not, I think I'll leave it like that. __pkvm_init can return an
error, which is why I did this in the first place And it is useful for
debugging to have it propagated back to the host.
Thanks,
Quentin
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Tue, Feb 09, 2021 at 10:00:29AM +, Quentin Perret wrote:
> On Thursday 04 Feb 2021 at 10:47:08 (+), Quentin Perret wrote:
> > On Wednesday 03 Feb 2021 at 14:37:10 (+), Will Deacon wrote:
> > > > +static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
> > > > +{
> > > > + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1);
> > > > + DECLARE_REG(unsigned long, size, host_ctxt, 2);
> > > > + DECLARE_REG(unsigned long, nr_cpus, host_ctxt, 3);
> > > > + DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 4);
> > > > +
> > > > + cpu_reg(host_ctxt, 1) = __pkvm_init(phys, size, nr_cpus,
> > > > per_cpu_base);
> > >
> > > __pkvm_init() doesn't return, so I think this assignment back into
> > > host_ctxt
> > > is confusing.
> >
> > Very good point, I'll get rid of this.
>
> Actually not, I think I'll leave it like that. __pkvm_init can return an
> error, which is why I did this in the first place And it is useful for
> debugging to have it propagated back to the host.
Good point, but please add a comment!
Will
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Thu, Feb 04, 2021 at 10:47:08AM +, Quentin Perret wrote:
> On Wednesday 03 Feb 2021 at 14:37:10 (+), Will Deacon wrote:
> > On Fri, Jan 08, 2021 at 12:15:14PM +, Quentin Perret wrote:
> > > +static inline unsigned long __hyp_pgtable_max_pages(unsigned long
> > > nr_pages)
> > > +{
> > > + unsigned long total = 0, i;
> > > +
> > > + /* Provision the worst case scenario with 4 levels of page-table */
> > > + for (i = 0; i < 4; i++) {
> >
> > Looks like you want KVM_PGTABLE_MAX_LEVELS, so maybe move that into a
> > header?
>
> Will do.
>
> >
> > > + nr_pages = DIV_ROUND_UP(nr_pages, PTRS_PER_PTE);
> > > + total += nr_pages;
> > > + }
> >
> > ... that said, I'm not sure this needs to iterate at all. What exactly are
> > you trying to compute?
>
> I'm trying to figure out how many pages I will need to construct a
> page-table covering nr_pages contiguous pages. The first iteration tells
> me how many level 0 pages I need to cover nr_pages, the second iteration
> how many level 1 pages I need to cover the level 0 pages, and so on...
Ah, you iterate from leaves back to the root. Got it, thanks.
> I might be doing this naively though. Got a better idea?
I thought I did, but I ended up with something based on a geometric series
and it looks terrible to code-up in C without, err, iterating like you do.
So yeah, ignore me :)
> > > +
> > > + return total;
> > > +}
> > > +
> > > +static inline unsigned long hyp_s1_pgtable_size(void)
> > > +{
> > > + struct hyp_memblock_region *reg;
> > > + unsigned long nr_pages, res = 0;
> > > + int i;
> > > +
> > > + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0)
> > > + return 0;
> >
> > It's a bit grotty having this be signed. Why do we need to encode the error
> > case differently from the 0 case?
>
> Here specifically we don't, but it is needed in early_init_dt_add_memory_hyp()
> to distinguish the overflow case from the first memblock being added.
Fair enough, but if you figure out a way for hyp_memblock_nr to be unsigned,
I think that would be preferable.
Will
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Wednesday 03 Feb 2021 at 14:37:10 (+), Will Deacon wrote:
> On Fri, Jan 08, 2021 at 12:15:14PM +, Quentin Perret wrote:
> > +static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages)
> > +{
> > + unsigned long total = 0, i;
> > +
> > + /* Provision the worst case scenario with 4 levels of page-table */
> > + for (i = 0; i < 4; i++) {
>
> Looks like you want KVM_PGTABLE_MAX_LEVELS, so maybe move that into a
> header?
Will do.
>
> > + nr_pages = DIV_ROUND_UP(nr_pages, PTRS_PER_PTE);
> > + total += nr_pages;
> > + }
>
> ... that said, I'm not sure this needs to iterate at all. What exactly are
> you trying to compute?
I'm trying to figure out how many pages I will need to construct a
page-table covering nr_pages contiguous pages. The first iteration tells
me how many level 0 pages I need to cover nr_pages, the second iteration
how many level 1 pages I need to cover the level 0 pages, and so on...
I might be doing this naively though. Got a better idea?
> > +
> > + return total;
> > +}
> > +
> > +static inline unsigned long hyp_s1_pgtable_size(void)
> > +{
> > + struct hyp_memblock_region *reg;
> > + unsigned long nr_pages, res = 0;
> > + int i;
> > +
> > + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0)
> > + return 0;
>
> It's a bit grotty having this be signed. Why do we need to encode the error
> case differently from the 0 case?
Here specifically we don't, but it is needed in early_init_dt_add_memory_hyp()
to distinguish the overflow case from the first memblock being added.
>
> > +
> > + for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) {
> > + reg = _nvhe_sym(hyp_memory)[i];
>
> You could declare reg in the loop body.
I found it prettier like that and assumed the compiler would optimize it
anyway, but ok.
> > + nr_pages = (reg->end - reg->start) >> PAGE_SHIFT;
> > + nr_pages = __hyp_pgtable_max_pages(nr_pages);
>
> Maybe it would make more sense for __hyp_pgtable_max_pages to take the
> size in bytes rather than pages, since most callers seem to have to do the
> conversion?
Yes, and it seems I can apply small cleanups in other places, so I'll
fix this throughout the patch.
> > + res += nr_pages << PAGE_SHIFT;
> > + }
> > +
> > + /* Allow 1 GiB for private mappings */
> > + nr_pages = (1 << 30) >> PAGE_SHIFT;
>
> SZ_1G >> PAGE_SHIFT
Much nicer, thanks. I was also considering adding a Kconfig option for
that, because 1GiB is totally arbitrary. Toughts?
> > + nr_pages = __hyp_pgtable_max_pages(nr_pages);
> > + res += nr_pages << PAGE_SHIFT;
> > +
> > + return res;
>
> Might make more sense to keep res in pages until here, then just shift when
> returning.
>
> > +}
> > +
> > +#endif /* __KVM_HYP_MM_H */
> > diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile
> > b/arch/arm64/kvm/hyp/nvhe/Makefile
> > index 72cfe53f106f..d7381a503182 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> > +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> > @@ -11,9 +11,9 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs))
> >
> > obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
> > host.o \
> > hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \
> > -cache.o cpufeature.o
> > +cache.o cpufeature.o setup.o mm.o
> > obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
> > -../fpsimd.o ../hyp-entry.o ../exception.o
> > +../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o
> > obj-y += $(lib-objs)
> >
> > ##
> > diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> > b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> > index 31b060a44045..ad943966c39f 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> > +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> > @@ -251,4 +251,35 @@ alternative_else_nop_endif
> >
> > SYM_CODE_END(__kvm_handle_stub_hvc)
> >
> > +SYM_FUNC_START(__pkvm_init_switch_pgd)
> > + /* Turn the MMU off */
> > + pre_disable_mmu_workaround
> > + mrs x2, sctlr_el2
> > + bic x3, x2, #SCTLR_ELx_M
> > + msr sctlr_el2, x3
> > + isb
> > +
> > + tlbialle2
> > +
> > + /* Install the new pgtables */
> > + ldr x3, [x0, #NVHE_INIT_PGD_PA]
> > + phys_to_ttbr x4, x3
> > +alternative_if ARM64_HAS_CNP
> > + orr x4, x4, #TTBR_CNP_BIT
> > +alternative_else_nop_endif
> > + msr ttbr0_el2, x4
> > +
> > + /* Set the new stack pointer */
> > + ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA]
> > + mov sp, x0
> > +
> > + /* And turn the MMU back on! */
> > + dsb nsh
> > + isb
> > + msr sctlr_el2, x2
> > + ic iallu
> > + isb
> > + ret x1
> > +SYM_FUNC_END(__pkvm_init_switch_pgd)
> > +
> > .popsection
> > diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> > b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> > index a906f9e2ff34..3075f117651c 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> > +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> > @@ -6,12
Re: [RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
On Fri, Jan 08, 2021 at 12:15:14PM +, Quentin Perret wrote:
> When memory protection is enabled, the Hyp code needs the ability to
> create and manage its own page-table. To do so, introduce a new set of
> hypercalls to initialize Hyp memory protection.
>
> During the init hcall, the hypervisor runs with the host-provided
> page-table and uses the trivial early page allocator to create its own
> set of page-tables, using a memory pool that was donated by the host.
> Specifically, the hypervisor creates its own mappings for __hyp_text,
> the Hyp memory pool, the __hyp_bss, the portion of hyp_vmemmap
> corresponding to the Hyp pool, among other things. It then jumps back in
> the idmap page, switches to use the newly-created pgd (instead of the
> temporary one provided by the host) and then installs the full-fledged
> buddy allocator which will then be the only one in used from then on.
>
> Note that for the sake of symplifying the review, this only introduces
> the code doing this operation, without actually being called by anyhing
> yet. This will be done in a subsequent patch, which will introduce the
> necessary host kernel changes.
>
> Credits to Will for __pkvm_init_switch_pgd.
>
> Co-authored-by: Will Deacon
> Signed-off-by: Will Deacon
> Signed-off-by: Quentin Perret
[...]
> diff --git a/arch/arm64/include/asm/kvm_hyp.h
> b/arch/arm64/include/asm/kvm_hyp.h
> index c0450828378b..a0e113734b20 100644
> --- a/arch/arm64/include/asm/kvm_hyp.h
> +++ b/arch/arm64/include/asm/kvm_hyp.h
> @@ -100,4 +100,12 @@ void __noreturn hyp_panic(void);
> void __noreturn __hyp_do_panic(bool restore_host, u64 spsr, u64 elr, u64
> par);
> #endif
>
> +#ifdef __KVM_NVHE_HYPERVISOR__
> +void __pkvm_init_switch_pgd(phys_addr_t phys, unsigned long size,
> + phys_addr_t pgd, void *sp, void *cont_fn);
> +int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus,
> + unsigned long *per_cpu_base);
> +void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt);
> +#endif
> +
> #endif /* __ARM64_KVM_HYP_H__ */
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index 43f3a1d6e92d..366d837f0d39 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -113,6 +113,25 @@ KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy);
> KVM_NVHE_ALIAS_HYP(__memset, __pi_memset);
> #endif
>
> +/* Hypevisor VA size */
typo: Hypervisor
> +KVM_NVHE_ALIAS(hyp_va_bits);
> +
> +/* Kernel memory sections */
> +KVM_NVHE_ALIAS(__start_rodata);
> +KVM_NVHE_ALIAS(__end_rodata);
> +KVM_NVHE_ALIAS(__bss_start);
> +KVM_NVHE_ALIAS(__bss_stop);
> +
> +/* Hyp memory sections */
> +KVM_NVHE_ALIAS(__hyp_idmap_text_start);
> +KVM_NVHE_ALIAS(__hyp_idmap_text_end);
> +KVM_NVHE_ALIAS(__hyp_text_start);
> +KVM_NVHE_ALIAS(__hyp_text_end);
> +KVM_NVHE_ALIAS(__hyp_data_ro_after_init_start);
> +KVM_NVHE_ALIAS(__hyp_data_ro_after_init_end);
> +KVM_NVHE_ALIAS(__hyp_bss_start);
> +KVM_NVHE_ALIAS(__hyp_bss_end);
> +
> #endif /* CONFIG_KVM */
>
> #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
> diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
> index 687598e41b21..b726332eec49 100644
> --- a/arch/arm64/kvm/hyp/Makefile
> +++ b/arch/arm64/kvm/hyp/Makefile
> @@ -10,4 +10,4 @@ subdir-ccflags-y := -I$(incdir)
> \
> -DDISABLE_BRANCH_PROFILING \
> $(DISABLE_STACKLEAK_PLUGIN)
>
> -obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o
> +obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o reserved_mem.o
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h
> b/arch/arm64/kvm/hyp/include/nvhe/memory.h
> index ed47674bc988..c8af6fe87bfb 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/memory.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h
> @@ -6,6 +6,12 @@
>
> #include
>
> +#define HYP_MEMBLOCK_REGIONS 128
> +struct hyp_memblock_region {
> + phys_addr_t start;
> + phys_addr_t end;
> +};
> +
> struct hyp_pool;
> struct hyp_page {
> unsigned int refcount;
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> new file mode 100644
> index ..f0cc09b127a5
> --- /dev/null
> +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> @@ -0,0 +1,79 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +#ifndef __KVM_HYP_MM_H
> +#define __KVM_HYP_MM_H
> +
> +#include
> +#include
> +#include
> +
> +#include
> +#include
> +
> +extern struct hyp_memblock_region kvm_nvhe_sym(hyp_memory)[];
> +extern int kvm_nvhe_sym(hyp_memblock_nr);
> +extern struct kvm_pgtable pkvm_pgtable;
> +extern hyp_spinlock_t pkvm_pgd_lock;
> +extern struct hyp_pool hpool;
> +extern u64 __io_map_base;
> +extern u32 hyp_va_bits;
> +
> +int hyp_create_idmap(void);
> +int hyp_map_vectors(void);
> +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back);
> +int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot);
> +int
[RFC PATCH v2 16/26] KVM: arm64: Prepare Hyp memory protection
When memory protection is enabled, the Hyp code needs the ability to
create and manage its own page-table. To do so, introduce a new set of
hypercalls to initialize Hyp memory protection.
During the init hcall, the hypervisor runs with the host-provided
page-table and uses the trivial early page allocator to create its own
set of page-tables, using a memory pool that was donated by the host.
Specifically, the hypervisor creates its own mappings for __hyp_text,
the Hyp memory pool, the __hyp_bss, the portion of hyp_vmemmap
corresponding to the Hyp pool, among other things. It then jumps back in
the idmap page, switches to use the newly-created pgd (instead of the
temporary one provided by the host) and then installs the full-fledged
buddy allocator which will then be the only one in used from then on.
Note that for the sake of symplifying the review, this only introduces
the code doing this operation, without actually being called by anyhing
yet. This will be done in a subsequent patch, which will introduce the
necessary host kernel changes.
Credits to Will for __pkvm_init_switch_pgd.
Co-authored-by: Will Deacon
Signed-off-by: Will Deacon
Signed-off-by: Quentin Perret
---
arch/arm64/include/asm/kvm_asm.h | 4 +
arch/arm64/include/asm/kvm_host.h| 8 +
arch/arm64/include/asm/kvm_hyp.h | 8 +
arch/arm64/kernel/image-vars.h | 19 +++
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/include/nvhe/memory.h | 6 +
arch/arm64/kvm/hyp/include/nvhe/mm.h | 79 +
arch/arm64/kvm/hyp/nvhe/Makefile | 4 +-
arch/arm64/kvm/hyp/nvhe/hyp-init.S | 31
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 42 +
arch/arm64/kvm/hyp/nvhe/mm.c | 174
arch/arm64/kvm/hyp/nvhe/setup.c | 196 +++
arch/arm64/kvm/hyp/reserved_mem.c| 102
arch/arm64/kvm/mmu.c | 2 +-
arch/arm64/mm/init.c | 3 +
15 files changed, 676 insertions(+), 4 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mm.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/mm.c
create mode 100644 arch/arm64/kvm/hyp/nvhe/setup.c
create mode 100644 arch/arm64/kvm/hyp/reserved_mem.c
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 7ccf770c53d9..4fc27ac08836 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -57,6 +57,10 @@
#define __KVM_HOST_SMCCC_FUNC___kvm_get_mdcr_el2 12
#define __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs 13
#define __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs 14
+#define __KVM_HOST_SMCCC_FUNC___pkvm_init 15
+#define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16
+#define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping17
+#define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector18
#ifndef __ASSEMBLY__
diff --git a/arch/arm64/include/asm/kvm_host.h
b/arch/arm64/include/asm/kvm_host.h
index 81212958ef55..9a2feb83eea0 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -777,4 +777,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
#define kvm_vcpu_has_pmu(vcpu) \
(test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features))
+#ifdef CONFIG_KVM
+extern phys_addr_t hyp_mem_base;
+extern phys_addr_t hyp_mem_size;
+void __init kvm_hyp_reserve(void);
+#else
+static inline void kvm_hyp_reserve(void) { }
+#endif
+
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index c0450828378b..a0e113734b20 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -100,4 +100,12 @@ void __noreturn hyp_panic(void);
void __noreturn __hyp_do_panic(bool restore_host, u64 spsr, u64 elr, u64 par);
#endif
+#ifdef __KVM_NVHE_HYPERVISOR__
+void __pkvm_init_switch_pgd(phys_addr_t phys, unsigned long size,
+ phys_addr_t pgd, void *sp, void *cont_fn);
+int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus,
+ unsigned long *per_cpu_base);
+void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt);
+#endif
+
#endif /* __ARM64_KVM_HYP_H__ */
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 43f3a1d6e92d..366d837f0d39 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -113,6 +113,25 @@ KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy);
KVM_NVHE_ALIAS_HYP(__memset, __pi_memset);
#endif
+/* Hypevisor VA size */
+KVM_NVHE_ALIAS(hyp_va_bits);
+
+/* Kernel memory sections */
+KVM_NVHE_ALIAS(__start_rodata);
+KVM_NVHE_ALIAS(__end_rodata);
+KVM_NVHE_ALIAS(__bss_start);
+KVM_NVHE_ALIAS(__bss_stop);
+
+/* Hyp memory sections */
+KVM_NVHE_ALIAS(__hyp_idmap_text_start);
