On Wed, Mar 10, 2021 at 05:57:45PM +0000, Quentin Perret wrote:
> As the host stage 2 will be identity mapped, all the .hyp memory regions
> and/or memory pages donated to protected guestis will have to marked
> invalid in the host stage 2 page-table. At the same time, the hypervisor
> will need a way to track the ownership of each physical page to ensure
> memory sharing or donation between entities (host, guests, hypervisor) is
> legal.
>
> In order to enable this tracking at EL2, let's use the host stage 2
> page-table itself. The idea is to use the top bits of invalid mappings
> to store the unique identifier of the page owner. The page-table owner
> (the host) gets identifier 0 such that, at boot time, it owns the entire
> IPA space as the pgd starts zeroed.
>
> Provide kvm_pgtable_stage2_set_owner() which allows to modify the
> ownership of pages in the host stage 2. It re-uses most of the map()
> logic, but ends up creating invalid mappings instead. This impacts
> how we do refcount as we now need to count invalid mappings when they
> are used for ownership tracking.
>
> Signed-off-by: Quentin Perret <qper...@google.com>
> ---
> arch/arm64/include/asm/kvm_pgtable.h | 21 +++++++
> arch/arm64/kvm/hyp/pgtable.c | 92 ++++++++++++++++++++++++----
> 2 files changed, 101 insertions(+), 12 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h
> b/arch/arm64/include/asm/kvm_pgtable.h
> index 4ae19247837b..b09af4612656 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -238,6 +238,27 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64
> addr, u64 size,
> u64 phys, enum kvm_pgtable_prot prot,
> void *mc);
>
> +/**
> + * kvm_pgtable_stage2_set_owner() - Annotate invalid mappings with metadata
> + * encoding the ownership of a page in the
> + * IPA space.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address at which to place the annotation.
This confused me a bit, as the annotation is stored in the page-table, not
at the memory identified by @addr. How about:
"Base intermediate physical address to annotate"
> + * @size: Size of the IPA range to annotate.
"Size of the annotated range"
> + * @mc: Cache of pre-allocated and zeroed memory from which to
> allocate
> + * page-table pages.
> + * @owner_id: Unique identifier for the owner of the page.
> + *
> + * The page-table owner has identifier 0.
Perhaps, "By default, all page-tables are owned by identifier 0"
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + void *mc, u32 owner_id);
Is there a need for the owner_id to be 32-bit rather than e.g. 16-bit? Just
strikes me that it might be difficult to recover these bits in future if we
give them out freely now.
> +
> /**
> * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2
> page-table.
> * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index f37b4179b880..e4670b639726 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -48,6 +48,8 @@
> KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \
> KVM_PTE_LEAF_ATTR_HI_S2_XN)
>
> +#define KVM_INVALID_PTE_OWNER_MASK GENMASK(63, 32)
Ah, so that '02' earlier was a typo for '32'.
> +
> struct kvm_pgtable_walk_data {
> struct kvm_pgtable *pgt;
> struct kvm_pgtable_walker *walker;
> @@ -186,6 +188,11 @@ static kvm_pte_t kvm_init_valid_leaf_pte(u64 pa,
> kvm_pte_t attr, u32 level)
> return pte;
> }
>
> +static kvm_pte_t kvm_init_invalid_leaf_owner(u32 owner_id)
> +{
> + return FIELD_PREP(KVM_INVALID_PTE_OWNER_MASK, owner_id);
> +}
> +
> static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64
> addr,
> u32 level, kvm_pte_t *ptep,
> enum kvm_pgtable_walk_flags flag)
> @@ -440,6 +447,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
> struct stage2_map_data {
> u64 phys;
> kvm_pte_t attr;
> + u32 owner_id;
>
> kvm_pte_t *anchor;
> kvm_pte_t *childp;
> @@ -506,6 +514,24 @@ static int stage2_map_set_prot_attr(enum
> kvm_pgtable_prot prot,
> return 0;
> }
>
> +static bool stage2_is_permission_change(kvm_pte_t old, kvm_pte_t new)
> +{
> + if (!kvm_pte_valid(old) || !kvm_pte_valid(new))
> + return false;
> +
> + return !((old ^ new) & (~KVM_PTE_LEAF_ATTR_S2_PERMS));
> +}
This new name throws me, because it will return true if old == new.
> +static bool stage2_pte_is_counted(kvm_pte_t pte)
> +{
> + /*
> + * The refcount tracks valid entries as well as invalid entries if they
> + * encode ownership of a page to another entity than the page-table
> + * owner, whose id is 0.
> + */
> + return !!pte;
> +}
> +
> static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level,
> kvm_pte_t *ptep,
> struct stage2_map_data *data)
> @@ -517,28 +543,36 @@ static int stage2_map_walker_try_leaf(u64 addr, u64
> end, u32 level,
> if (!kvm_block_mapping_supported(addr, end, phys, level))
> return -E2BIG;
>
> - new = kvm_init_valid_leaf_pte(phys, data->attr, level);
> - if (kvm_pte_valid(old)) {
> + if (kvm_pte_valid(data->attr))
This feels like a bit of a hack to me: the 'attr' field in stage2_map_data
is intended to correspond directly to the lower/upper attributes of the
descriptor as per the architecture, so tagging the valid bit in there is
pretty grotty. However, I can see the significant advantage in being able
to re-use the stage2_map_walker functionality, so about instead of nobbling
attr, you set phys to something invalid instead, e.g.:
#define KVM_PHYS_SET_OWNER (-1ULL)
> + new = kvm_init_valid_leaf_pte(phys, data->attr, level);
> + else
> + new = kvm_init_invalid_leaf_owner(data->owner_id);
> +
> + if (stage2_pte_is_counted(old)) {
> /*
> * Skip updating the PTE if we are trying to recreate the exact
> * same mapping or only change the access permissions. Instead,
> * the vCPU will exit one more time from guest if still needed
> * and then go through the path of relaxing permissions.
> */
> - if (!((old ^ new) & (~KVM_PTE_LEAF_ATTR_S2_PERMS)))
> + if (stage2_is_permission_change(old, new))
> return -EAGAIN;
>
> /*
> - * There's an existing different valid leaf entry, so perform
> - * break-before-make.
> + * Clear the existing PTE, and perform break-before-make with
> + * TLB maintenance if it was valid.
> */
> kvm_clear_pte(ptep);
> - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
> + if (kvm_pte_valid(old)) {
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr,
> + level);
> + }
Why do you clear the pte unconditionally here? I think you can move that
into the 'if' as well.
> mm_ops->put_page(ptep);
> }
>
> smp_store_release(ptep, new);
> - mm_ops->get_page(ptep);
> + if (stage2_pte_is_counted(new))
> + mm_ops->get_page(ptep);
> data->phys += granule;
> return 0;
> }
> @@ -574,7 +608,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32
> level, kvm_pte_t *ptep,
> int ret;
>
> if (data->anchor) {
> - if (kvm_pte_valid(pte))
> + if (stage2_pte_is_counted(pte))
> mm_ops->put_page(ptep);
>
> return 0;
> @@ -599,9 +633,10 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32
> level, kvm_pte_t *ptep,
> * a table. Accesses beyond 'end' that fall within the new table
> * will be mapped lazily.
> */
> - if (kvm_pte_valid(pte)) {
> + if (stage2_pte_is_counted(pte)) {
> kvm_clear_pte(ptep);
> - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
> + if (kvm_pte_valid(pte))
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr,
> level);
Same here.
> mm_ops->put_page(ptep);
> }
>
> @@ -683,6 +718,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64
> addr, u64 size,
> .mmu = pgt->mmu,
> .memcache = mc,
> .mm_ops = pgt->mm_ops,
> + .owner_id = 0,
Not needed.
> };
> struct kvm_pgtable_walker walker = {
> .cb = stage2_map_walker,
> @@ -696,6 +732,33 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64
> addr, u64 size,
> if (ret)
> return ret;
>
> + /* Set the valid flag to distinguish with the set_owner() path. */
> + map_data.attr |= KVM_PTE_VALID;
(see earlier comments)
> + ret = kvm_pgtable_walk(pgt, addr, size, &walker);
> + dsb(ishst);
> + return ret;
> +}
> +
> +int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + void *mc, u32 owner_id)
> +{
> + int ret;
> + struct stage2_map_data map_data = {
> + .mmu = pgt->mmu,
> + .memcache = mc,
> + .mm_ops = pgt->mm_ops,
> + .owner_id = owner_id,
> + .attr = 0,
Not needed.
> + };
> + struct kvm_pgtable_walker walker = {
> + .cb = stage2_map_walker,
> + .flags = KVM_PGTABLE_WALK_TABLE_PRE |
> + KVM_PGTABLE_WALK_LEAF |
> + KVM_PGTABLE_WALK_TABLE_POST,
Oh man, now I see why phys is zero -- so that the table callbacks will
put down the largest possible block. That needs a comment because it's
pretty horrible, and also means my idea of using -1 won't help you. Hmm.
Is there ever a reason to use kvm_pgtable_stage2_set_owner() to set an
owner of 0, or should you just use the map/unmap APIs for that? If so,
then maybe the key is simply if owner_id is non-zero, then an invalid
entry is installed?
Will
