Re: [Intel-gfx] [PATCH v4 16/29] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
On Thu, Aug 31, 2023, Like Xu wrote:
> On 31/8/2023 4:50 am, Sean Christopherson wrote:
> > On Wed, Aug 30, 2023, Like Xu wrote:
> > > On 2023/7/29 09:35, Sean Christopherson wrote:
> > > > Disallow moving memslots if the VM has external page-track users, i.e.
> > > > if
> > > > KVMGT is being used to expose a virtual GPU to the guest, as KVMGT
> > > > doesn't
> > > > correctly handle moving memory regions.
> > > >
> > > > Note, this is potential ABI breakage! E.g. userspace could move regions
> > > > that aren't shadowed by KVMGT without harming the guest. However, the
> > > > only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
> > >
> > > This change breaks two kvm selftests:
> > >
> > > - set_memory_region_test;
> > > - memslot_perf_test;
> >
> > It shoudn't. As of this patch, KVM doesn't register itself as a page-track
> > user,
> > i.e. KVMGT is the only remaining caller to
> > kvm_page_track_register_notifier().
> > Unless I messed up, the only way kvm_page_track_has_external_user() can
> > return
> > true is if KVMGT is attached to the VM. The selftests most definitely
> > don't do
> > anything with KVMGT, so I don't see how they can fail.
> >
> > Are you seeing actually failures?
>
> $ set_memory_region_test
...
> At one point I wondered if some of the less common kconfig's were enabled,
> but the above two test failures could be easily fixed with the following diff:
Argh, none of the configs I actually ran selftests on selected
CONFIG_KVM_EXTERNAL_WRITE_TRACKING=y.
> diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
> index 62f98c6c5af3..d4d72ed999b1 100644
> --- a/arch/x86/kvm/mmu/page_track.h
> +++ b/arch/x86/kvm/mmu/page_track.h
> @@ -32,7 +32,7 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
> kvm_memory_slot *slot);
>
> static inline bool kvm_page_track_has_external_user(struct kvm *kvm)
> {
> - return hlist_empty(>arch.track_notifier_head.track_notifier_list);
> + return !hlist_empty(>arch.track_notifier_head.track_notifier_list);
> }
> #else
> static inline int kvm_page_track_init(struct kvm *kvm) { return 0; }
>
> , so I guess it's pretty obvious what's going on here.
Yes. I'll rerun tests today and get the above posted on your behalf (unless you
don't want me to do that).
Sorry for yet another screw up, and thanks for testing!
Re: [Intel-gfx] [PATCH v4 16/29] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
On Wed, Aug 30, 2023, Like Xu wrote:
> On 2023/7/29 09:35, Sean Christopherson wrote:
> > Disallow moving memslots if the VM has external page-track users, i.e. if
> > KVMGT is being used to expose a virtual GPU to the guest, as KVMGT doesn't
> > correctly handle moving memory regions.
> >
> > Note, this is potential ABI breakage! E.g. userspace could move regions
> > that aren't shadowed by KVMGT without harming the guest. However, the
> > only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
>
> This change breaks two kvm selftests:
>
> - set_memory_region_test;
> - memslot_perf_test;
It shoudn't. As of this patch, KVM doesn't register itself as a page-track
user,
i.e. KVMGT is the only remaining caller to kvm_page_track_register_notifier().
Unless I messed up, the only way kvm_page_track_has_external_user() can return
true is if KVMGT is attached to the VM. The selftests most definitely don't do
anything with KVMGT, so I don't see how they can fail.
Are you seeing actually failures?
> Please help confirm if the tests/doc needs to be updated,
> or if the assumption needs to be further clarified.
What assumption?
> > regions. KVM's own support for moving memory regions was also broken for
> > multiple years (albeit for an edge case, but arguably moving RAM is
> > itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
> > new rmap and large page tracking when moving memslot").
> >
> > Reviewed-by: Yan Zhao
> > Tested-by: Yongwei Ma
> > Signed-off-by: Sean Christopherson
> > ---
> > arch/x86/include/asm/kvm_page_track.h | 3 +++
> > arch/x86/kvm/mmu/page_track.c | 5 +
> > arch/x86/kvm/x86.c| 7 +++
> > 3 files changed, 15 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/kvm_page_track.h
> > b/arch/x86/include/asm/kvm_page_track.h
> > index 8c4d216e3b2b..f744682648e7 100644
> > --- a/arch/x86/include/asm/kvm_page_track.h
> > +++ b/arch/x86/include/asm/kvm_page_track.h
> > @@ -75,4 +75,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
> > void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
> > int bytes);
> > void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot
> > *slot);
> > +
> > +bool kvm_page_track_has_external_user(struct kvm *kvm);
> > +
> > #endif
> > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
> > index 891e5cc52b45..e6de9638e560 100644
> > --- a/arch/x86/kvm/mmu/page_track.c
> > +++ b/arch/x86/kvm/mmu/page_track.c
> > @@ -303,3 +303,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
> > kvm_memory_slot *slot)
> > n->track_flush_slot(kvm, slot, n);
> > srcu_read_unlock(>track_srcu, idx);
> > }
> > +
> > +bool kvm_page_track_has_external_user(struct kvm *kvm)
> > +{
> > + return hlist_empty(>arch.track_notifier_head.track_notifier_list);
> > +}
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 059571d5abed..4394bb49051f 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -12606,6 +12606,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
> >struct kvm_memory_slot *new,
> >enum kvm_mr_change change)
> > {
> > + /*
> > +* KVM doesn't support moving memslots when there are external page
> > +* trackers attached to the VM, i.e. if KVMGT is in use.
> > +*/
> > + if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
> > + return -EINVAL;
> > +
> > if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
> > if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
> > return -EINVAL;
Re: [Intel-gfx] [PATCH 19/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Thu, Aug 10, 2023, Yan Zhao wrote:
> On Thu, Aug 10, 2023 at 07:21:03AM +0800, Yan Zhao wrote:
> > > Reading the value after acquiring mmu_lock ensures that both vCPUs will
> > > see whatever
> > > value "loses" the race, i.e. whatever written value is processed second
> > > ('Y' in the
> > > above sequence).
> > I suspect that vCPU0 may still generate a wrong SPTE if vCPU1 wrote 4
> > bytes while vCPU0 wrote 8 bytes, though the chances are very low.
> >
> This could happen in below sequence:
> vCPU0 updates a PTE to AABBCCDD;
> vCPU1 updates a PTE to EEFFGGHH in two writes.
> (each character stands for a byte)
>
> vCPU0 vCPU1
> write AABBCCDD
>write GGHH
>detect 4 bytes write and hold on sync
> sync SPTE w/ AABBGGHH
>write EEFF
>sync SPTE w/ EEFFGGHH
>
>
> Do you think it worth below serialization work?
No, because I don't see any KVM bugs with the above sequence. If the guest
doesn't
ensure *all* writes from vCPU0 and vCPU1 are fully serialized, then it is
completely
legal for hardware (KVM in this case) to consume AABBGGHH as a PTE. The only
thing
the guest shouldn't see is EEFFCCDD, but I don't see how that can happen.
Re: [Intel-gfx] [PATCH 19/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Wed, Aug 09, 2023, Yan Zhao wrote:
> On Mon, Aug 07, 2023 at 10:19:07AM -0700, Sean Christopherson wrote:
> > On Mon, Aug 07, 2023, Like Xu wrote:
> > > On 23/12/2022 8:57 am, Sean Christopherson wrote:
> > > > +static inline void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t
> > > > gpa,
> > > > + const u8 *new, int bytes)
> > > > +{
> > > > + __kvm_page_track_write(vcpu, gpa, new, bytes);
> > > > +
> > > > + kvm_mmu_track_write(vcpu, gpa, new, bytes);
> > > > +}
> > >
> > > The kvm_mmu_track_write() is only used for x86, where the incoming
> > > parameter
> > > "u8 *new" has not been required since 0e0fee5c539b ("kvm: mmu: Fix race in
> > > emulated page table writes"), please help confirm if it's still needed ?
> > > Thanks.
> > > A minor clean up is proposed.
> >
> > Hmm, unless I'm misreading things, KVMGT ultimately doesn't consume @new
> > either.
> > So I think we can remove @new from kvm_page_track_write() entirely.
> Sorry for the late reply.
> Yes, KVMGT does not consume @new and it reads the guest PTE again in the
> page track write handler.
>
> But I have a couple of questions related to the memtioned commit as
> below:
>
> (1) If "re-reading the current value of the guest PTE after the MMU lock has
> been acquired", then should KVMGT also acquire the MMU lock too?
No. If applicable, KVMGT should read the new/current value after acquiring
whatever lock protects the generation (or update) of the shadow entries. I
suspect KVMGT already does this, but I don't have time to confirm that at this
exact memory.
The race that was fixed in KVM was:
vCPU0 vCPU1
write X
write Y
sync SPTE w/ Y
sync SPTE w/ X
Reading the value after acquiring mmu_lock ensures that both vCPUs will see
whatever
value "loses" the race, i.e. whatever written value is processed second ('Y' in
the
above sequence).
> If so, could we move the MMU lock and unlock into kvm_page_track_write()
> as it's common.
>
> (2) Even if KVMGT consumes @new,
> will kvm_page_track_write() be called for once or twice if there are two
> concurent emulated write?
Twice, kvm_page_track_write() is wired up directly to the emulation of the
write,
i.e. there is no batching.
Re: [Intel-gfx] [PATCH 19/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Mon, Aug 07, 2023, Like Xu wrote:
> On 23/12/2022 8:57 am, Sean Christopherson wrote:
> > +static inline void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
> > + const u8 *new, int bytes)
> > +{
> > + __kvm_page_track_write(vcpu, gpa, new, bytes);
> > +
> > + kvm_mmu_track_write(vcpu, gpa, new, bytes);
> > +}
>
> The kvm_mmu_track_write() is only used for x86, where the incoming parameter
> "u8 *new" has not been required since 0e0fee5c539b ("kvm: mmu: Fix race in
> emulated page table writes"), please help confirm if it's still needed ?
> Thanks.
> A minor clean up is proposed.
Hmm, unless I'm misreading things, KVMGT ultimately doesn't consume @new either.
So I think we can remove @new from kvm_page_track_write() entirely.
Feel free to send a patch, otherwise I'll get to it later this week.
Re: [Intel-gfx] [PATCH v4 00/29] drm/i915/gvt: KVM: KVMGT fixes and page-track cleanups
On Fri, 28 Jul 2023 18:35:06 -0700, Sean Christopherson wrote:
> Fix a handful of minor bugs in KVMGT, and overhaul KVM's page-track APIs
> to provide a leaner and cleaner interface. The motivation for this
> series is to (significantly) reduce the number of KVM APIs that KVMGT
> uses, with a long-term goal of making all kvm_host.h headers KVM-internal.
>
> If there are no objections or issues, my plan is to take this through the
> KVM tree for 6.6 (I had it ready early last week, and then forgot to actually
> post v4, /facepalm).
>
> [...]
Applied to kvm-x86 mmu, thanks!
[01/29] drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
https://github.com/kvm-x86/linux/commit/865327865164
[02/29] drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
https://github.com/kvm-x86/linux/commit/823ab2ea8429
[03/29] drm/i915/gvt: Verify hugepages are contiguous in physical address space
https://github.com/kvm-x86/linux/commit/e27395fb1b87
[04/29] drm/i915/gvt: Don't try to unpin an empty page range
https://github.com/kvm-x86/linux/commit/6a718c54c2ee
[05/29] drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
https://github.com/kvm-x86/linux/commit/f969ecabe30b
[06/29] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
https://github.com/kvm-x86/linux/commit/537eef32e720
[07/29] drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT entry
type
https://github.com/kvm-x86/linux/commit/c94811471997
[08/29] drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M GTT
https://github.com/kvm-x86/linux/commit/f018c319cc2f
[09/29] drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
https://github.com/kvm-x86/linux/commit/4879a4370304
[10/29] drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
https://github.com/kvm-x86/linux/commit/ac5e77621712
[11/29] drm/i915/gvt: Protect gfn hash table with vgpu_lock
https://github.com/kvm-x86/linux/commit/49a83e190b5b
[12/29] KVM: x86/mmu: Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c
https://github.com/kvm-x86/linux/commit/2f502998b046
[13/29] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
https://github.com/kvm-x86/linux/commit/e2fe84fb5eae
[14/29] KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
https://github.com/kvm-x86/linux/commit/f1c58cdb8e04
[15/29] KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
https://github.com/kvm-x86/linux/commit/3f8eb1d7d3ee
[16/29] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
https://github.com/kvm-x86/linux/commit/aa611a99adb4
[17/29] drm/i915/gvt: Don't bother removing write-protection on to-be-deleted
slot
https://github.com/kvm-x86/linux/commit/a41e34b05da7
[18/29] KVM: x86: Add a new page-track hook to handle memslot deletion
https://github.com/kvm-x86/linux/commit/cc49e12d8d3b
[19/29] drm/i915/gvt: switch from ->track_flush_slot() to
->track_remove_region()
https://github.com/kvm-x86/linux/commit/b9ae8a09f357
[20/29] KVM: x86: Remove the unused page-track hook track_flush_slot()
https://github.com/kvm-x86/linux/commit/1265fb534fa1
[21/29] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
https://github.com/kvm-x86/linux/commit/c87966b313cc
[22/29] KVM: x86/mmu: Use page-track notifiers iff there are external users
https://github.com/kvm-x86/linux/commit/b5e33f265acd
[23/29] KVM: x86/mmu: Drop infrastructure for multiple page-track modes
https://github.com/kvm-x86/linux/commit/2431c9ab231a
[24/29] KVM: x86/mmu: Rename page-track APIs to reflect the new reality
https://github.com/kvm-x86/linux/commit/4b42f39917c1
[25/29] KVM: x86/mmu: Assert that correct locks are held for page write-tracking
https://github.com/kvm-x86/linux/commit/21e0e1efd880
[26/29] KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
https://github.com/kvm-x86/linux/commit/6d6ff9e6db15
[27/29] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
https://github.com/kvm-x86/linux/commit/ca181aa50724
[28/29] KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
https://github.com/kvm-x86/linux/commit/eae2d71635a1
[29/29] drm/i915/gvt: Drop final dependencies on KVM internal details
https://github.com/kvm-x86/linux/commit/2d28b1230c1c
--
https://github.com/kvm-x86/linux/tree/next
https://github.com/kvm-x86/linux/tree/fixes
Re: [Intel-gfx] [PATCH v4 06/29] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
On Tue, Aug 01, 2023, Yan Zhao wrote: > On Fri, Jul 28, 2023 at 06:35:12PM -0700, Sean Christopherson wrote: > > Move the check that a vGPU is attacked from is_2MB_gtt_possible() all the > typo: "attacked" --> "attached" > > > way up to shadow_ppgtt_mm() to avoid unnecessary work, and to make it more > This commit message does not match to what the patch does. > The check in the patch is in ppgtt_populate_shadow_entry(). > > What you want is like below? Yeah, I completely botched this and forgot to actually change the code. I'll send a replacement patch in-reply (or rather, I already sent it; I missed up again and forgot to hit send on this one earlier in the day...). Thanks yet again for the review!
[Intel-gfx] [PATCH v4.1] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
Move the check that a vGPU is attached from is_2MB_gtt_possible() all the
way up to shadow_ppgtt_mm() to avoid unnecessary work, and to make it more
obvious that a future cleanup of is_2MB_gtt_possible() isn't introducing a
bug.
is_2MB_gtt_possible() has only one caller, ppgtt_populate_shadow_entry(),
and all paths in ppgtt_populate_shadow_entry() eventually check for
attachment by way of intel_gvt_dma_map_guest_page().
And of the paths that lead to ppgtt_populate_shadow_entry(),
shadow_ppgtt_mm() is the only one that doesn't already check for
INTEL_VGPU_STATUS_ACTIVE or INTEL_VGPU_STATUS_ATTACHED.
workload_thread() <= pick_next_workload() => INTEL_VGPU_STATUS_ACTIVE
|
-> dispatch_workload()
|
|-> prepare_workload()
|
-> intel_vgpu_sync_oos_pages()
| |
| |-> ppgtt_set_guest_page_sync()
| |
| |-> sync_oos_page()
| |
| |-> ppgtt_populate_shadow_entry()
|
|-> intel_vgpu_flush_post_shadow()
|
1: |-> ppgtt_handle_guest_write_page_table()
|
|-> ppgtt_handle_guest_entry_add()
|
2: | -> ppgtt_populate_spt_by_guest_entry()
||
||-> ppgtt_populate_spt()
||
||-> ppgtt_populate_shadow_entry()
||
||-> ppgtt_populate_spt_by_guest_entry() [see
2]
|
|-> ppgtt_populate_shadow_entry()
kvmgt_page_track_write() <= KVM callback => INTEL_VGPU_STATUS_ATTACHED
|
|-> intel_vgpu_page_track_handler()
|
|-> ppgtt_write_protection_handler()
|
|-> ppgtt_handle_guest_write_page_table_bytes()
|
|-> ppgtt_handle_guest_write_page_table() [see 1]
Signed-off-by: Sean Christopherson
---
v4.1:
- Actually make the code do what the changelog says. [Yan]
- Fix a typo in the changelog. [Yan]
drivers/gpu/drm/i915/gvt/gtt.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 5426a27c1b71..de6a484090d7 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1163,8 +1163,6 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return -EINVAL;
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
@@ -1827,6 +1825,9 @@ static int shadow_ppgtt_mm(struct intel_vgpu_mm *mm)
if (mm->ppgtt_mm.shadowed)
return 0;
+ if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
+ return -EINVAL;
+
mm->ppgtt_mm.shadowed = true;
for (index = 0; index < ARRAY_SIZE(mm->ppgtt_mm.guest_pdps); index++) {
base-commit: 03e8f77e106ba1d2fd980f8b38339dad3a07
--
2.41.0.585.gd2178a4bd4-goog
[Intel-gfx] [PATCH v4 27/29] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
Refactor KVM's exported/external page-track, a.k.a. write-track, APIs
to take only the gfn and do the required memslot lookup in KVM proper.
Forcing users of the APIs to get the memslot unnecessarily bleeds
KVM internals into KVMGT and complicates usage of the APIs.
No functional change intended.
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 7 +--
arch/x86/kvm/mmu/mmu.c| 4 +-
arch/x86/kvm/mmu/page_track.c | 85 ---
arch/x86/kvm/mmu/page_track.h | 5 ++
drivers/gpu/drm/i915/gvt/kvmgt.c | 37 +++-
5 files changed, 80 insertions(+), 58 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index f5c1db36cdb7..4afab697e21c 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,11 +4,6 @@
#include
-void kvm_write_track_add_gfn(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn);
-
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
@@ -55,6 +50,8 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
+int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
/*
* Allow defining a node in a structure even if page tracking is disabled, e.g.
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index a0309fde3549..c6ae1885371c 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -840,7 +840,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_add_gfn(kvm, slot, gfn);
+ return __kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -886,7 +886,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_remove_gfn(kvm, slot, gfn);
+ return __kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index eedb5889d73e..2a64df38ccab 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -74,16 +74,8 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
slot->arch.gfn_write_track[index] += count;
}
-/*
- * add guest page to the tracking pool so that corresponding access on that
- * page will be intercepted.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-gfn_t gfn)
+void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -104,18 +96,9 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct
kvm_memory_slot *slot,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
-/*
- * remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_remove_gfn(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void __kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -133,7 +116,6 @@ void kvm_write_track_remove_gfn(struct kvm *kvm,
*/
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
/*
* check if the corresponding access on the specified guest page is tracked.
@@ -257,4 +239,63 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
srcu_read_unlock(>track_srcu, idx);
}
+/*
+ * add guest page to the tracking pool so that corresponding access on that
+ * page will be intercepted.
+ *
+ * @kvm: the guest instance we are interested in.
+ * @gfn
[Intel-gfx] [PATCH v4 25/29] KVM: x86/mmu: Assert that correct locks are held for page write-tracking
When adding/removing gfns to/from write-tracking, assert that mmu_lock
is held for write, and that either slots_lock or kvm->srcu is held.
mmu_lock must be held for write to protect gfn_write_track's refcount,
and SRCU or slots_lock must be held to protect the memslot itself.
Tested-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/page_track.c | 17 +++--
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index b835ba7f325c..29ae61f1e303 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -12,6 +12,7 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include
#include
#include
@@ -77,9 +78,6 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* add guest page to the tracking pool so that corresponding access on that
* page will be intercepted.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -87,6 +85,11 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -107,9 +110,6 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
* remove the guest page from the tracking pool which stops the interception
* of corresponding access on that page.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -117,6 +117,11 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
void kvm_write_track_remove_gfn(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 26/29] KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
Bug the VM if something attempts to write-track a gfn, but write-tracking isn't enabled. The VM is doomed (and KVM has an egregious bug) if KVM or KVMGT wants to shadow guest page tables but can't because write-tracking isn't enabled. Tested-by: Yongwei Ma Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/page_track.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index 29ae61f1e303..eedb5889d73e 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -90,7 +90,7 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, 1); @@ -122,7 +122,7 @@ void kvm_write_track_remove_gfn(struct kvm *kvm, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, -1); -- 2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 24/29] KVM: x86/mmu: Rename page-track APIs to reflect the new reality
Rename the page-track APIs to capture that they're all about tracking
writes, now that the facade of supporting multiple modes is gone.
Opportunstically replace "slot" with "gfn" in anticipation of removing
the @slot param from the external APIs.
No functional change intended.
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 8
arch/x86/kvm/mmu/mmu.c| 8
arch/x86/kvm/mmu/page_track.c | 21 +
arch/x86/kvm/mmu/page_track.h | 4 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 ++--
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 9e4ee26d1779..f5c1db36cdb7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,10 +4,10 @@
#include
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_add_gfn(struct kvm *kvm,
+struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b8dce17bffdc..a0309fde3549 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -840,7 +840,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn);
+ return kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -886,7 +886,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_remove_page(kvm, slot, gfn);
+ return kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
@@ -2830,7 +2830,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct
kvm_memory_slot *slot,
* track machinery is used to write-protect upper-level shadow pages,
* i.e. this guards the role.level == 4K assertion below!
*/
- if (kvm_slot_page_track_is_active(kvm, slot, gfn))
+ if (kvm_gfn_is_write_tracked(kvm, slot, gfn))
return -EPERM;
/*
@@ -4231,7 +4231,7 @@ static bool page_fault_handle_page_track(struct kvm_vcpu
*vcpu,
* guest is writing the page which is write tracked which can
* not be fixed by page fault handler.
*/
- if (kvm_slot_page_track_is_active(vcpu->kvm, fault->slot, fault->gfn))
+ if (kvm_gfn_is_write_tracked(vcpu->kvm, fault->slot, fault->gfn))
return true;
return false;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index cdc6069b8caf..b835ba7f325c 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -84,10 +84,9 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+gfn_t gfn)
{
-
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -102,12 +101,11 @@ void kvm_slot_page_track_add_page(struct kvm *kvm,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
+EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
/*
* remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page. It is the opposed operation of
- * kvm_slot_page_track_add_page().
+ * of corresponding access on that page.
*
* It should be called under the protection both of mmu-lock and kvm->srcu
* or kvm->slots_lock.
@@ -116,8 +114,8 @@ EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_remove_gfn(struct kvm *kvm,
+
[Intel-gfx] [PATCH v4 28/29] KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
Get/put references to KVM when a page-track notifier is (un)registered
instead of relying on the caller to do so. Forcing the caller to do the
bookkeeping is unnecessary and adds one more thing for users to get
wrong, e.g. see commit 9ed1fdee9ee3 ("drm/i915/gvt: Get reference to KVM
iff attachment to VM is successful").
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 11 +--
arch/x86/kvm/mmu/page_track.c | 18 --
drivers/gpu/drm/i915/gvt/kvmgt.c | 17 +++--
3 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 4afab697e21c..3d040741044b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -44,12 +44,11 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n);
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n);
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n);
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n);
+
int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a64df38ccab..fd04e618ad2d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -157,17 +157,22 @@ int kvm_page_track_init(struct kvm *kvm)
* register the notifier so that event interception for the tracked guest
* pages can be received.
*/
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n)
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
+ if (!kvm || kvm->mm != current->mm)
+ return -ESRCH;
+
+ kvm_get_kvm(kvm);
+
head = >arch.track_notifier_head;
write_lock(>mmu_lock);
hlist_add_head_rcu(>node, >track_notifier_list);
write_unlock(>mmu_lock);
+ return 0;
}
EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
@@ -175,9 +180,8 @@ EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
* stop receiving the event interception. It is the opposed operation of
* kvm_page_track_register_notifier().
*/
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n)
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
@@ -187,6 +191,8 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
hlist_del_rcu(>node);
write_unlock(>mmu_lock);
synchronize_srcu(>track_srcu);
+
+ kvm_put_kvm(kvm);
}
EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 21342a93e418..eb50997dd369 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -654,21 +654,19 @@ static bool __kvmgt_vgpu_exist(struct intel_vgpu *vgpu)
static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
{
struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
-
- if (!vgpu->vfio_device.kvm ||
- vgpu->vfio_device.kvm->mm != current->mm) {
- gvt_vgpu_err("KVM is required to use Intel vGPU\n");
- return -ESRCH;
- }
+ int ret;
if (__kvmgt_vgpu_exist(vgpu))
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
- kvm_get_kvm(vgpu->vfio_device.kvm);
- kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
->track_node);
+ ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
+ >track_node);
+ if (ret) {
+ gvt_vgpu_err("KVM is required to use Intel vGPU\n");
+ return ret;
+ }
set_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status);
@@ -703,7 +701,6 @@ static void intel_vgpu_close_device(s
[Intel-gfx] [PATCH v4 29/29] drm/i915/gvt: Drop final dependencies on KVM internal details
Open code gpa_to_gfn() in kvmgt_page_track_write() and drop KVMGT's dependency on kvm_host.h, i.e. include only kvm_page_track.h. KVMGT assumes "gfn == gpa >> PAGE_SHIFT" all over the place, including a few lines below in the same function with the same gpa, i.e. there's no reason to use KVM's helper for this one case. No functional change intended. Reviewed-by: Yan Zhao Tested-by: Yongwei Ma Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/gvt.h | 3 ++- drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index 2d65800d8e93..53a0a42a50db 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -34,10 +34,11 @@ #define _GVT_H_ #include -#include #include #include +#include + #include "i915_drv.h" #include "intel_gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index eb50997dd369..aaed3969f204 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -1585,7 +1585,7 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len, mutex_lock(>vgpu_lock); - if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa))) + if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT)) intel_vgpu_page_track_handler(info, gpa, (void *)val, len); -- 2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 21/29] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
Bury the declaration of the page-track helpers that are intended only for
internal KVM use in a "private" header. In addition to guarding against
unwanted usage of the internal-only helpers, dropping their definitions
avoids exposing other structures that should be KVM-internal, e.g. for
memslots. This is a baby step toward making kvm_host.h a KVM-internal
header in the very distant future.
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 21 ++---
arch/x86/kvm/mmu/mmu.c| 3 ++-
arch/x86/kvm/mmu/page_track.c | 8 +--
arch/x86/kvm/mmu/page_track.h | 33 +++
arch/x86/kvm/x86.c| 1 +
5 files changed, 39 insertions(+), 27 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 5c348ffdc194..76c0070dfe2a 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -2,6 +2,8 @@
#ifndef _ASM_X86_KVM_PAGE_TRACK_H
#define _ASM_X86_KVM_PAGE_TRACK_H
+#include
+
enum kvm_page_track_mode {
KVM_PAGE_TRACK_WRITE,
KVM_PAGE_TRACK_MAX,
@@ -46,26 +48,12 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
-
-bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
-int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
-
-void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
-int kvm_page_track_create_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- unsigned long npages);
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot,
- gfn_t gfn, enum kvm_page_track_mode mode);
void
kvm_page_track_register_notifier(struct kvm *kvm,
@@ -73,10 +61,5 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
-void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
-
-bool kvm_page_track_has_external_user(struct kvm *kvm);
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c1d3ac303964..88923b1eb510 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -25,6 +25,7 @@
#include "kvm_cache_regs.h"
#include "smm.h"
#include "kvm_emulate.h"
+#include "page_track.h"
#include "cpuid.h"
#include "spte.h"
@@ -53,7 +54,7 @@
#include
#include
#include
-#include
+
#include "trace.h"
extern bool itlb_multihit_kvm_mitigation;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a6ab7c455c0..e15329d48f95 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -15,10 +15,9 @@
#include
#include
-#include
-
#include "mmu.h"
#include "mmu_internal.h"
+#include "page_track.h"
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm)
{
@@ -300,8 +299,3 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
n->track_remove_region(slot->base_gfn, slot->npages, n);
srcu_read_unlock(>track_srcu, idx);
}
-
-bool kvm_page_track_has_external_user(struct kvm *kvm)
-{
- return hlist_empty(>arch.track_notifier_head.track_notifier_list);
-}
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
new file mode 100644
index ..89712f123ad3
--- /dev/null
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_PAGE_TRACK_H
+#define __KVM_X86_PAGE_TRACK_H
+
+#include
+
+#include
+
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
+int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
+
+void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
+int kvm_page_track_create_memslot(struct kvm *kvm,
+
[Intel-gfx] [PATCH v4 22/29] KVM: x86/mmu: Use page-track notifiers iff there are external users
Disable the page-track notifier code at compile time if there are no
external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
now hooks emulated writes directly instead of relying on the page-track
mechanism.
Provide a stub for "struct kvm_page_track_notifier_node" so that including
headers directly from the command line, e.g. for testing include guards,
doesn't fail due to a struct having an incomplete type.
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 2 ++
arch/x86/include/asm/kvm_page_track.h | 22 +---
arch/x86/kvm/mmu/page_track.c | 10 -
arch/x86/kvm/mmu/page_track.h | 29 +++
4 files changed, 47 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 85605f2497bb..33b1ceb30dd2 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1247,7 +1247,9 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
struct kvm_page_track_notifier_head track_notifier_head;
+#endif
/*
* Protects marking pages unsync during page faults, as TDP MMU page
* faults only take mmu_lock for read. For simplicity, the unsync
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 76c0070dfe2a..61adb07b5927 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -9,6 +9,14 @@ enum kvm_page_track_mode {
KVM_PAGE_TRACK_MAX,
};
+void kvm_slot_page_track_add_page(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn,
+ enum kvm_page_track_mode mode);
+void kvm_slot_page_track_remove_page(struct kvm *kvm,
+struct kvm_memory_slot *slot, gfn_t gfn,
+enum kvm_page_track_mode mode);
+
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
* the head which will be notified when guest is triggering the track event.
@@ -48,18 +56,18 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn,
-enum kvm_page_track_mode mode);
-
void
kvm_page_track_register_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+#else
+/*
+ * Allow defining a node in a structure even if page tracking is disabled, e.g.
+ * to play nice with testing headers via direct inclusion from the command
line.
+ */
+struct kvm_page_track_notifier_node {};
+#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e15329d48f95..b20aad7ac3fe 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -194,6 +194,7 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
return !!READ_ONCE(slot->arch.gfn_track[mode][index]);
}
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
void kvm_page_track_cleanup(struct kvm *kvm)
{
struct kvm_page_track_notifier_head *head;
@@ -255,14 +256,13 @@ EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
* The node should figure out if the written page is the one that node is
* interested in by itself.
*/
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes)
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int
bytes)
{
struct kvm_page_track_notifier_head *head;
struct kvm_page_track_notifier_node *n;
int idx;
- head = >kvm->arch.track_notifier_head;
+ head = >arch.track_notifier_head;
if (hlist_empty(>track_notifier_list))
return;
@@ -273,8 +273,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
-
- kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
@@ -299,3 +297,5 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slo
[Intel-gfx] [PATCH v4 20/29] KVM: x86: Remove the unused page-track hook track_flush_slot()
From: Yan Zhao
Remove ->track_remove_slot(), there are no longer any users and it's
unlikely a "flush" hook will ever be the correct API to provide to an
external page-track user.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 11 ---
arch/x86/kvm/mmu/mmu.c| 2 --
arch/x86/kvm/mmu/page_track.c | 26 --
3 files changed, 39 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index cfd36c22b467..5c348ffdc194 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -33,16 +33,6 @@ struct kvm_page_track_notifier_node {
*/
void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
struct kvm_page_track_notifier_node *node);
- /*
-* It is called when memory slot is being moved or removed
-* users can drop write-protection for the pages in that memory slot
-*
-* @kvm: the kvm where memory slot being moved or removed
-* @slot: the memory slot being moved or removed
-* @node: this node
-*/
- void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
/*
* Invoked when a memory region is removed from the guest. Or in KVM
@@ -85,7 +75,6 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c404264f8de5..c1d3ac303964 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6749,8 +6749,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
kvm_mmu_zap_all_fast(kvm);
-
- kvm_page_track_flush_slot(kvm, slot);
}
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index d971c28be99d..2a6ab7c455c0 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -278,32 +278,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t
gpa, const u8 *new,
kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
-/*
- * Notify the node that memory slot is being removed or moved so that it can
- * drop write-protection for the pages in the memory slot.
- *
- * The node should figure out it has any write-protected pages in this slot
- * by itself.
- */
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
-{
- struct kvm_page_track_notifier_head *head;
- struct kvm_page_track_notifier_node *n;
- int idx;
-
- head = >arch.track_notifier_head;
-
- if (hlist_empty(>track_notifier_list))
- return;
-
- idx = srcu_read_lock(>track_srcu);
- hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
- if (n->track_flush_slot)
- n->track_flush_slot(kvm, slot, n);
- srcu_read_unlock(>track_srcu, idx);
-}
-
/*
* Notify external page track nodes that a memory region is being removed from
* the VM, e.g. so that users can free any associated metadata.
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 19/29] drm/i915/gvt: switch from ->track_flush_slot() to ->track_remove_region()
From: Yan Zhao
Switch from the poorly named and flawed ->track_flush_slot() to the newly
introduced ->track_remove_region(). From KVMGT's perspective, the two
hooks are functionally equivalent, the only difference being that
->track_remove_region() is called only when KVM is 100% certain the
memory region will be removed, i.e. is invoked slightly later in KVM's
memslot modification flow.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
[sean: handle name change, massage changelog, rebase]
Tested-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 21 +
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 3ea3cb9eb599..3f2327455d85 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -108,9 +108,8 @@ struct gvt_dma {
static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
struct kvm_page_track_notifier_node *node);
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node);
static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf)
{
@@ -666,7 +665,7 @@ static int intel_vgpu_open_device(struct vfio_device
*vfio_dev)
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
- vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
+ vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
kvm_get_kvm(vgpu->vfio_device.kvm);
kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
>track_node);
@@ -1617,22 +1616,20 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8
*val, int len,
mutex_unlock(>vgpu_lock);
}
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node)
{
unsigned long i;
- gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
mutex_lock(>vgpu_lock);
- for (i = 0; i < slot->npages; i++) {
- gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn))
- kvmgt_protect_table_del(info, gfn);
+ for (i = 0; i < nr_pages; i++) {
+ if (kvmgt_gfn_is_write_protected(info, gfn + i))
+ kvmgt_protect_table_del(info, gfn + i);
}
+
mutex_unlock(>vgpu_lock);
}
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 18/29] KVM: x86: Add a new page-track hook to handle memslot deletion
From: Yan Zhao
Add a new page-track hook, track_remove_region(), that is called when a
memslot DELETE operation is about to be committed. The "remove" hook
will be used by KVMGT and will effectively replace the existing
track_flush_slot() altogether now that KVM itself doesn't rely on the
"flush" hook either.
The "flush" hook is flawed as it's invoked before the memslot operation
is guaranteed to succeed, i.e. KVM might ultimately keep the existing
memslot without notifying external page track users, a.k.a. KVMGT. In
practice, this can't currently happen on x86, but there are no guarantees
that won't change in the future, not to mention that "flush" does a very
poor job of describing what is happening.
Pass in the gfn+nr_pages instead of the slot itself so external users,
i.e. KVMGT, don't need to exposed to KVM internals (memslots). This will
help set the stage for additional cleanups to the page-track APIs.
Opportunistically align the existing srcu_read_lock_held() usage so that
the new case doesn't stand out like a sore thumb (and not aligning the
new code makes bots unhappy).
Cc: Zhenyu Wang
Tested-by: Yongwei Ma
Signed-off-by: Yan Zhao
Co-developed-by: Sean Christopherson
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 12
arch/x86/kvm/mmu/page_track.c | 27 +--
arch/x86/kvm/x86.c| 3 +++
3 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index f744682648e7..cfd36c22b467 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,6 +43,17 @@ struct kvm_page_track_notifier_node {
*/
void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
+
+ /*
+* Invoked when a memory region is removed from the guest. Or in KVM
+* terms, when a memslot is deleted.
+*
+* @gfn: base gfn of the region being removed
+* @nr_pages: number of pages in the to-be-removed region
+* @node: this node
+*/
+ void (*track_remove_region)(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node);
};
int kvm_page_track_init(struct kvm *kvm);
@@ -75,6 +86,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e6de9638e560..d971c28be99d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -270,7 +270,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
idx = srcu_read_lock(>track_srcu);
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
+ srcu_read_lock_held(>track_srcu))
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
@@ -298,12 +298,35 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
idx = srcu_read_lock(>track_srcu);
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
+ srcu_read_lock_held(>track_srcu))
if (n->track_flush_slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(>track_srcu, idx);
}
+/*
+ * Notify external page track nodes that a memory region is being removed from
+ * the VM, e.g. so that users can free any associated metadata.
+ */
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+ struct kvm_page_track_notifier_head *head;
+ struct kvm_page_track_notifier_node *n;
+ int idx;
+
+ head = >arch.track_notifier_head;
+
+ if (hlist_empty(>track_notifier_list))
+ return;
+
+ idx = srcu_read_lock(>track_srcu);
+ hlist_for_each_entry_srcu(n, >track_notifier_list, node,
+ srcu_read_lock_held(>track_srcu))
+ if (n->track_remove_region)
+ n->track_remove_region(slot->base_gfn, slot->npages, n);
+ srcu_read_unlock(>track_srcu, idx);
+}
+
bool kvm_page_track_has_external_user(struct kvm *kvm)
[Intel-gfx] [PATCH v4 23/29] KVM: x86/mmu: Drop infrastructure for multiple page-track modes
Drop "support" for multiple page-track modes, as there is no evidence
that array-based and refcounted metadata is the optimal solution for
other modes, nor is there any evidence that other use cases, e.g. for
access-tracking, will be a good fit for the page-track machinery in
general.
E.g. one potential use case of access-tracking would be to prevent guest
access to poisoned memory (from the guest's perspective). In that case,
the number of poisoned pages is likely to be a very small percentage of
the guest memory, and there is no need to reference count the number of
access-tracking users, i.e. expanding gfn_track[] for a new mode would be
grossly inefficient. And for poisoned memory, host userspace would also
likely want to trap accesses, e.g. to inject #MC into the guest, and that
isn't currently supported by the page-track framework.
A better alternative for that poisoned page use case is likely a
variation of the proposed per-gfn attributes overlay (linked), which
would allow efficiently tracking the sparse set of poisoned pages, and by
default would exit to userspace on access.
Link: https://lore.kernel.org/all/y2wb48kd0j4vg...@google.com
Cc: Ben Gardon
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 12 +--
arch/x86/include/asm/kvm_page_track.h | 11 +--
arch/x86/kvm/mmu/mmu.c| 14 ++--
arch/x86/kvm/mmu/page_track.c | 111 --
arch/x86/kvm/mmu/page_track.h | 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 +-
6 files changed, 51 insertions(+), 104 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 33b1ceb30dd2..a915e23d61fa 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -288,13 +288,13 @@ struct kvm_kernel_irq_routing_entry;
* kvm_mmu_page_role tracks the properties of a shadow page (where shadow page
* also includes TDP pages) to determine whether or not a page can be used in
* the given MMU context. This is a subset of the overall kvm_cpu_role to
- * minimize the size of kvm_memory_slot.arch.gfn_track, i.e. allows allocating
- * 2 bytes per gfn instead of 4 bytes per gfn.
+ * minimize the size of kvm_memory_slot.arch.gfn_write_track, i.e. allows
+ * allocating 2 bytes per gfn instead of 4 bytes per gfn.
*
* Upper-level shadow pages having gptes are tracked for write-protection via
- * gfn_track. As above, gfn_track is a 16 bit counter, so KVM must not create
- * more than 2^16-1 upper-level shadow pages at a single gfn, otherwise
- * gfn_track will overflow and explosions will ensure.
+ * gfn_write_track. As above, gfn_write_track is a 16 bit counter, so KVM must
+ * not create more than 2^16-1 upper-level shadow pages at a single gfn,
+ * otherwise gfn_write_track will overflow and explosions will ensue.
*
* A unique shadow page (SP) for a gfn is created if and only if an existing SP
* cannot be reused. The ability to reuse a SP is tracked by its role, which
@@ -1005,7 +1005,7 @@ struct kvm_lpage_info {
struct kvm_arch_memory_slot {
struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES];
struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1];
- unsigned short *gfn_track[KVM_PAGE_TRACK_MAX];
+ unsigned short *gfn_write_track;
};
/*
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 61adb07b5927..9e4ee26d1779 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,17 +4,10 @@
#include
-enum kvm_page_track_mode {
- KVM_PAGE_TRACK_WRITE,
- KVM_PAGE_TRACK_MAX,
-};
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
+ struct kvm_memory_slot *slot, gfn_t gfn);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn,
-enum kvm_page_track_mode mode);
+struct kvm_memory_slot *slot, gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 88923b1eb510..b8dce17bffdc 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -840,8 +840,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
+ return kvm_slot_page_track_add_page(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -887,8 +886,7 @@ static void unaccount_shadowed(struct
[Intel-gfx] [PATCH v4 16/29] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
Disallow moving memslots if the VM has external page-track users, i.e. if
KVMGT is being used to expose a virtual GPU to the guest, as KVMGT doesn't
correctly handle moving memory regions.
Note, this is potential ABI breakage! E.g. userspace could move regions
that aren't shadowed by KVMGT without harming the guest. However, the
only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
regions. KVM's own support for moving memory regions was also broken for
multiple years (albeit for an edge case, but arguably moving RAM is
itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
new rmap and large page tracking when moving memslot").
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 3 +++
arch/x86/kvm/mmu/page_track.c | 5 +
arch/x86/kvm/x86.c| 7 +++
3 files changed, 15 insertions(+)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 8c4d216e3b2b..f744682648e7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -75,4 +75,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+
+bool kvm_page_track_has_external_user(struct kvm *kvm);
+
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 891e5cc52b45..e6de9638e560 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -303,3 +303,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(>track_srcu, idx);
}
+
+bool kvm_page_track_has_external_user(struct kvm *kvm)
+{
+ return hlist_empty(>arch.track_notifier_head.track_notifier_list);
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 059571d5abed..4394bb49051f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12606,6 +12606,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ /*
+* KVM doesn't support moving memslots when there are external page
+* trackers attached to the VM, i.e. if KVMGT is in use.
+*/
+ if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
+ return -EINVAL;
+
if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
return -EINVAL;
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 15/29] KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
Drop @vcpu from KVM's ->track_write() hook provided for external users of
the page-track APIs now that KVM itself doesn't use the page-track
mechanism.
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 5 ++---
arch/x86/kvm/mmu/page_track.c | 2 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 10 --
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index eb186bc57f6a..8c4d216e3b2b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -26,14 +26,13 @@ struct kvm_page_track_notifier_node {
* It is called when guest is writing the write-tracked page
* and write emulation is finished at that time.
*
-* @vcpu: the vcpu where the write access happened.
* @gpa: the physical address written by guest.
* @new: the data was written to the address.
* @bytes: the written length.
* @node: this node
*/
- void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes, struct kvm_page_track_notifier_node
*node);
+ void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
+ struct kvm_page_track_notifier_node *node);
/*
* It is called when memory slot is being moved or removed
* users can drop write-protection for the pages in that memory slot
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 23088c90d2fd..891e5cc52b45 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -272,7 +272,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
srcu_read_lock_held(>track_srcu))
if (n->track_write)
- n->track_write(vcpu, gpa, new, bytes, n);
+ n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
kvm_mmu_track_write(vcpu, gpa, new, bytes);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 034be0655daa..e9276500435d 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -106,9 +106,8 @@ struct gvt_dma {
#define vfio_dev_to_vgpu(vfio_dev) \
container_of((vfio_dev), struct intel_vgpu, vfio_device)
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node);
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
@@ -1603,9 +1602,8 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info,
u64 gfn)
return 0;
}
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node)
{
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 17/29] drm/i915/gvt: Don't bother removing write-protection on to-be-deleted slot
When handling a slot "flush", don't call back into KVM to drop write
protection for gfns in the slot. Now that KVM rejects attempts to move
memory slots while KVMGT is attached, the only time a slot is "flushed"
is when it's being removed, i.e. the memslot and all its write-tracking
metadata is about to be deleted.
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 8 +---
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index e9276500435d..3ea3cb9eb599 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1630,14 +1630,8 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn)) {
- write_lock(>mmu_lock);
- kvm_slot_page_track_remove_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
- write_unlock(>mmu_lock);
-
+ if (kvmgt_gfn_is_write_protected(info, gfn))
kvmgt_protect_table_del(info, gfn);
- }
}
mutex_unlock(>vgpu_lock);
}
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 14/29] KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
Don't use the generic page-track mechanism to handle writes to guest PTEs
in KVM's MMU. KVM's MMU needs access to information that should not be
exposed to external page-track users, e.g. KVM needs (for some definitions
of "need") the vCPU to query the current paging mode, whereas external
users, i.e. KVMGT, have no ties to the current vCPU and so should never
need the vCPU.
Moving away from the page-track mechanism will allow dropping use of the
page-track mechanism for KVM's own MMU, and will also allow simplifying
and cleaning up the page-track APIs.
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu.h | 2 ++
arch/x86/kvm/mmu/mmu.c | 13 ++---
arch/x86/kvm/mmu/page_track.c | 2 ++
4 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 856ec22aceb6..85605f2497bb 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1247,7 +1247,6 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
- struct kvm_page_track_notifier_node mmu_sp_tracker;
struct kvm_page_track_notifier_head track_notifier_head;
/*
* Protects marking pages unsync during page faults, as TDP MMU page
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 92d5a1924fc1..253fb2093d5d 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -121,6 +121,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes);
static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
{
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 79ea57396d97..c404264f8de5 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5684,9 +5684,8 @@ static u64 *get_written_sptes(struct kvm_mmu_page *sp,
gpa_t gpa, int *nspte)
return spte;
}
-static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *new, int bytes,
- struct kvm_page_track_notifier_node *node)
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes)
{
gfn_t gfn = gpa >> PAGE_SHIFT;
struct kvm_mmu_page *sp;
@@ -6201,7 +6200,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
int kvm_mmu_init_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
int r;
INIT_LIST_HEAD(>arch.active_mmu_pages);
@@ -6215,9 +6213,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
return r;
}
- node->track_write = kvm_mmu_pte_write;
- kvm_page_track_register_notifier(kvm, node);
-
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6238,10 +6233,6 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
void kvm_mmu_uninit_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
-
- kvm_page_track_unregister_notifier(kvm, node);
-
if (tdp_mmu_enabled)
kvm_mmu_uninit_tdp_mmu(kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 0a2ac438d647..23088c90d2fd 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -274,6 +274,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(vcpu, gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
+
+ kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 12/29] KVM: x86/mmu: Move kvm_arch_flush_shadow_{all, memslot}() to mmu.c
Move x86's implementation of kvm_arch_flush_shadow_{all,memslot}() into
mmu.c, and make kvm_mmu_zap_all() static as it was globally visible only
for kvm_arch_flush_shadow_all(). This will allow refactoring
kvm_arch_flush_shadow_memslot() to call kvm_mmu_zap_all() directly without
having to expose kvm_mmu_zap_all_fast() outside of mmu.c. Keeping
everything in mmu.c will also likely simplify supporting TDX, which
intends to do zap only relevant SPTEs on memslot updates.
No functional change intended.
Suggested-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu/mmu.c | 13 -
arch/x86/kvm/x86.c | 11 ---
3 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 28bd38303d70..856ec22aceb6 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1832,7 +1832,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
-void kvm_mmu_zap_all(struct kvm *kvm);
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index ec169f5c7dce..c6dee659d592 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6732,7 +6732,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
*/
}
-void kvm_mmu_zap_all(struct kvm *kvm)
+static void kvm_mmu_zap_all(struct kvm *kvm)
{
struct kvm_mmu_page *sp, *node;
LIST_HEAD(invalid_list);
@@ -6757,6 +6757,17 @@ void kvm_mmu_zap_all(struct kvm *kvm)
write_unlock(>mmu_lock);
}
+void kvm_arch_flush_shadow_all(struct kvm *kvm)
+{
+ kvm_mmu_zap_all(kvm);
+}
+
+void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
+ struct kvm_memory_slot *slot)
+{
+ kvm_page_track_flush_slot(kvm, slot);
+}
+
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
{
WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index a6b9bea62fb8..059571d5abed 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12776,17 +12776,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
kvm_arch_free_memslot(kvm, old);
}
-void kvm_arch_flush_shadow_all(struct kvm *kvm)
-{
- kvm_mmu_zap_all(kvm);
-}
-
-void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot)
-{
- kvm_page_track_flush_slot(kvm, slot);
-}
-
static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)
{
return (is_guest_mode(vcpu) &&
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 11/29] drm/i915/gvt: Protect gfn hash table with vgpu_lock
Use vgpu_lock instead of KVM's mmu_lock to protect accesses to the hash
table used to track which gfns are write-protected when shadowing the
guest's GTT, and hoist the acquisition of vgpu_lock from
intel_vgpu_page_track_handler() out to its sole caller,
kvmgt_page_track_write().
This fixes a bug where kvmgt_page_track_write(), which doesn't hold
kvm->mmu_lock, could race with intel_gvt_page_track_remove() and trigger
a use-after-free.
Fixing kvmgt_page_track_write() by taking kvm->mmu_lock is not an option
as mmu_lock is a r/w spinlock, and intel_vgpu_page_track_handler() might
sleep when acquiring vgpu->cache_lock deep down the callstack:
intel_vgpu_page_track_handler()
|
|-> page_track->handler / ppgtt_write_protection_handler()
|
|-> ppgtt_handle_guest_write_page_table_bytes()
|
|-> ppgtt_handle_guest_write_page_table()
|
|-> ppgtt_handle_guest_entry_removal()
|
|-> ppgtt_invalidate_pte()
|
|-> intel_gvt_dma_unmap_guest_page()
|
|-> mutex_lock(>cache_lock);
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 55 +++
drivers/gpu/drm/i915/gvt/page_track.c | 10 +
2 files changed, 33 insertions(+), 32 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 6f52886c4051..034be0655daa 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -352,6 +352,8 @@ __kvmgt_protect_table_find(struct intel_vgpu *info, gfn_t
gfn)
{
struct kvmgt_pgfn *p, *res = NULL;
+ lockdep_assert_held(>vgpu_lock);
+
hash_for_each_possible(info->ptable, p, hnode, gfn) {
if (gfn == p->gfn) {
res = p;
@@ -1553,6 +1555,9 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64
gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
+ if (kvmgt_gfn_is_write_protected(info, gfn))
+ return 0;
+
idx = srcu_read_lock(>srcu);
slot = gfn_to_memslot(kvm, gfn);
if (!slot) {
@@ -1561,16 +1566,12 @@ int intel_gvt_page_track_add(struct intel_vgpu *info,
u64 gfn)
}
write_lock(>mmu_lock);
-
- if (kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
-
kvm_slot_page_track_add_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(>mmu_lock);
+
+ srcu_read_unlock(>srcu, idx);
+
kvmgt_protect_table_add(info, gfn);
-
-out:
- write_unlock(>mmu_lock);
- srcu_read_unlock(>srcu, idx);
return 0;
}
@@ -1583,24 +1584,22 @@ int intel_gvt_page_track_remove(struct intel_vgpu
*info, u64 gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
- idx = srcu_read_lock(>srcu);
- slot = gfn_to_memslot(kvm, gfn);
- if (!slot) {
- srcu_read_unlock(>srcu, idx);
- return -EINVAL;
- }
-
- write_lock(>mmu_lock);
-
if (!kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
+ return 0;
+ idx = srcu_read_lock(>srcu);
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot) {
+ srcu_read_unlock(>srcu, idx);
+ return -EINVAL;
+ }
+
+ write_lock(>mmu_lock);
kvm_slot_page_track_remove_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(>mmu_lock);
+ srcu_read_unlock(>srcu, idx);
+
kvmgt_protect_table_del(info, gfn);
-
-out:
- write_unlock(>mmu_lock);
- srcu_read_unlock(>srcu, idx);
return 0;
}
@@ -1611,9 +1610,13 @@ static void kvmgt_page_track_write(struct kvm_vcpu
*vcpu, gpa_t gpa,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
+ mutex_lock(>vgpu_lock);
+
if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
intel_vgpu_page_track_handler(info, gpa,
(void *)val, len);
+
+ mutex_unlock(>vgpu_lock);
}
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
@@ -1625,16 +1628,20 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
- write_lock(>mmu_lock);
+ mutex_lock(>vgpu_lock);
+
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
if (kvmgt_gfn_is_write_protected(info, gfn)) {
+ write_lock(>mmu_lock);
[Intel-gfx] [PATCH v4 10/29] drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
Drop intel_vgpu_reset_gtt() as it no longer has any callers. In addition
to eliminating dead code, this eliminates the last possible scenario where
__kvmgt_protect_table_find() can be reached without holding vgpu_lock.
Requiring vgpu_lock to be held when calling __kvmgt_protect_table_find()
will allow a protecting the gfn hash with vgpu_lock without too much fuss.
No functional change intended.
Fixes: ba25d977571e ("drm/i915/gvt: Do not destroy ppgtt_mm during vGPU
D3->D0.")
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 18 --
drivers/gpu/drm/i915/gvt/gtt.h | 1 -
2 files changed, 19 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f505be9e647a..c3c623b929ce 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -2817,24 +2817,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool
invalidate_old)
ggtt_invalidate(gvt->gt);
}
-/**
- * intel_vgpu_reset_gtt - reset the all GTT related status
- * @vgpu: a vGPU
- *
- * This function is called from vfio core to reset reset all
- * GTT related status, including GGTT, PPGTT, scratch page.
- *
- */
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu)
-{
- /* Shadow pages are only created when there is no page
-* table tracking data, so remove page tracking data after
-* removing the shadow pages.
-*/
- intel_vgpu_destroy_all_ppgtt_mm(vgpu);
- intel_vgpu_reset_ggtt(vgpu, true);
-}
-
/**
* intel_gvt_restore_ggtt - restore all vGPU's ggtt entries
* @gvt: intel gvt device
diff --git a/drivers/gpu/drm/i915/gvt/gtt.h b/drivers/gpu/drm/i915/gvt/gtt.h
index a3b0f59ec8bd..4cb183e06e95 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.h
+++ b/drivers/gpu/drm/i915/gvt/gtt.h
@@ -224,7 +224,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool
invalidate_old);
void intel_vgpu_invalidate_ppgtt(struct intel_vgpu *vgpu);
int intel_gvt_init_gtt(struct intel_gvt *gvt);
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu);
void intel_gvt_clean_gtt(struct intel_gvt *gvt);
struct intel_vgpu_mm *intel_gvt_find_ppgtt_mm(struct intel_vgpu *vgpu,
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 13/29] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of
bouncing through the page-track mechanism. KVM (unfortunately) needs to
zap and flush all page tables on memslot DELETE/MOVE irrespective of
whether KVM is shadowing guest page tables.
This will allow changing KVM to register a page-track notifier on the
first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a
different method for reacting to memslot changes.
No functional change intended.
Cc: Yan Zhao
Link: https://lore.kernel.org/r/20221110014821.1548347-2-sea...@google.com
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/mmu.c | 10 ++
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c6dee659d592..79ea57396d97 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6199,13 +6199,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm
*kvm)
return unlikely(!list_empty_careful(>arch.zapped_obsolete_pages));
}
-static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
-{
- kvm_mmu_zap_all_fast(kvm);
-}
-
int kvm_mmu_init_vm(struct kvm *kvm)
{
struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
@@ -6223,7 +6216,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
}
node->track_write = kvm_mmu_pte_write;
- node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
kvm_page_track_register_notifier(kvm, node);
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
@@ -6765,6 +6757,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
+ kvm_mmu_zap_all_fast(kvm);
+
kvm_page_track_flush_slot(kvm, slot);
}
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 08/29] drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M GTT
Now that gvt_pin_guest_page() explicitly verifies the pinned PFN is a
transparent hugepage page, don't use KVM's gfn_to_pfn() to pre-check if a
2MiB GTT entry is possible and instead just try to map the GFN with a 2MiB
entry. Using KVM to query pfn that is ultimately managed through VFIO is
odd, and KVM's gfn_to_pfn() is not intended for non-KVM consumption; it's
exported only because of KVM vendor modules (x86 and PPC).
Open code the check on 2MiB support instead of keeping
is_2MB_gtt_possible() around for a single line of code.
Move the call to intel_gvt_dma_map_guest_page() for a 4KiB entry into its
case statement, i.e. fork the common path into the 4KiB and 2MiB "direct"
shadow paths. Keeping the call in the "common" path is arguably more in
the spirit of "one change per patch", but retaining the local "page_size"
variable is silly, i.e. the call site will be changed either way, and
jumping around the no-longer-common code is more subtle and rather odd,
i.e. would just need to be immediately cleaned up.
Drop the error message from gvt_pin_guest_page() when KVMGT attempts to
shadow a 2MiB guest page that isn't backed by a compatible hugepage in the
host. Dropping the pre-check on a THP makes it much more likely that the
"error" will be encountered in normal operation.
Reviewed-by: Yan Zhao
Tested-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 49 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 1 -
2 files changed, 8 insertions(+), 42 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 61e38acee2d5..f505be9e647a 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1145,36 +1145,6 @@ static inline void ppgtt_generate_shadow_entry(struct
intel_gvt_gtt_entry *se,
ops->set_pfn(se, s->shadow_page.mfn);
}
-/*
- * Check if can do 2M page
- * @vgpu: target vgpu
- * @entry: target pfn's gtt entry
- *
- * Return 1 if 2MB huge gtt shadowing is possible, 0 if miscondition,
- * negative if found err.
- */
-static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
- struct intel_gvt_gtt_entry *entry)
-{
- const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
- kvm_pfn_t pfn;
- int ret;
-
- if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
- return 0;
-
- pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
- if (is_error_noslot_pfn(pfn))
- return -EINVAL;
-
- if (!pfn_valid(pfn))
- return -EINVAL;
-
- ret = PageTransHuge(pfn_to_page(pfn));
- kvm_release_pfn_clean(pfn);
- return ret;
-}
-
static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
struct intel_vgpu_ppgtt_spt *spt, unsigned long index,
struct intel_gvt_gtt_entry *se)
@@ -1268,7 +1238,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
{
const struct intel_gvt_gtt_pte_ops *pte_ops = vgpu->gvt->gtt.pte_ops;
struct intel_gvt_gtt_entry se = *ge;
- unsigned long gfn, page_size = PAGE_SIZE;
+ unsigned long gfn;
dma_addr_t dma_addr;
int ret;
@@ -1283,6 +1253,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
switch (ge->type) {
case GTT_TYPE_PPGTT_PTE_4K_ENTRY:
gvt_vdbg_mm("shadow 4K gtt entry\n");
+ ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE,
_addr);
+ if (ret)
+ return -ENXIO;
break;
case GTT_TYPE_PPGTT_PTE_64K_ENTRY:
gvt_vdbg_mm("shadow 64K gtt entry\n");
@@ -1294,12 +1267,10 @@ static int ppgtt_populate_shadow_entry(struct
intel_vgpu *vgpu,
return split_64KB_gtt_entry(vgpu, spt, index, );
case GTT_TYPE_PPGTT_PTE_2M_ENTRY:
gvt_vdbg_mm("shadow 2M gtt entry\n");
- ret = is_2MB_gtt_possible(vgpu, ge);
- if (ret == 0)
+ if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M)
||
+ intel_gvt_dma_map_guest_page(vgpu, gfn,
+I915_GTT_PAGE_SIZE_2M,
_addr))
return split_2MB_gtt_entry(vgpu, spt, index, );
- else if (ret < 0)
- return ret;
- page_size = I915_GTT_PAGE_SIZE_2M;
break;
case GTT_TYPE_PPGTT_PTE_1G_ENTRY:
gvt_vgpu_err("GVT doesn't support 1GB entry\n");
@@ -1309,11 +1280,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
return -EINVAL;
}
- /* direct shadow */
- ret = intel_gvt_dma_map_guest_page(vgpu, gfn, page_size, _addr);
- if (ret)
-
[Intel-gfx] [PATCH v4 09/29] drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
Use an "unsigned long" instead of an "int" when iterating over the gfns
in a memslot. The number of pages in the memslot is tracked as an
"unsigned long", e.g. KVMGT could theoretically break if a KVM memslot
larger than 16TiB were deleted (2^32 * 4KiB).
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 97c6d3c53710..6f52886c4051 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1620,7 +1620,7 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node)
{
- int i;
+ unsigned long i;
gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 07/29] drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT entry type
Bail from ppgtt_populate_shadow_entry() if an unexpected GTT entry type is encountered instead of subtly falling through to the common "direct shadow" path. Eliminating the default/error path's reliance on the common handling will allow hoisting intel_gvt_dma_map_guest_page() into the case statements so that the 2MiB case can try intel_gvt_dma_map_guest_page() and fallback to splitting the entry on failure. Reviewed-by: Zhi Wang Tested-by: Yongwei Ma Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/gtt.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 2aed31b497c9..61e38acee2d5 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1306,6 +1306,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, return -EINVAL; default: GEM_BUG_ON(1); + return -EINVAL; } /* direct shadow */ -- 2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 05/29] drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
Put the struct page reference acquired by gfn_to_pfn(), KVM's API is that
the caller is ultimately responsible for dropping any reference.
Note, kvm_release_pfn_clean() ensures the pfn is actually a refcounted
struct page before trying to put any references.
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f30922c55a0c..5426a27c1b71 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1158,6 +1158,7 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
{
const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
kvm_pfn_t pfn;
+ int ret;
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
@@ -1171,7 +1172,9 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!pfn_valid(pfn))
return -EINVAL;
- return PageTransHuge(pfn_to_page(pfn));
+ ret = PageTransHuge(pfn_to_page(pfn));
+ kvm_release_pfn_clean(pfn);
+ return ret;
}
static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 06/29] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
Move the check that a vGPU is attacked from is_2MB_gtt_possible() all the
way up to shadow_ppgtt_mm() to avoid unnecessary work, and to make it more
obvious that a future cleanup of is_2MB_gtt_possible() isn't introducing a
bug.
is_2MB_gtt_possible() has only one caller, ppgtt_populate_shadow_entry(),
and all paths in ppgtt_populate_shadow_entry() eventually check for
attachment by way of intel_gvt_dma_map_guest_page().
And of the paths that lead to ppgtt_populate_shadow_entry(),
shadow_ppgtt_mm() is the only one that doesn't already check for
INTEL_VGPU_STATUS_ACTIVE or INTEL_VGPU_STATUS_ATTACHED.
workload_thread() <= pick_next_workload() => INTEL_VGPU_STATUS_ACTIVE
|
-> dispatch_workload()
|
|-> prepare_workload()
|
-> intel_vgpu_sync_oos_pages()
| |
| |-> ppgtt_set_guest_page_sync()
| |
| |-> sync_oos_page()
| |
| |-> ppgtt_populate_shadow_entry()
|
|-> intel_vgpu_flush_post_shadow()
|
1: |-> ppgtt_handle_guest_write_page_table()
|
|-> ppgtt_handle_guest_entry_add()
|
2: | -> ppgtt_populate_spt_by_guest_entry()
||
||-> ppgtt_populate_spt()
||
||-> ppgtt_populate_shadow_entry()
||
||-> ppgtt_populate_spt_by_guest_entry() [see
2]
|
|-> ppgtt_populate_shadow_entry()
kvmgt_page_track_write() <= KVM callback => INTEL_VGPU_STATUS_ATTACHED
|
|-> intel_vgpu_page_track_handler()
|
|-> ppgtt_write_protection_handler()
|
|-> ppgtt_handle_guest_write_page_table_bytes()
|
|-> ppgtt_handle_guest_write_page_table() [see 1]
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 5426a27c1b71..2aed31b497c9 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1163,8 +1163,6 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return -EINVAL;
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
@@ -1277,6 +1275,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
if (!pte_ops->test_present(ge))
return 0;
+ if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
+ return -EINVAL;
+
gfn = pte_ops->get_pfn(ge);
switch (ge->type) {
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 04/29] drm/i915/gvt: Don't try to unpin an empty page range
From: Yan Zhao Attempt to unpin pages in the error path of gvt_pin_guest_page() if and only if at least one page was successfully pinned. Unpinning doesn't cause functional problems, but vfio_device_container_unpin_pages() rightfully warns about being asked to unpin zero pages. Signed-off-by: Yan Zhao [sean: write changelog] Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/kvmgt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index 429f0f993a13..0366a699baf5 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -172,7 +172,8 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn, *page = base_page; return 0; err: - gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); + if (npage) + gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); return ret; } -- 2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 03/29] drm/i915/gvt: Verify hugepages are contiguous in physical address space
When shadowing a GTT entry with a 2M page, verify that the pfns are
contiguous, not just that the struct page pointers are contiguous. The
memory map is virtual contiguous if "CONFIG_FLATMEM=y ||
CONFIG_SPARSEMEM_VMEMMAP=y", but not for "CONFIG_SPARSEMEM=y &&
CONFIG_SPARSEMEM_VMEMMAP=n", so theoretically KVMGT could encounter struct
pages that are virtually contiguous, but not physically contiguous.
In practice, this flaw is likely a non-issue as it would cause functional
problems iff a section isn't 2M aligned _and_ is directly adjacent to
another section with discontiguous pfns.
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index de675d799c7d..429f0f993a13 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -161,7 +161,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu,
unsigned long gfn,
if (npage == 0)
base_page = cur_page;
- else if (base_page + npage != cur_page) {
+ else if (page_to_pfn(base_page) + npage !=
page_to_pfn(cur_page)) {
gvt_vgpu_err("The pages are not continuous\n");
ret = -EINVAL;
npage++;
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 01/29] drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
Check that the pfn found by gfn_to_pfn() is actually backed by "struct
page" memory prior to retrieving and dereferencing the page. KVM
supports backing guest memory with VM_PFNMAP, VM_IO, etc., and so
there is no guarantee the pfn returned by gfn_to_pfn() has an associated
"struct page".
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 4
1 file changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 4ec85308379a..58b9b316ae46 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1183,6 +1183,10 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
+
+ if (!pfn_valid(pfn))
+ return -EINVAL;
+
return PageTransHuge(pfn_to_page(pfn));
}
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 02/29] drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
From: Yan Zhao
Currently intel_gvt_is_valid_gfn() is called in two places:
(1) shadowing guest GGTT entry
(2) shadowing guest PPGTT leaf entry,
which was introduced in commit cc753fbe1ac4
("drm/i915/gvt: validate gfn before set shadow page entry").
However, now it's not necessary to call this interface any more, because
a. GGTT partial write issue has been fixed by
commit bc0686ff5fad
("drm/i915/gvt: support inconsecutive partial gtt entry write")
commit 510fe10b6180
("drm/i915/gvt: fix a bug of partially write ggtt enties")
b. PPGTT resides in normal guest RAM and we only treat 8-byte writes
as valid page table writes. Any invalid GPA found is regarded as
an error, either due to guest misbehavior/attack or bug in host
shadow code.
So,rather than do GFN pre-checking and replace invalid GFNs with
scratch GFN and continue silently, just remove the pre-checking and
abort PPGTT shadowing on error detected.
c. GFN validity check is still performed in
intel_gvt_dma_map_guest_page() --> gvt_pin_guest_page().
It's more desirable to call VFIO interface to do both validity check
and mapping.
Calling intel_gvt_is_valid_gfn() to do GFN validity check from KVM side
while later mapping the GFN through VFIO interface is unnecessarily
fragile and confusing for unaware readers.
Signed-off-by: Yan Zhao
[sean: remove now-unused local variables]
Acked-by: Zhi Wang
Tested-by: Yongwei Ma
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 36 +-
1 file changed, 1 insertion(+), 35 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 58b9b316ae46..f30922c55a0c 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -49,22 +49,6 @@
static bool enable_out_of_sync = false;
static int preallocated_oos_pages = 8192;
-static bool intel_gvt_is_valid_gfn(struct intel_vgpu *vgpu, unsigned long gfn)
-{
- struct kvm *kvm = vgpu->vfio_device.kvm;
- int idx;
- bool ret;
-
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return false;
-
- idx = srcu_read_lock(>srcu);
- ret = kvm_is_visible_gfn(kvm, gfn);
- srcu_read_unlock(>srcu, idx);
-
- return ret;
-}
-
/*
* validate a gm address and related range size,
* translate it to host gm address
@@ -1333,11 +1317,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt)
{
struct intel_vgpu *vgpu = spt->vgpu;
- struct intel_gvt *gvt = vgpu->gvt;
- const struct intel_gvt_gtt_pte_ops *ops = gvt->gtt.pte_ops;
struct intel_vgpu_ppgtt_spt *s;
struct intel_gvt_gtt_entry se, ge;
- unsigned long gfn, i;
+ unsigned long i;
int ret;
trace_spt_change(spt->vgpu->id, "born", spt,
@@ -1354,13 +1336,6 @@ static int ppgtt_populate_spt(struct
intel_vgpu_ppgtt_spt *spt)
ppgtt_generate_shadow_entry(, s, );
ppgtt_set_shadow_entry(spt, , i);
} else {
- gfn = ops->get_pfn();
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(, gvt->gtt.scratch_mfn);
- ppgtt_set_shadow_entry(spt, , i);
- continue;
- }
-
ret = ppgtt_populate_shadow_entry(vgpu, spt, i, );
if (ret)
goto fail;
@@ -2335,14 +2310,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu
*vgpu, unsigned int off,
m.val64 = e.val64;
m.type = e.type;
- /* one PTE update may be issued in multiple writes and the
-* first write may not construct a valid gfn
-*/
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(, gvt->gtt.scratch_mfn);
- goto out;
- }
-
ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE,
_addr);
if (ret) {
@@ -2359,7 +2326,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu
*vgpu, unsigned int off,
ops->clear_present();
}
-out:
ggtt_set_guest_entry(ggtt_mm, , g_gtt_index);
ggtt_get_host_entry(ggtt_mm, , g_gtt_index);
--
2.41.0.487.g6d72f3e995-goog
[Intel-gfx] [PATCH v4 00/29] drm/i915/gvt: KVM: KVMGT fixes and page-track cleanups
Fix a handful of minor bugs in KVMGT, and overhaul KVM's page-track APIs
to provide a leaner and cleaner interface. The motivation for this
series is to (significantly) reduce the number of KVM APIs that KVMGT
uses, with a long-term goal of making all kvm_host.h headers KVM-internal.
If there are no objections or issues, my plan is to take this through the
KVM tree for 6.6 (I had it ready early last week, and then forgot to actually
post v4, /facepalm).
Thanks much for all the help!
v4:
- Collect tags. [Yongwei, Zhi, Yan]
- Add a patch to fix a benign (other than a WARN) bug where KVMGT would
attempt to unpin an empty range. [Yan]
- Move the check for an attached vGPU all the way up to shadow_ppgtt_mm().
[Zhi]
v3:
- https://lore.kernel.org/all/20230513003600.818142-1-sea...@google.com
- Collect reviewed/tested tags (I apologize if I missed any, I manually
gathered them this time due to a goof in my workflow). [Yan]
- Drop check on max KVM paging size from KVMGT. [Yan]
- Drop the explicit change on THP pages, and instead validate that the
pfns (not struct page pointers) are contiguous. [Yan]
- Fix buggy intel_gvt_dma_map_guest_page() usage by eliminating a helper
for shadowing 2MiB GTT entries. [Yan]
- Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c instead of exposing
kvm_mmu_zap_all_fast() outside of mmu.c. [Yan]
- Fix an alignment goof in hlist_for_each_entry_srcu() usage. [Yan]
- Wrap full definition of external page track structures with
CONFIG_KVM_EXTERNAL_WRITE_TRACKING. [Yan]
v2:
- https://lore.kernel.org/all/20230311002258.852397-1-sea...@google.com
- Reuse vgpu_lock to protect gfn hash instead of introducing a new (and
buggy) mutext. [Yan]
- Remove a spurious return from kvm_page_track_init(). [Yan]
- Take @kvm directly in the inner __kvm_page_track_write(). [Yan]
- Delete the gfn sanity check that relies on kvm_is_visible_gfn() instead
of providing a dedicated interface. [Yan]
v1: https://lore.kernel.org/lkml/20221223005739.1295925-1-sea...@google.com
Sean Christopherson (24):
drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
drm/i915/gvt: Verify hugepages are contiguous in physical address
space
drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT
entry type
drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M
GTT
drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
drm/i915/gvt: Protect gfn hash table with vgpu_lock
KVM: x86/mmu: Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
KVM: x86: Reject memslot MOVE operations if KVMGT is attached
drm/i915/gvt: Don't bother removing write-protection on to-be-deleted
slot
KVM: x86/mmu: Move KVM-only page-track declarations to internal header
KVM: x86/mmu: Use page-track notifiers iff there are external users
KVM: x86/mmu: Drop infrastructure for multiple page-track modes
KVM: x86/mmu: Rename page-track APIs to reflect the new reality
KVM: x86/mmu: Assert that correct locks are held for page
write-tracking
KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
drm/i915/gvt: Drop final dependencies on KVM internal details
Yan Zhao (5):
drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
drm/i915/gvt: Don't try to unpin an empty page range
KVM: x86: Add a new page-track hook to handle memslot deletion
drm/i915/gvt: switch from ->track_flush_slot() to
->track_remove_region()
KVM: x86: Remove the unused page-track hook track_flush_slot()
arch/x86/include/asm/kvm_host.h | 16 +-
arch/x86/include/asm/kvm_page_track.h | 73 +++-
arch/x86/kvm/mmu.h| 2 +
arch/x86/kvm/mmu/mmu.c| 51 +++--
arch/x86/kvm/mmu/page_track.c | 256 +-
arch/x86/kvm/mmu/page_track.h | 58 ++
arch/x86/kvm/x86.c| 22 +--
drivers/gpu/drm/i915/gvt/gtt.c| 102 ++
drivers/gpu/drm/i915/gvt/gtt.h| 1 -
drivers/gpu/drm/i915/gvt/gvt.h| 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 120 +---
drivers/gpu/drm/i915/gvt/page_track.c | 10 +-
12 files changed, 322 insertions(+), 392 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
base-commit: fdf0eaf11452d72945af31804e2a1048ee1b574c
--
2.41.0.487.g6d72f3e995-goog
Re: [Intel-gfx] [PATCH 2/2] eventfd: simplify eventfd_signal_mask()
On Thu, Jul 13, 2023, Christian Brauner wrote:
> diff --git a/fs/eventfd.c b/fs/eventfd.c
> index dc9e01053235..077be5da72bd 100644
> --- a/fs/eventfd.c
> +++ b/fs/eventfd.c
> @@ -43,9 +43,10 @@ struct eventfd_ctx {
> int id;
> };
>
> -__u64 eventfd_signal_mask(struct eventfd_ctx *ctx, __u64 n, __poll_t mask)
> +bool eventfd_signal_mask(struct eventfd_ctx *ctx, __poll_t mask)
> {
> unsigned long flags;
> + __u64 n = 1;
>
> /*
>* Deadlock or stack overflow issues can happen if we recurse here
> @@ -68,7 +69,7 @@ __u64 eventfd_signal_mask(struct eventfd_ctx *ctx, __u64 n,
> __poll_t mask)
> current->in_eventfd = 0;
> spin_unlock_irqrestore(>wqh.lock, flags);
>
> - return n;
> + return n == 1;
> }
...
> @@ -58,13 +58,12 @@ static inline struct eventfd_ctx *eventfd_ctx_fdget(int
> fd)
> return ERR_PTR(-ENOSYS);
> }
>
> -static inline int eventfd_signal(struct eventfd_ctx *ctx)
> +static inline bool eventfd_signal(struct eventfd_ctx *ctx)
> {
> return -ENOSYS;
> }
>
> -static inline int eventfd_signal_mask(struct eventfd_ctx *ctx, __u64 n,
> - unsigned mask)
> +static inline bool eventfd_signal_mask(struct eventfd_ctx *ctx, unsigned
> mask)
> {
> return -ENOSYS;
This will morph to "true" for what should be an error case. One option would be
to have eventfd_signal_mask() return 0/-errno instead of the count, but looking
at all the callers, nothing ever actually consumes the result.
KVMGT morphs failure into -EFAULT
if (vgpu->msi_trigger && eventfd_signal(vgpu->msi_trigger, 1) != 1)
return -EFAULT;
but the only caller of that user ignores the return value.
if (vgpu_vreg(vgpu, i915_mmio_reg_offset(GEN8_MASTER_IRQ))
& ~GEN8_MASTER_IRQ_CONTROL)
inject_virtual_interrupt(vgpu);
The sample driver in samples/vfio-mdev/mtty.c uses a similar pattern: prints an
error but otherwise ignores the result.
So why not return nothing? That will simplify eventfd_signal_mask() a wee bit
more, and eliminate that bizarre return value confusion for the ugly stubs, e.g.
void eventfd_signal_mask(struct eventfd_ctx *ctx, unsigned mask)
{
unsigned long flags;
/*
* Deadlock or stack overflow issues can happen if we recurse here
* through waitqueue wakeup handlers. If the caller users potentially
* nested waitqueues with custom wakeup handlers, then it should
* check eventfd_signal_allowed() before calling this function. If
* it returns false, the eventfd_signal() call should be deferred to a
* safe context.
*/
if (WARN_ON_ONCE(current->in_eventfd))
return;
spin_lock_irqsave(>wqh.lock, flags);
current->in_eventfd = 1;
if (ctx->count < ULLONG_MAX)
ctx->count++;
if (waitqueue_active(>wqh))
wake_up_locked_poll(>wqh, EPOLLIN | mask);
current->in_eventfd = 0;
spin_unlock_irqrestore(>wqh.lock, flags);
}
You could even go further and unify the real and stub versions of
eventfd_signal().
Re: [Intel-gfx] [PATCH v3 03/28] drm/i915/gvt: Verify hugepages are contiguous in physical address space
On Thu, May 18, 2023, Yan Zhao wrote: > On Wed, May 17, 2023 at 07:50:26AM -0700, Sean Christopherson wrote: > > On Tue, May 16, 2023, Yan Zhao wrote: > > > hi Sean > > > > > > Do you think it's necessary to double check that struct page pointers > > > are also contiguous? > > > > No, the virtual address space should be irrelevant. The only way it would > > be > > problematic is if something in dma_map_page() expected to be able to access > > the > > entire chunk of memory by getting the virtual address of only the first > > page, > > but I can't imagine that code is reading or writing memory, let alone doing > > so > > across a huge range of memory. > Yes, I do find arm_iommu version of dma_map_page() access the memory by > getting > virtual address of pages passed in, but it's implemented as page by page, not > only > from the first page. > > dma_map_page > dma_map_page_attrs > ops->map_page > arm_iommu_map_page Heh, thankfully this is ARM specific, which presumably doesn't collide with KVMGT. > __dma_page_cpu_to_dev >dma_cache_maint_page > > Just a little worried about the condition of PFNs are contiguous > while they belong to different backends, e.g. one from system memory and > one from MMIO. > But I don't know how to avoid this without complicated checks. > And this condition might not happen in practice. IMO, assuming that contiguous pfns are vritually contiguous is wrong, i.e. would be a bug in the other code. The above dma_cache_maint_page() get's this right, and even has a well written comment to boot.
Re: [Intel-gfx] [PATCH v3 03/28] drm/i915/gvt: Verify hugepages are contiguous in physical address space
On Tue, May 16, 2023, Yan Zhao wrote: > hi Sean > > Do you think it's necessary to double check that struct page pointers > are also contiguous? No, the virtual address space should be irrelevant. The only way it would be problematic is if something in dma_map_page() expected to be able to access the entire chunk of memory by getting the virtual address of only the first page, but I can't imagine that code is reading or writing memory, let alone doing so across a huge range of memory. > And do you like to also include a fix as below, which is to remove the > warning in vfio_device_container_unpin_pages() when npage is 0? > > @ -169,7 +173,8 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, > unsigned long gfn, > *page = base_page; > return 0; > err: > - gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); > + if (npage) > + gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); > return ret; > } Sure. Want to give your SoB? I'll write a changelog. Thanks again!
Re: [Intel-gfx] [PATCH v3 05/28] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
On Mon, May 15, 2023, Zhi A Wang wrote: > On 5/13/2023 8:35 AM, Sean Christopherson wrote: > > Move the check that a vGPU is attacked from is_2MB_gtt_possible() to its > > sole caller, ppgtt_populate_shadow_entry(). All of the paths in > > ppgtt_populate_shadow_entry() eventually check for attachment by way of > > intel_gvt_dma_map_guest_page(), but explicitly checking can avoid > > unnecessary work and will make it more obvious that a future cleanup of > > is_2MB_gtt_possible() isn't introducing a bug. > > > > It might be better move this check to shadow_ppgtt_mm() which is used > in both shadow page table creation and pinning path so that the path > can bail out even earlier when creating a shadow page table but a vGPU > has not been attached to KVM yet. Ah, yes, that'll work. I traced through all of the paths that lead to ppgtt_populate_shadow_entry(), and shadow_ppgtt_mm() is the only caller that isn't already gated by INTEL_VGPU_STATUS_ATTACHED or INTEL_VGPU_STATUS_ACTIVE (ACTIVE is set iff ATTACHED is set). I'll move the check up to shadow_ppgtt_mm() in the next version. Thanks! workload_thread() <= pick_next_workload() => INTEL_VGPU_STATUS_ACTIVE | -> dispatch_workload() | |-> prepare_workload() | -> intel_vgpu_sync_oos_pages() | | | |-> ppgtt_set_guest_page_sync() | | | |-> sync_oos_page() | | | |-> ppgtt_populate_shadow_entry() | |-> intel_vgpu_flush_post_shadow() | 1: |-> ppgtt_handle_guest_write_page_table() | |-> ppgtt_handle_guest_entry_add() | 2: | -> ppgtt_populate_spt_by_guest_entry() || ||-> ppgtt_populate_spt() || ||-> ppgtt_populate_shadow_entry() || ||-> ppgtt_populate_spt_by_guest_entry() [see 2] | |-> ppgtt_populate_shadow_entry() kvmgt_page_track_write() <= KVM callback => INTEL_VGPU_STATUS_ATTACHED | |-> intel_vgpu_page_track_handler() | |-> ppgtt_write_protection_handler() | |-> ppgtt_handle_guest_write_page_table_bytes() | |-> ppgtt_handle_guest_write_page_table() [see 1]
[Intel-gfx] [PATCH v3 27/28] KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
Get/put references to KVM when a page-track notifier is (un)registered
instead of relying on the caller to do so. Forcing the caller to do the
bookkeeping is unnecessary and adds one more thing for users to get
wrong, e.g. see commit 9ed1fdee9ee3 ("drm/i915/gvt: Get reference to KVM
iff attachment to VM is successful").
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 11 +--
arch/x86/kvm/mmu/page_track.c | 18 --
drivers/gpu/drm/i915/gvt/kvmgt.c | 17 +++--
3 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 4afab697e21c..3d040741044b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -44,12 +44,11 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n);
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n);
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n);
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n);
+
int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a64df38ccab..fd04e618ad2d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -157,17 +157,22 @@ int kvm_page_track_init(struct kvm *kvm)
* register the notifier so that event interception for the tracked guest
* pages can be received.
*/
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n)
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
+ if (!kvm || kvm->mm != current->mm)
+ return -ESRCH;
+
+ kvm_get_kvm(kvm);
+
head = >arch.track_notifier_head;
write_lock(>mmu_lock);
hlist_add_head_rcu(>node, >track_notifier_list);
write_unlock(>mmu_lock);
+ return 0;
}
EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
@@ -175,9 +180,8 @@ EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
* stop receiving the event interception. It is the opposed operation of
* kvm_page_track_register_notifier().
*/
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n)
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
@@ -187,6 +191,8 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
hlist_del_rcu(>node);
write_unlock(>mmu_lock);
synchronize_srcu(>track_srcu);
+
+ kvm_put_kvm(kvm);
}
EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index b995d75a19c3..597ffc9d12fd 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -653,21 +653,19 @@ static bool __kvmgt_vgpu_exist(struct intel_vgpu *vgpu)
static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
{
struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
-
- if (!vgpu->vfio_device.kvm ||
- vgpu->vfio_device.kvm->mm != current->mm) {
- gvt_vgpu_err("KVM is required to use Intel vGPU\n");
- return -ESRCH;
- }
+ int ret;
if (__kvmgt_vgpu_exist(vgpu))
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
- kvm_get_kvm(vgpu->vfio_device.kvm);
- kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
->track_node);
+ ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
+ >track_node);
+ if (ret) {
+ gvt_vgpu_err("KVM is required to use Intel vGPU\n");
+ return ret;
+ }
set_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status);
@@ -702,7 +700,6 @@ static void intel_vgpu_close_device(struct vfio_device
*vfio_dev)
kvm_page_track_u
[Intel-gfx] [PATCH v3 25/28] KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
Bug the VM if something attempts to write-track a gfn, but write-tracking isn't enabled. The VM is doomed (and KVM has an egregious bug) if KVM or KVMGT wants to shadow guest page tables but can't because write-tracking isn't enabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/page_track.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index 29ae61f1e303..eedb5889d73e 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -90,7 +90,7 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, 1); @@ -122,7 +122,7 @@ void kvm_write_track_remove_gfn(struct kvm *kvm, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, -1); -- 2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 22/28] KVM: x86/mmu: Drop infrastructure for multiple page-track modes
Drop "support" for multiple page-track modes, as there is no evidence
that array-based and refcounted metadata is the optimal solution for
other modes, nor is there any evidence that other use cases, e.g. for
access-tracking, will be a good fit for the page-track machinery in
general.
E.g. one potential use case of access-tracking would be to prevent guest
access to poisoned memory (from the guest's perspective). In that case,
the number of poisoned pages is likely to be a very small percentage of
the guest memory, and there is no need to reference count the number of
access-tracking users, i.e. expanding gfn_track[] for a new mode would be
grossly inefficient. And for poisoned memory, host userspace would also
likely want to trap accesses, e.g. to inject #MC into the guest, and that
isn't currently supported by the page-track framework.
A better alternative for that poisoned page use case is likely a
variation of the proposed per-gfn attributes overlay (linked), which
would allow efficiently tracking the sparse set of poisoned pages, and by
default would exit to userspace on access.
Link: https://lore.kernel.org/all/y2wb48kd0j4vg...@google.com
Cc: Ben Gardon
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 12 +--
arch/x86/include/asm/kvm_page_track.h | 11 +--
arch/x86/kvm/mmu/mmu.c| 14 ++--
arch/x86/kvm/mmu/page_track.c | 111 --
arch/x86/kvm/mmu/page_track.h | 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 +-
6 files changed, 51 insertions(+), 104 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 5ce06a75d3de..3dde3a3a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -288,13 +288,13 @@ struct kvm_kernel_irq_routing_entry;
* kvm_mmu_page_role tracks the properties of a shadow page (where shadow page
* also includes TDP pages) to determine whether or not a page can be used in
* the given MMU context. This is a subset of the overall kvm_cpu_role to
- * minimize the size of kvm_memory_slot.arch.gfn_track, i.e. allows allocating
- * 2 bytes per gfn instead of 4 bytes per gfn.
+ * minimize the size of kvm_memory_slot.arch.gfn_write_track, i.e. allows
+ * allocating 2 bytes per gfn instead of 4 bytes per gfn.
*
* Upper-level shadow pages having gptes are tracked for write-protection via
- * gfn_track. As above, gfn_track is a 16 bit counter, so KVM must not create
- * more than 2^16-1 upper-level shadow pages at a single gfn, otherwise
- * gfn_track will overflow and explosions will ensure.
+ * gfn_write_track. As above, gfn_write_track is a 16 bit counter, so KVM must
+ * not create more than 2^16-1 upper-level shadow pages at a single gfn,
+ * otherwise gfn_write_track will overflow and explosions will ensue.
*
* A unique shadow page (SP) for a gfn is created if and only if an existing SP
* cannot be reused. The ability to reuse a SP is tracked by its role, which
@@ -1005,7 +1005,7 @@ struct kvm_lpage_info {
struct kvm_arch_memory_slot {
struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES];
struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1];
- unsigned short *gfn_track[KVM_PAGE_TRACK_MAX];
+ unsigned short *gfn_write_track;
};
/*
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 61adb07b5927..9e4ee26d1779 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,17 +4,10 @@
#include
-enum kvm_page_track_mode {
- KVM_PAGE_TRACK_WRITE,
- KVM_PAGE_TRACK_MAX,
-};
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
+ struct kvm_memory_slot *slot, gfn_t gfn);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn,
-enum kvm_page_track_mode mode);
+struct kvm_memory_slot *slot, gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 0d9fe54ecb01..8041f5747704 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -837,8 +837,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
+ return kvm_slot_page_track_add_page(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -884,8 +883,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
[Intel-gfx] [PATCH v3 28/28] drm/i915/gvt: Drop final dependencies on KVM internal details
Open code gpa_to_gfn() in kvmgt_page_track_write() and drop KVMGT's dependency on kvm_host.h, i.e. include only kvm_page_track.h. KVMGT assumes "gfn == gpa >> PAGE_SHIFT" all over the place, including a few lines below in the same function with the same gpa, i.e. there's no reason to use KVM's helper for this one case. No functional change intended. Reviewed-by: Yan Zhao Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/gvt.h | 3 ++- drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index 2d65800d8e93..53a0a42a50db 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -34,10 +34,11 @@ #define _GVT_H_ #include -#include #include #include +#include + #include "i915_drv.h" #include "intel_gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index 597ffc9d12fd..191b4484cc7e 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -1584,7 +1584,7 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len, mutex_lock(>vgpu_lock); - if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa))) + if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT)) intel_vgpu_page_track_handler(info, gpa, (void *)val, len); -- 2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 26/28] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
Refactor KVM's exported/external page-track, a.k.a. write-track, APIs
to take only the gfn and do the required memslot lookup in KVM proper.
Forcing users of the APIs to get the memslot unnecessarily bleeds
KVM internals into KVMGT and complicates usage of the APIs.
No functional change intended.
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 7 +--
arch/x86/kvm/mmu/mmu.c| 4 +-
arch/x86/kvm/mmu/page_track.c | 85 ---
arch/x86/kvm/mmu/page_track.h | 5 ++
drivers/gpu/drm/i915/gvt/kvmgt.c | 37 +++-
5 files changed, 80 insertions(+), 58 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index f5c1db36cdb7..4afab697e21c 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,11 +4,6 @@
#include
-void kvm_write_track_add_gfn(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn);
-
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
@@ -55,6 +50,8 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
+int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
/*
* Allow defining a node in a structure even if page tracking is disabled, e.g.
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 1818c047891f..22f13963c320 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -837,7 +837,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_add_gfn(kvm, slot, gfn);
+ return __kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -883,7 +883,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_remove_gfn(kvm, slot, gfn);
+ return __kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index eedb5889d73e..2a64df38ccab 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -74,16 +74,8 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
slot->arch.gfn_write_track[index] += count;
}
-/*
- * add guest page to the tracking pool so that corresponding access on that
- * page will be intercepted.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-gfn_t gfn)
+void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -104,18 +96,9 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct
kvm_memory_slot *slot,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
-/*
- * remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_remove_gfn(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void __kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -133,7 +116,6 @@ void kvm_write_track_remove_gfn(struct kvm *kvm,
*/
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
/*
* check if the corresponding access on the specified guest page is tracked.
@@ -257,4 +239,63 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
srcu_read_unlock(>track_srcu, idx);
}
+/*
+ * add guest page to the tracking pool so that corresponding access on that
+ * page will be intercepted.
+ *
+ * @kvm: the guest instance we are interested in.
+ * @gfn: the guest page.
+ */
+int kvm_wri
[Intel-gfx] [PATCH v3 21/28] KVM: x86/mmu: Use page-track notifiers iff there are external users
Disable the page-track notifier code at compile time if there are no
external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
now hooks emulated writes directly instead of relying on the page-track
mechanism.
Provide a stub for "struct kvm_page_track_notifier_node" so that including
headers directly from the command line, e.g. for testing include guards,
doesn't fail due to a struct having an incomplete type.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 2 ++
arch/x86/include/asm/kvm_page_track.h | 22 +---
arch/x86/kvm/mmu/page_track.c | 10 -
arch/x86/kvm/mmu/page_track.h | 29 +++
4 files changed, 47 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 113598d3e886..5ce06a75d3de 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1247,7 +1247,9 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
struct kvm_page_track_notifier_head track_notifier_head;
+#endif
/*
* Protects marking pages unsync during page faults, as TDP MMU page
* faults only take mmu_lock for read. For simplicity, the unsync
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 76c0070dfe2a..61adb07b5927 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -9,6 +9,14 @@ enum kvm_page_track_mode {
KVM_PAGE_TRACK_MAX,
};
+void kvm_slot_page_track_add_page(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn,
+ enum kvm_page_track_mode mode);
+void kvm_slot_page_track_remove_page(struct kvm *kvm,
+struct kvm_memory_slot *slot, gfn_t gfn,
+enum kvm_page_track_mode mode);
+
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
* the head which will be notified when guest is triggering the track event.
@@ -48,18 +56,18 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn,
-enum kvm_page_track_mode mode);
-
void
kvm_page_track_register_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+#else
+/*
+ * Allow defining a node in a structure even if page tracking is disabled, e.g.
+ * to play nice with testing headers via direct inclusion from the command
line.
+ */
+struct kvm_page_track_notifier_node {};
+#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e15329d48f95..b20aad7ac3fe 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -194,6 +194,7 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
return !!READ_ONCE(slot->arch.gfn_track[mode][index]);
}
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
void kvm_page_track_cleanup(struct kvm *kvm)
{
struct kvm_page_track_notifier_head *head;
@@ -255,14 +256,13 @@ EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
* The node should figure out if the written page is the one that node is
* interested in by itself.
*/
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes)
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int
bytes)
{
struct kvm_page_track_notifier_head *head;
struct kvm_page_track_notifier_node *n;
int idx;
- head = >kvm->arch.track_notifier_head;
+ head = >arch.track_notifier_head;
if (hlist_empty(>track_notifier_list))
return;
@@ -273,8 +273,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
-
- kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
@@ -299,3 +297,5 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
n->track_remove_r
[Intel-gfx] [PATCH v3 24/28] KVM: x86/mmu: Assert that correct locks are held for page write-tracking
When adding/removing gfns to/from write-tracking, assert that mmu_lock
is held for write, and that either slots_lock or kvm->srcu is held.
mmu_lock must be held for write to protect gfn_write_track's refcount,
and SRCU or slots_lock must be held to protect the memslot itself.
Tested-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/page_track.c | 17 +++--
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index b835ba7f325c..29ae61f1e303 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -12,6 +12,7 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include
#include
#include
@@ -77,9 +78,6 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* add guest page to the tracking pool so that corresponding access on that
* page will be intercepted.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -87,6 +85,11 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -107,9 +110,6 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
* remove the guest page from the tracking pool which stops the interception
* of corresponding access on that page.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -117,6 +117,11 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
void kvm_write_track_remove_gfn(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 20/28] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
Bury the declaration of the page-track helpers that are intended only for
internal KVM use in a "private" header. In addition to guarding against
unwanted usage of the internal-only helpers, dropping their definitions
avoids exposing other structures that should be KVM-internal, e.g. for
memslots. This is a baby step toward making kvm_host.h a KVM-internal
header in the very distant future.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 21 ++---
arch/x86/kvm/mmu/mmu.c| 3 ++-
arch/x86/kvm/mmu/page_track.c | 8 +--
arch/x86/kvm/mmu/page_track.h | 33 +++
arch/x86/kvm/x86.c| 1 +
5 files changed, 39 insertions(+), 27 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 5c348ffdc194..76c0070dfe2a 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -2,6 +2,8 @@
#ifndef _ASM_X86_KVM_PAGE_TRACK_H
#define _ASM_X86_KVM_PAGE_TRACK_H
+#include
+
enum kvm_page_track_mode {
KVM_PAGE_TRACK_WRITE,
KVM_PAGE_TRACK_MAX,
@@ -46,26 +48,12 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
-
-bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
-int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
-
-void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
-int kvm_page_track_create_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- unsigned long npages);
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot,
- gfn_t gfn, enum kvm_page_track_mode mode);
void
kvm_page_track_register_notifier(struct kvm *kvm,
@@ -73,10 +61,5 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
-void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
-
-bool kvm_page_track_has_external_user(struct kvm *kvm);
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 3f9030650c3d..0d9fe54ecb01 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -25,6 +25,7 @@
#include "kvm_cache_regs.h"
#include "smm.h"
#include "kvm_emulate.h"
+#include "page_track.h"
#include "cpuid.h"
#include "spte.h"
@@ -53,7 +54,7 @@
#include
#include
#include
-#include
+
#include "trace.h"
extern bool itlb_multihit_kvm_mitigation;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a6ab7c455c0..e15329d48f95 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -15,10 +15,9 @@
#include
#include
-#include
-
#include "mmu.h"
#include "mmu_internal.h"
+#include "page_track.h"
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm)
{
@@ -300,8 +299,3 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
n->track_remove_region(slot->base_gfn, slot->npages, n);
srcu_read_unlock(>track_srcu, idx);
}
-
-bool kvm_page_track_has_external_user(struct kvm *kvm)
-{
- return hlist_empty(>arch.track_notifier_head.track_notifier_list);
-}
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
new file mode 100644
index ..89712f123ad3
--- /dev/null
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_PAGE_TRACK_H
+#define __KVM_X86_PAGE_TRACK_H
+
+#include
+
+#include
+
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
+int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
+
+void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
+int kvm_page_track_create_memslot(struct kvm *kvm,
+ struct kvm
[Intel-gfx] [PATCH v3 23/28] KVM: x86/mmu: Rename page-track APIs to reflect the new reality
Rename the page-track APIs to capture that they're all about tracking
writes, now that the facade of supporting multiple modes is gone.
Opportunstically replace "slot" with "gfn" in anticipation of removing
the @slot param from the external APIs.
No functional change intended.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 8
arch/x86/kvm/mmu/mmu.c| 8
arch/x86/kvm/mmu/page_track.c | 21 +
arch/x86/kvm/mmu/page_track.h | 4 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 ++--
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 9e4ee26d1779..f5c1db36cdb7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,10 +4,10 @@
#include
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_add_gfn(struct kvm *kvm,
+struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 8041f5747704..1818c047891f 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -837,7 +837,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn);
+ return kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -883,7 +883,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_remove_page(kvm, slot, gfn);
+ return kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
@@ -2823,7 +2823,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct
kvm_memory_slot *slot,
* track machinery is used to write-protect upper-level shadow pages,
* i.e. this guards the role.level == 4K assertion below!
*/
- if (kvm_slot_page_track_is_active(kvm, slot, gfn))
+ if (kvm_gfn_is_write_tracked(kvm, slot, gfn))
return -EPERM;
/*
@@ -4224,7 +4224,7 @@ static bool page_fault_handle_page_track(struct kvm_vcpu
*vcpu,
* guest is writing the page which is write tracked which can
* not be fixed by page fault handler.
*/
- if (kvm_slot_page_track_is_active(vcpu->kvm, fault->slot, fault->gfn))
+ if (kvm_gfn_is_write_tracked(vcpu->kvm, fault->slot, fault->gfn))
return true;
return false;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index cdc6069b8caf..b835ba7f325c 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -84,10 +84,9 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+gfn_t gfn)
{
-
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -102,12 +101,11 @@ void kvm_slot_page_track_add_page(struct kvm *kvm,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
+EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
/*
* remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page. It is the opposed operation of
- * kvm_slot_page_track_add_page().
+ * of corresponding access on that page.
*
* It should be called under the protection both of mmu-lock and kvm->srcu
* or kvm->slots_lock.
@@ -116,8 +114,8 @@ EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_remove_gfn(struct kvm *kvm,
+ stru
[Intel-gfx] [PATCH v3 17/28] KVM: x86: Add a new page-track hook to handle memslot deletion
From: Yan Zhao
Add a new page-track hook, track_remove_region(), that is called when a
memslot DELETE operation is about to be committed. The "remove" hook
will be used by KVMGT and will effectively replace the existing
track_flush_slot() altogether now that KVM itself doesn't rely on the
"flush" hook either.
The "flush" hook is flawed as it's invoked before the memslot operation
is guaranteed to succeed, i.e. KVM might ultimately keep the existing
memslot without notifying external page track users, a.k.a. KVMGT. In
practice, this can't currently happen on x86, but there are no guarantees
that won't change in the future, not to mention that "flush" does a very
poor job of describing what is happening.
Pass in the gfn+nr_pages instead of the slot itself so external users,
i.e. KVMGT, don't need to exposed to KVM internals (memslots). This will
help set the stage for additional cleanups to the page-track APIs.
Opportunistically align the existing srcu_read_lock_held() usage so that
the new case doesn't stand out like a sore thumb (and not aligning the
new code makes bots unhappy).
Cc: Zhenyu Wang
Signed-off-by: Yan Zhao
Co-developed-by: Sean Christopherson
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 12
arch/x86/kvm/mmu/page_track.c | 27 +--
arch/x86/kvm/x86.c| 3 +++
3 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index f744682648e7..cfd36c22b467 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,6 +43,17 @@ struct kvm_page_track_notifier_node {
*/
void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
+
+ /*
+* Invoked when a memory region is removed from the guest. Or in KVM
+* terms, when a memslot is deleted.
+*
+* @gfn: base gfn of the region being removed
+* @nr_pages: number of pages in the to-be-removed region
+* @node: this node
+*/
+ void (*track_remove_region)(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node);
};
int kvm_page_track_init(struct kvm *kvm);
@@ -75,6 +86,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e6de9638e560..d971c28be99d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -270,7 +270,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
idx = srcu_read_lock(>track_srcu);
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
+ srcu_read_lock_held(>track_srcu))
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
@@ -298,12 +298,35 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
idx = srcu_read_lock(>track_srcu);
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
+ srcu_read_lock_held(>track_srcu))
if (n->track_flush_slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(>track_srcu, idx);
}
+/*
+ * Notify external page track nodes that a memory region is being removed from
+ * the VM, e.g. so that users can free any associated metadata.
+ */
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+ struct kvm_page_track_notifier_head *head;
+ struct kvm_page_track_notifier_node *n;
+ int idx;
+
+ head = >arch.track_notifier_head;
+
+ if (hlist_empty(>track_notifier_list))
+ return;
+
+ idx = srcu_read_lock(>track_srcu);
+ hlist_for_each_entry_srcu(n, >track_notifier_list, node,
+ srcu_read_lock_held(>track_srcu))
+ if (n->track_remove_region)
+ n->track_remove_region(slot->base_gfn, slot->npages, n);
+ srcu_read_unlock(>track_srcu, idx);
+}
+
bool kvm_page_track_has_external_user(struct kvm *kvm)
{
return hlist_
[Intel-gfx] [PATCH v3 15/28] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
Disallow moving memslots if the VM has external page-track users, i.e. if
KVMGT is being used to expose a virtual GPU to the guest, as KVMGT doesn't
correctly handle moving memory regions.
Note, this is potential ABI breakage! E.g. userspace could move regions
that aren't shadowed by KVMGT without harming the guest. However, the
only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
regions. KVM's own support for moving memory regions was also broken for
multiple years (albeit for an edge case, but arguably moving RAM is
itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
new rmap and large page tracking when moving memslot").
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 3 +++
arch/x86/kvm/mmu/page_track.c | 5 +
arch/x86/kvm/x86.c| 7 +++
3 files changed, 15 insertions(+)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 8c4d216e3b2b..f744682648e7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -75,4 +75,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+
+bool kvm_page_track_has_external_user(struct kvm *kvm);
+
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 891e5cc52b45..e6de9638e560 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -303,3 +303,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(>track_srcu, idx);
}
+
+bool kvm_page_track_has_external_user(struct kvm *kvm)
+{
+ return hlist_empty(>arch.track_notifier_head.track_notifier_list);
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index b2d9c5979df7..c6bbd8ffd8c8 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12588,6 +12588,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ /*
+* KVM doesn't support moving memslots when there are external page
+* trackers attached to the VM, i.e. if KVMGT is in use.
+*/
+ if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
+ return -EINVAL;
+
if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
return -EINVAL;
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 19/28] KVM: x86: Remove the unused page-track hook track_flush_slot()
From: Yan Zhao
Remove ->track_remove_slot(), there are no longer any users and it's
unlikely a "flush" hook will ever be the correct API to provide to an
external page-track user.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 11 ---
arch/x86/kvm/mmu/mmu.c| 2 --
arch/x86/kvm/mmu/page_track.c | 26 --
3 files changed, 39 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index cfd36c22b467..5c348ffdc194 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -33,16 +33,6 @@ struct kvm_page_track_notifier_node {
*/
void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
struct kvm_page_track_notifier_node *node);
- /*
-* It is called when memory slot is being moved or removed
-* users can drop write-protection for the pages in that memory slot
-*
-* @kvm: the kvm where memory slot being moved or removed
-* @slot: the memory slot being moved or removed
-* @node: this node
-*/
- void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
/*
* Invoked when a memory region is removed from the guest. Or in KVM
@@ -85,7 +75,6 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index af3e562d3106..3f9030650c3d 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6734,8 +6734,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
kvm_mmu_zap_all_fast(kvm);
-
- kvm_page_track_flush_slot(kvm, slot);
}
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index d971c28be99d..2a6ab7c455c0 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -278,32 +278,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t
gpa, const u8 *new,
kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
-/*
- * Notify the node that memory slot is being removed or moved so that it can
- * drop write-protection for the pages in the memory slot.
- *
- * The node should figure out it has any write-protected pages in this slot
- * by itself.
- */
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
-{
- struct kvm_page_track_notifier_head *head;
- struct kvm_page_track_notifier_node *n;
- int idx;
-
- head = >arch.track_notifier_head;
-
- if (hlist_empty(>track_notifier_list))
- return;
-
- idx = srcu_read_lock(>track_srcu);
- hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
- if (n->track_flush_slot)
- n->track_flush_slot(kvm, slot, n);
- srcu_read_unlock(>track_srcu, idx);
-}
-
/*
* Notify external page track nodes that a memory region is being removed from
* the VM, e.g. so that users can free any associated metadata.
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 18/28] drm/i915/gvt: switch from ->track_flush_slot() to ->track_remove_region()
From: Yan Zhao
Switch from the poorly named and flawed ->track_flush_slot() to the newly
introduced ->track_remove_region(). From KVMGT's perspective, the two
hooks are functionally equivalent, the only difference being that
->track_remove_region() is called only when KVM is 100% certain the
memory region will be removed, i.e. is invoked slightly later in KVM's
memslot modification flow.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
[sean: handle name change, massage changelog, rebase]
Tested-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 21 +
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index c309a77f495f..2e65901270ca 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -108,9 +108,8 @@ struct gvt_dma {
static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
struct kvm_page_track_notifier_node *node);
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node);
static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf)
{
@@ -665,7 +664,7 @@ static int intel_vgpu_open_device(struct vfio_device
*vfio_dev)
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
- vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
+ vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
kvm_get_kvm(vgpu->vfio_device.kvm);
kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
>track_node);
@@ -1616,22 +1615,20 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8
*val, int len,
mutex_unlock(>vgpu_lock);
}
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node)
{
unsigned long i;
- gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
mutex_lock(>vgpu_lock);
- for (i = 0; i < slot->npages; i++) {
- gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn))
- kvmgt_protect_table_del(info, gfn);
+ for (i = 0; i < nr_pages; i++) {
+ if (kvmgt_gfn_is_write_protected(info, gfn + i))
+ kvmgt_protect_table_del(info, gfn + i);
}
+
mutex_unlock(>vgpu_lock);
}
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 16/28] drm/i915/gvt: Don't bother removing write-protection on to-be-deleted slot
When handling a slot "flush", don't call back into KVM to drop write
protection for gfns in the slot. Now that KVM rejects attempts to move
memory slots while KVMGT is attached, the only time a slot is "flushed"
is when it's being removed, i.e. the memslot and all its write-tracking
metadata is about to be deleted.
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 8 +---
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index aaebb44c139f..c309a77f495f 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1629,14 +1629,8 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn)) {
- write_lock(>mmu_lock);
- kvm_slot_page_track_remove_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
- write_unlock(>mmu_lock);
-
+ if (kvmgt_gfn_is_write_protected(info, gfn))
kvmgt_protect_table_del(info, gfn);
- }
}
mutex_unlock(>vgpu_lock);
}
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 13/28] KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
Don't use the generic page-track mechanism to handle writes to guest PTEs
in KVM's MMU. KVM's MMU needs access to information that should not be
exposed to external page-track users, e.g. KVM needs (for some definitions
of "need") the vCPU to query the current paging mode, whereas external
users, i.e. KVMGT, have no ties to the current vCPU and so should never
need the vCPU.
Moving away from the page-track mechanism will allow dropping use of the
page-track mechanism for KVM's own MMU, and will also allow simplifying
and cleaning up the page-track APIs.
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu.h | 2 ++
arch/x86/kvm/mmu/mmu.c | 13 ++---
arch/x86/kvm/mmu/page_track.c | 2 ++
4 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 564a29153cee..113598d3e886 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1247,7 +1247,6 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
- struct kvm_page_track_notifier_node mmu_sp_tracker;
struct kvm_page_track_notifier_head track_notifier_head;
/*
* Protects marking pages unsync during page faults, as TDP MMU page
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 92d5a1924fc1..253fb2093d5d 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -121,6 +121,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes);
static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
{
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 23a79723031b..af3e562d3106 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5677,9 +5677,8 @@ static u64 *get_written_sptes(struct kvm_mmu_page *sp,
gpa_t gpa, int *nspte)
return spte;
}
-static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *new, int bytes,
- struct kvm_page_track_notifier_node *node)
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes)
{
gfn_t gfn = gpa >> PAGE_SHIFT;
struct kvm_mmu_page *sp;
@@ -6186,7 +6185,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
int kvm_mmu_init_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
int r;
INIT_LIST_HEAD(>arch.active_mmu_pages);
@@ -6200,9 +6198,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
return r;
}
- node->track_write = kvm_mmu_pte_write;
- kvm_page_track_register_notifier(kvm, node);
-
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6223,10 +6218,6 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
void kvm_mmu_uninit_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
-
- kvm_page_track_unregister_notifier(kvm, node);
-
if (tdp_mmu_enabled)
kvm_mmu_uninit_tdp_mmu(kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 0a2ac438d647..23088c90d2fd 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -274,6 +274,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(vcpu, gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
+
+ kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 14/28] KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
Drop @vcpu from KVM's ->track_write() hook provided for external users of
the page-track APIs now that KVM itself doesn't use the page-track
mechanism.
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 5 ++---
arch/x86/kvm/mmu/page_track.c | 2 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 10 --
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index eb186bc57f6a..8c4d216e3b2b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -26,14 +26,13 @@ struct kvm_page_track_notifier_node {
* It is called when guest is writing the write-tracked page
* and write emulation is finished at that time.
*
-* @vcpu: the vcpu where the write access happened.
* @gpa: the physical address written by guest.
* @new: the data was written to the address.
* @bytes: the written length.
* @node: this node
*/
- void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes, struct kvm_page_track_notifier_node
*node);
+ void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
+ struct kvm_page_track_notifier_node *node);
/*
* It is called when memory slot is being moved or removed
* users can drop write-protection for the pages in that memory slot
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 23088c90d2fd..891e5cc52b45 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -272,7 +272,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
srcu_read_lock_held(>track_srcu))
if (n->track_write)
- n->track_write(vcpu, gpa, new, bytes, n);
+ n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
kvm_mmu_track_write(vcpu, gpa, new, bytes);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 0785f9cb2c20..aaebb44c139f 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -106,9 +106,8 @@ struct gvt_dma {
#define vfio_dev_to_vgpu(vfio_dev) \
container_of((vfio_dev), struct intel_vgpu, vfio_device)
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node);
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
@@ -1602,9 +1601,8 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info,
u64 gfn)
return 0;
}
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node)
{
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 09/28] drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
Drop intel_vgpu_reset_gtt() as it no longer has any callers. In addition
to eliminating dead code, this eliminates the last possible scenario where
__kvmgt_protect_table_find() can be reached without holding vgpu_lock.
Requiring vgpu_lock to be held when calling __kvmgt_protect_table_find()
will allow a protecting the gfn hash with vgpu_lock without too much fuss.
No functional change intended.
Fixes: ba25d977571e ("drm/i915/gvt: Do not destroy ppgtt_mm during vGPU
D3->D0.")
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 18 --
drivers/gpu/drm/i915/gvt/gtt.h | 1 -
2 files changed, 19 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f505be9e647a..c3c623b929ce 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -2817,24 +2817,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool
invalidate_old)
ggtt_invalidate(gvt->gt);
}
-/**
- * intel_vgpu_reset_gtt - reset the all GTT related status
- * @vgpu: a vGPU
- *
- * This function is called from vfio core to reset reset all
- * GTT related status, including GGTT, PPGTT, scratch page.
- *
- */
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu)
-{
- /* Shadow pages are only created when there is no page
-* table tracking data, so remove page tracking data after
-* removing the shadow pages.
-*/
- intel_vgpu_destroy_all_ppgtt_mm(vgpu);
- intel_vgpu_reset_ggtt(vgpu, true);
-}
-
/**
* intel_gvt_restore_ggtt - restore all vGPU's ggtt entries
* @gvt: intel gvt device
diff --git a/drivers/gpu/drm/i915/gvt/gtt.h b/drivers/gpu/drm/i915/gvt/gtt.h
index a3b0f59ec8bd..4cb183e06e95 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.h
+++ b/drivers/gpu/drm/i915/gvt/gtt.h
@@ -224,7 +224,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool
invalidate_old);
void intel_vgpu_invalidate_ppgtt(struct intel_vgpu *vgpu);
int intel_gvt_init_gtt(struct intel_gvt *gvt);
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu);
void intel_gvt_clean_gtt(struct intel_gvt *gvt);
struct intel_vgpu_mm *intel_gvt_find_ppgtt_mm(struct intel_vgpu *vgpu,
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 12/28] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of
bouncing through the page-track mechanism. KVM (unfortunately) needs to
zap and flush all page tables on memslot DELETE/MOVE irrespective of
whether KVM is shadowing guest page tables.
This will allow changing KVM to register a page-track notifier on the
first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a
different method for reacting to memslot changes.
No functional change intended.
Cc: Yan Zhao
Link: https://lore.kernel.org/r/20221110014821.1548347-2-sea...@google.com
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/mmu.c | 10 ++
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 2e4476d38377..23a79723031b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6184,13 +6184,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm
*kvm)
return unlikely(!list_empty_careful(>arch.zapped_obsolete_pages));
}
-static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
-{
- kvm_mmu_zap_all_fast(kvm);
-}
-
int kvm_mmu_init_vm(struct kvm *kvm)
{
struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
@@ -6208,7 +6201,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
}
node->track_write = kvm_mmu_pte_write;
- node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
kvm_page_track_register_notifier(kvm, node);
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
@@ -6750,6 +6742,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
+ kvm_mmu_zap_all_fast(kvm);
+
kvm_page_track_flush_slot(kvm, slot);
}
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 10/28] drm/i915/gvt: Protect gfn hash table with vgpu_lock
Use vgpu_lock instead of KVM's mmu_lock to protect accesses to the hash
table used to track which gfns are write-protected when shadowing the
guest's GTT, and hoist the acquisition of vgpu_lock from
intel_vgpu_page_track_handler() out to its sole caller,
kvmgt_page_track_write().
This fixes a bug where kvmgt_page_track_write(), which doesn't hold
kvm->mmu_lock, could race with intel_gvt_page_track_remove() and trigger
a use-after-free.
Fixing kvmgt_page_track_write() by taking kvm->mmu_lock is not an option
as mmu_lock is a r/w spinlock, and intel_vgpu_page_track_handler() might
sleep when acquiring vgpu->cache_lock deep down the callstack:
intel_vgpu_page_track_handler()
|
|-> page_track->handler / ppgtt_write_protection_handler()
|
|-> ppgtt_handle_guest_write_page_table_bytes()
|
|-> ppgtt_handle_guest_write_page_table()
|
|-> ppgtt_handle_guest_entry_removal()
|
|-> ppgtt_invalidate_pte()
|
|-> intel_gvt_dma_unmap_guest_page()
|
|-> mutex_lock(>cache_lock);
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 55 +++
drivers/gpu/drm/i915/gvt/page_track.c | 10 +
2 files changed, 33 insertions(+), 32 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 117bac85ac2c..0785f9cb2c20 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -351,6 +351,8 @@ __kvmgt_protect_table_find(struct intel_vgpu *info, gfn_t
gfn)
{
struct kvmgt_pgfn *p, *res = NULL;
+ lockdep_assert_held(>vgpu_lock);
+
hash_for_each_possible(info->ptable, p, hnode, gfn) {
if (gfn == p->gfn) {
res = p;
@@ -1552,6 +1554,9 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64
gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
+ if (kvmgt_gfn_is_write_protected(info, gfn))
+ return 0;
+
idx = srcu_read_lock(>srcu);
slot = gfn_to_memslot(kvm, gfn);
if (!slot) {
@@ -1560,16 +1565,12 @@ int intel_gvt_page_track_add(struct intel_vgpu *info,
u64 gfn)
}
write_lock(>mmu_lock);
-
- if (kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
-
kvm_slot_page_track_add_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(>mmu_lock);
+
+ srcu_read_unlock(>srcu, idx);
+
kvmgt_protect_table_add(info, gfn);
-
-out:
- write_unlock(>mmu_lock);
- srcu_read_unlock(>srcu, idx);
return 0;
}
@@ -1582,24 +1583,22 @@ int intel_gvt_page_track_remove(struct intel_vgpu
*info, u64 gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
- idx = srcu_read_lock(>srcu);
- slot = gfn_to_memslot(kvm, gfn);
- if (!slot) {
- srcu_read_unlock(>srcu, idx);
- return -EINVAL;
- }
-
- write_lock(>mmu_lock);
-
if (!kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
+ return 0;
+ idx = srcu_read_lock(>srcu);
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot) {
+ srcu_read_unlock(>srcu, idx);
+ return -EINVAL;
+ }
+
+ write_lock(>mmu_lock);
kvm_slot_page_track_remove_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(>mmu_lock);
+ srcu_read_unlock(>srcu, idx);
+
kvmgt_protect_table_del(info, gfn);
-
-out:
- write_unlock(>mmu_lock);
- srcu_read_unlock(>srcu, idx);
return 0;
}
@@ -1610,9 +1609,13 @@ static void kvmgt_page_track_write(struct kvm_vcpu
*vcpu, gpa_t gpa,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
+ mutex_lock(>vgpu_lock);
+
if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
intel_vgpu_page_track_handler(info, gpa,
(void *)val, len);
+
+ mutex_unlock(>vgpu_lock);
}
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
@@ -1624,16 +1627,20 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
- write_lock(>mmu_lock);
+ mutex_lock(>vgpu_lock);
+
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
if (kvmgt_gfn_is_write_protected(info, gfn)) {
+ write_lock(>mmu_lock);
[Intel-gfx] [PATCH v3 11/28] KVM: x86/mmu: Move kvm_arch_flush_shadow_{all, memslot}() to mmu.c
Move x86's implementation of kvm_arch_flush_shadow_{all,memslot}() into
mmu.c, and make kvm_mmu_zap_all() static as it was globally visible only
for kvm_arch_flush_shadow_all(). This will allow refactoring
kvm_arch_flush_shadow_memslot() to call kvm_mmu_zap_all() directly without
having to expose kvm_mmu_zap_all_fast() outside of mmu.c. Keeping
everything in mmu.c will also likely simplify supporting TDX, which
intends to do zap only relevant SPTEs on memslot updates.
No functional change intended.
Suggested-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu/mmu.c | 13 -
arch/x86/kvm/x86.c | 11 ---
3 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index fb9d1f2d6136..564a29153cee 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1832,7 +1832,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
-void kvm_mmu_zap_all(struct kvm *kvm);
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c8961f45e3b1..2e4476d38377 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6717,7 +6717,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
*/
}
-void kvm_mmu_zap_all(struct kvm *kvm)
+static void kvm_mmu_zap_all(struct kvm *kvm)
{
struct kvm_mmu_page *sp, *node;
LIST_HEAD(invalid_list);
@@ -6742,6 +6742,17 @@ void kvm_mmu_zap_all(struct kvm *kvm)
write_unlock(>mmu_lock);
}
+void kvm_arch_flush_shadow_all(struct kvm *kvm)
+{
+ kvm_mmu_zap_all(kvm);
+}
+
+void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
+ struct kvm_memory_slot *slot)
+{
+ kvm_page_track_flush_slot(kvm, slot);
+}
+
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
{
WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 523c39a03c00..b2d9c5979df7 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12758,17 +12758,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
kvm_arch_free_memslot(kvm, old);
}
-void kvm_arch_flush_shadow_all(struct kvm *kvm)
-{
- kvm_mmu_zap_all(kvm);
-}
-
-void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot)
-{
- kvm_page_track_flush_slot(kvm, slot);
-}
-
static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)
{
return (is_guest_mode(vcpu) &&
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 07/28] drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M GTT
Now that gvt_pin_guest_page() explicitly verifies the pinned PFN is a
transparent hugepage page, don't use KVM's gfn_to_pfn() to pre-check if a
2MiB GTT entry is possible and instead just try to map the GFN with a 2MiB
entry. Using KVM to query pfn that is ultimately managed through VFIO is
odd, and KVM's gfn_to_pfn() is not intended for non-KVM consumption; it's
exported only because of KVM vendor modules (x86 and PPC).
Open code the check on 2MiB support instead of keeping
is_2MB_gtt_possible() around for a single line of code.
Move the call to intel_gvt_dma_map_guest_page() for a 4KiB entry into its
case statement, i.e. fork the common path into the 4KiB and 2MiB "direct"
shadow paths. Keeping the call in the "common" path is arguably more in
the spirit of "one change per patch", but retaining the local "page_size"
variable is silly, i.e. the call site will be changed either way, and
jumping around the no-longer-common code is more subtle and rather odd,
i.e. would just need to be immediately cleaned up.
Drop the error message from gvt_pin_guest_page() when KVMGT attempts to
shadow a 2MiB guest page that isn't backed by a compatible hugepage in the
host. Dropping the pre-check on a THP makes it much more likely that the
"error" will be encountered in normal operation.
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 49 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 1 -
2 files changed, 8 insertions(+), 42 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 61e38acee2d5..f505be9e647a 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1145,36 +1145,6 @@ static inline void ppgtt_generate_shadow_entry(struct
intel_gvt_gtt_entry *se,
ops->set_pfn(se, s->shadow_page.mfn);
}
-/*
- * Check if can do 2M page
- * @vgpu: target vgpu
- * @entry: target pfn's gtt entry
- *
- * Return 1 if 2MB huge gtt shadowing is possible, 0 if miscondition,
- * negative if found err.
- */
-static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
- struct intel_gvt_gtt_entry *entry)
-{
- const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
- kvm_pfn_t pfn;
- int ret;
-
- if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
- return 0;
-
- pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
- if (is_error_noslot_pfn(pfn))
- return -EINVAL;
-
- if (!pfn_valid(pfn))
- return -EINVAL;
-
- ret = PageTransHuge(pfn_to_page(pfn));
- kvm_release_pfn_clean(pfn);
- return ret;
-}
-
static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
struct intel_vgpu_ppgtt_spt *spt, unsigned long index,
struct intel_gvt_gtt_entry *se)
@@ -1268,7 +1238,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
{
const struct intel_gvt_gtt_pte_ops *pte_ops = vgpu->gvt->gtt.pte_ops;
struct intel_gvt_gtt_entry se = *ge;
- unsigned long gfn, page_size = PAGE_SIZE;
+ unsigned long gfn;
dma_addr_t dma_addr;
int ret;
@@ -1283,6 +1253,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
switch (ge->type) {
case GTT_TYPE_PPGTT_PTE_4K_ENTRY:
gvt_vdbg_mm("shadow 4K gtt entry\n");
+ ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE,
_addr);
+ if (ret)
+ return -ENXIO;
break;
case GTT_TYPE_PPGTT_PTE_64K_ENTRY:
gvt_vdbg_mm("shadow 64K gtt entry\n");
@@ -1294,12 +1267,10 @@ static int ppgtt_populate_shadow_entry(struct
intel_vgpu *vgpu,
return split_64KB_gtt_entry(vgpu, spt, index, );
case GTT_TYPE_PPGTT_PTE_2M_ENTRY:
gvt_vdbg_mm("shadow 2M gtt entry\n");
- ret = is_2MB_gtt_possible(vgpu, ge);
- if (ret == 0)
+ if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M)
||
+ intel_gvt_dma_map_guest_page(vgpu, gfn,
+I915_GTT_PAGE_SIZE_2M,
_addr))
return split_2MB_gtt_entry(vgpu, spt, index, );
- else if (ret < 0)
- return ret;
- page_size = I915_GTT_PAGE_SIZE_2M;
break;
case GTT_TYPE_PPGTT_PTE_1G_ENTRY:
gvt_vgpu_err("GVT doesn't support 1GB entry\n");
@@ -1309,11 +1280,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
return -EINVAL;
}
- /* direct shadow */
- ret = intel_gvt_dma_map_guest_page(vgpu, gfn, page_size, _addr);
- if (ret)
- return -ENXIO;
-
+ /* Successfully shadow
[Intel-gfx] [PATCH v3 08/28] drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
Use an "unsigned long" instead of an "int" when iterating over the gfns
in a memslot. The number of pages in the memslot is tracked as an
"unsigned long", e.g. KVMGT could theoretically break if a KVM memslot
larger than 16TiB were deleted (2^32 * 4KiB).
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 92ceefe1e6fb..117bac85ac2c 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1619,7 +1619,7 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node)
{
- int i;
+ unsigned long i;
gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 06/28] drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT entry type
Bail from ppgtt_populate_shadow_entry() if an unexpected GTT entry type is encountered instead of subtly falling through to the common "direct shadow" path. Eliminating the default/error path's reliance on the common handling will allow hoisting intel_gvt_dma_map_guest_page() into the case statements so that the 2MiB case can try intel_gvt_dma_map_guest_page() and fallback to splitting the entry on failure. Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/gtt.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 2aed31b497c9..61e38acee2d5 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1306,6 +1306,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, return -EINVAL; default: GEM_BUG_ON(1); + return -EINVAL; } /* direct shadow */ -- 2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 05/28] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
Move the check that a vGPU is attacked from is_2MB_gtt_possible() to its
sole caller, ppgtt_populate_shadow_entry(). All of the paths in
ppgtt_populate_shadow_entry() eventually check for attachment by way of
intel_gvt_dma_map_guest_page(), but explicitly checking can avoid
unnecessary work and will make it more obvious that a future cleanup of
is_2MB_gtt_possible() isn't introducing a bug.
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 5426a27c1b71..2aed31b497c9 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1163,8 +1163,6 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return -EINVAL;
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
@@ -1277,6 +1275,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
if (!pte_ops->test_present(ge))
return 0;
+ if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
+ return -EINVAL;
+
gfn = pte_ops->get_pfn(ge);
switch (ge->type) {
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 04/28] drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
Put the struct page reference acquired by gfn_to_pfn(), KVM's API is that
the caller is ultimately responsible for dropping any reference.
Note, kvm_release_pfn_clean() ensures the pfn is actually a refcounted
struct page before trying to put any references.
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f30922c55a0c..5426a27c1b71 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1158,6 +1158,7 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
{
const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
kvm_pfn_t pfn;
+ int ret;
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
@@ -1171,7 +1172,9 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!pfn_valid(pfn))
return -EINVAL;
- return PageTransHuge(pfn_to_page(pfn));
+ ret = PageTransHuge(pfn_to_page(pfn));
+ kvm_release_pfn_clean(pfn);
+ return ret;
}
static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 03/28] drm/i915/gvt: Verify hugepages are contiguous in physical address space
When shadowing a GTT entry with a 2M page, verify that the pfns are
contiguous, not just that the struct page pointers are contiguous. The
memory map is virtual contiguous if "CONFIG_FLATMEM=y ||
CONFIG_SPARSEMEM_VMEMMAP=y", but not for "CONFIG_SPARSEMEM=y &&
CONFIG_SPARSEMEM_VMEMMAP=n", so theoretically KVMGT could encounter struct
pages that are virtually contiguous, but not physically contiguous.
In practice, this flaw is likely a non-issue as it would cause functional
problems iff a section isn't 2M aligned _and_ is directly adjacent to
another section with discontiguous pfns.
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index de675d799c7d..429f0f993a13 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -161,7 +161,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu,
unsigned long gfn,
if (npage == 0)
base_page = cur_page;
- else if (base_page + npage != cur_page) {
+ else if (page_to_pfn(base_page) + npage !=
page_to_pfn(cur_page)) {
gvt_vgpu_err("The pages are not continuous\n");
ret = -EINVAL;
npage++;
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 02/28] drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
From: Yan Zhao
Currently intel_gvt_is_valid_gfn() is called in two places:
(1) shadowing guest GGTT entry
(2) shadowing guest PPGTT leaf entry,
which was introduced in commit cc753fbe1ac4
("drm/i915/gvt: validate gfn before set shadow page entry").
However, now it's not necessary to call this interface any more, because
a. GGTT partial write issue has been fixed by
commit bc0686ff5fad
("drm/i915/gvt: support inconsecutive partial gtt entry write")
commit 510fe10b6180
("drm/i915/gvt: fix a bug of partially write ggtt enties")
b. PPGTT resides in normal guest RAM and we only treat 8-byte writes
as valid page table writes. Any invalid GPA found is regarded as
an error, either due to guest misbehavior/attack or bug in host
shadow code.
So,rather than do GFN pre-checking and replace invalid GFNs with
scratch GFN and continue silently, just remove the pre-checking and
abort PPGTT shadowing on error detected.
c. GFN validity check is still performed in
intel_gvt_dma_map_guest_page() --> gvt_pin_guest_page().
It's more desirable to call VFIO interface to do both validity check
and mapping.
Calling intel_gvt_is_valid_gfn() to do GFN validity check from KVM side
while later mapping the GFN through VFIO interface is unnecessarily
fragile and confusing for unaware readers.
Signed-off-by: Yan Zhao
[sean: remove now-unused local variables]
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 36 +-
1 file changed, 1 insertion(+), 35 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 58b9b316ae46..f30922c55a0c 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -49,22 +49,6 @@
static bool enable_out_of_sync = false;
static int preallocated_oos_pages = 8192;
-static bool intel_gvt_is_valid_gfn(struct intel_vgpu *vgpu, unsigned long gfn)
-{
- struct kvm *kvm = vgpu->vfio_device.kvm;
- int idx;
- bool ret;
-
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return false;
-
- idx = srcu_read_lock(>srcu);
- ret = kvm_is_visible_gfn(kvm, gfn);
- srcu_read_unlock(>srcu, idx);
-
- return ret;
-}
-
/*
* validate a gm address and related range size,
* translate it to host gm address
@@ -1333,11 +1317,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu
*vgpu,
static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt)
{
struct intel_vgpu *vgpu = spt->vgpu;
- struct intel_gvt *gvt = vgpu->gvt;
- const struct intel_gvt_gtt_pte_ops *ops = gvt->gtt.pte_ops;
struct intel_vgpu_ppgtt_spt *s;
struct intel_gvt_gtt_entry se, ge;
- unsigned long gfn, i;
+ unsigned long i;
int ret;
trace_spt_change(spt->vgpu->id, "born", spt,
@@ -1354,13 +1336,6 @@ static int ppgtt_populate_spt(struct
intel_vgpu_ppgtt_spt *spt)
ppgtt_generate_shadow_entry(, s, );
ppgtt_set_shadow_entry(spt, , i);
} else {
- gfn = ops->get_pfn();
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(, gvt->gtt.scratch_mfn);
- ppgtt_set_shadow_entry(spt, , i);
- continue;
- }
-
ret = ppgtt_populate_shadow_entry(vgpu, spt, i, );
if (ret)
goto fail;
@@ -2335,14 +2310,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu
*vgpu, unsigned int off,
m.val64 = e.val64;
m.type = e.type;
- /* one PTE update may be issued in multiple writes and the
-* first write may not construct a valid gfn
-*/
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(, gvt->gtt.scratch_mfn);
- goto out;
- }
-
ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE,
_addr);
if (ret) {
@@ -2359,7 +2326,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu
*vgpu, unsigned int off,
ops->clear_present();
}
-out:
ggtt_set_guest_entry(ggtt_mm, , g_gtt_index);
ggtt_get_host_entry(ggtt_mm, , g_gtt_index);
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 01/28] drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
Check that the pfn found by gfn_to_pfn() is actually backed by "struct
page" memory prior to retrieving and dereferencing the page. KVM
supports backing guest memory with VM_PFNMAP, VM_IO, etc., and so
there is no guarantee the pfn returned by gfn_to_pfn() has an associated
"struct page".
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/gtt.c | 4
1 file changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 4ec85308379a..58b9b316ae46 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1183,6 +1183,10 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
+
+ if (!pfn_valid(pfn))
+ return -EINVAL;
+
return PageTransHuge(pfn_to_page(pfn));
}
--
2.40.1.606.ga4b1b128d6-goog
[Intel-gfx] [PATCH v3 00/28] drm/i915/gvt: KVM: KVMGT fixes and page-track cleanups
Fix a variety of found-by-inspection bugs in KVMGT, and overhaul KVM's
page-track APIs to provide a leaner and cleaner interface. The motivation
for this series is to (significantly) reduce the number of KVM APIs that
KVMGT uses, with a long-term goal of making all kvm_host.h headers
KVM-internal.
As always for this series, the KVMGT changes are compile tested only.
Based on "git://git.kernel.org/pub/scm/virt/kvm/kvm.git tags/kvm-6.4-1".
v3:
- Collect reviewed/tested tags (I apologize if I missed any, I manually
gathered them this time due to a goof in my workflow). [Yan]
- Drop check on max KVM paging size from KVMGT. [Yan]
- Drop the explicit change on THP pages, and instead validate that the
pfns (not struct page pointers) are contiguous. [Yan]
- Fix buggy intel_gvt_dma_map_guest_page() usage by eliminating a helper
for shadowing 2MiB GTT entries. [Yan]
- Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c instead of exposing
kvm_mmu_zap_all_fast() outside of mmu.c. [Yan]
- Fix an alignment goof in hlist_for_each_entry_srcu() usage. [Yan]
- Wrap full definition of external page track structures with
CONFIG_KVM_EXTERNAL_WRITE_TRACKING. [Yan]
v2:
- https://lore.kernel.org/all/20230311002258.852397-1-sea...@google.com
- Reuse vgpu_lock to protect gfn hash instead of introducing a new (and
buggy) mutext. [Yan]
- Remove a spurious return from kvm_page_track_init(). [Yan]
- Take @kvm directly in the inner __kvm_page_track_write(). [Yan]
- Delete the gfn sanity check that relies on kvm_is_visible_gfn() instead
of providing a dedicated interface. [Yan]
v1: https://lore.kernel.org/lkml/20221223005739.1295925-1-sea...@google.com
Sean Christopherson (24):
drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
drm/i915/gvt: Verify hugepages are contiguous in physical address
space
drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT
entry type
drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M
GTT
drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
drm/i915/gvt: Protect gfn hash table with vgpu_lock
KVM: x86/mmu: Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
KVM: x86: Reject memslot MOVE operations if KVMGT is attached
drm/i915/gvt: Don't bother removing write-protection on to-be-deleted
slot
KVM: x86/mmu: Move KVM-only page-track declarations to internal header
KVM: x86/mmu: Use page-track notifiers iff there are external users
KVM: x86/mmu: Drop infrastructure for multiple page-track modes
KVM: x86/mmu: Rename page-track APIs to reflect the new reality
KVM: x86/mmu: Assert that correct locks are held for page
write-tracking
KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
drm/i915/gvt: Drop final dependencies on KVM internal details
Yan Zhao (4):
drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
KVM: x86: Add a new page-track hook to handle memslot deletion
drm/i915/gvt: switch from ->track_flush_slot() to
->track_remove_region()
KVM: x86: Remove the unused page-track hook track_flush_slot()
arch/x86/include/asm/kvm_host.h | 16 +-
arch/x86/include/asm/kvm_page_track.h | 73 +++-
arch/x86/kvm/mmu.h| 2 +
arch/x86/kvm/mmu/mmu.c| 51 +++--
arch/x86/kvm/mmu/page_track.c | 256 +-
arch/x86/kvm/mmu/page_track.h | 58 ++
arch/x86/kvm/x86.c| 22 +--
drivers/gpu/drm/i915/gvt/gtt.c| 102 ++
drivers/gpu/drm/i915/gvt/gtt.h| 1 -
drivers/gpu/drm/i915/gvt/gvt.h| 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 117 +---
drivers/gpu/drm/i915/gvt/page_track.c | 10 +-
12 files changed, 320 insertions(+), 391 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
base-commit: b3c98052d46948a8d65d2778c7f306ff38366aac
--
2.40.1.606.ga4b1b128d6-goog
Re: [Intel-gfx] [PATCH v2 25/27] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
On Mon, May 08, 2023, Yan Zhao wrote:
> On Thu, May 04, 2023 at 10:17:20AM +0800, Yan Zhao wrote:
> > On Wed, May 03, 2023 at 04:16:10PM -0700, Sean Christopherson wrote:
> > > Finally getting back to this series...
> > >
> > > On Thu, Mar 23, 2023, Yan Zhao wrote:
> > > > On Fri, Mar 17, 2023 at 04:28:56PM +0800, Yan Zhao wrote:
> > > > > On Fri, Mar 10, 2023 at 04:22:56PM -0800, Sean Christopherson wrote:
> > > > > ...
> > > > > > +int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn)
> > > > > > +{
> > > > > > + struct kvm_memory_slot *slot;
> > > > > > + int idx;
> > > > > > +
> > > > > > + idx = srcu_read_lock(>srcu);
> > > > > > +
> > > > > > + slot = gfn_to_memslot(kvm, gfn);
> > > > > > + if (!slot) {
> > > > > > + srcu_read_unlock(>srcu, idx);
> > > > > > + return -EINVAL;
> > > > > > + }
> > > > > > +
> > > > > Also fail if slot->flags & KVM_MEMSLOT_INVALID is true?
> > > > > There should exist a window for external users to see an invalid slot
> > > > > when a slot is about to get deleted/moved.
> > > > > (It happens before MOVE is rejected in
> > > > > kvm_arch_prepare_memory_region()).
> > > >
> > > > Or using
> > > > if (!kvm_is_visible_memslot(slot)) {
> > > > srcu_read_unlock(>srcu, idx);
> > > > return -EINVAL;
> > > > }
> > >
> Hi Sean,
> After more thoughts, do you think checking KVM internal memslot is necessary?
I don't think it's necessary per se, but I also can't think of any reason to
allow
it.
> slot = gfn_to_memslot(kvm, gfn);
> if (!slot || slot->id >= KVM_USER_MEM_SLOTS) {
> srcu_read_unlock(>srcu, idx);
> return -EINVAL;
> }
>
> Do we allow write tracking to APIC access page when APIC-write VM exit
> is not desired?
Allow? Yes.
But KVM doesn't use write-tracking for anything APICv related, e.g. to disable
APICv, KVM instead zaps the SPTEs for the APIC access page and on page fault
goes
straight to MMIO emulation.
Theoretically, the guest could create an intermediate PTE in the APIC access
page
and AFAICT KVM would shadow the access and write-protect the APIC access page.
But that's benign as the resulting emulation would be handled just like emulated
APIC MMIO.
FWIW, the other internal memslots, TSS and idenity mapped page tables, are used
if and only if paging is disabled in the guest, i.e. there are no guest PTEs for
KVM to shadow (and paging must be enabled to enable VMX, so nested EPT is also
ruled out). So this is theoretically possible only for the APIC access page.
That changes with KVMGT, but that again should not be problematic. KVM will
emulate in response to the write-protected page and things go on. E.g. it's
arguably much weirder that the guest can read/write the identity mapped page
tables that are used for EPT without unrestricted guest.
There's no sane reason to allow creating PTEs in the APIC page, but I'm also not
all that motivated to "fix" things. account_shadowed() isn't expected to fail,
so KVM would need to check further up the stack, e.g. in walk_addr_generic() by
open coding a form of kvm_vcpu_gfn_to_hva_prot().
I _think_ that's the only place KVM would need to add a check, as KVM already
checks that the root, i.e. CR3, is in a "visible" memslot. I suppose KVM could
just synthesize triple fault, like it does for the root/CR3 case, but I don't
like making up behavior.
In other words, I'm not opposed to disallowing write-tracking internal memslots,
but I can't think of anything that will break, and so for me personally at
least,
the ROI isn't sufficient to justify writing tests and dealing with any fallout.
Re: [Intel-gfx] [PATCH v2 05/27] drm/i915/gvt: Verify VFIO-pinned page is THP when shadowing 2M gtt entry
On Sat, May 06, 2023, Yan Zhao wrote: > On Sat, May 06, 2023 at 02:35:41PM +0800, Yan Zhao wrote: > > > > Maybe the checking of PageTransHuge(cur_page) and bailing out is not > > > > necessary. > > > > If a page is not transparent huge, but there are 512 contigous 4K > > > > pages, I think it's still good to map them in IOMMU in 2M. > > > > See vfio_pin_map_dma() who does similar things. > > > > > > I agree that bailing isn't strictly necessary, and processing "blindly" > > > should > > > Just Work for HugeTLB and other hugepage types. I was going to argue > > > that it > > > would be safer to add this and then drop it at the end, but I think > > > that's a > > > specious argument. If not checking the page type is unsafe, then the > > > existing > > > code is buggy, and this changelog literally states that the check for > > > contiguous > > > pages guards against any such problems. > > > > > > I do think there's a (very, very theoretical) issue though. For > > > "CONFIG_SPARSEMEM=y > > > && CONFIG_SPARSEMEM_VMEMMAP=n", struct pages aren't virtually contiguous > > > with respect > > > to their pfns, i.e. it's possible (again, very theoretically) that two > > > struct pages > > > could be virtually contiguous but physically discontiguous. I suspect > > > I'm being > > > ridiculously paranoid, but for the efficient cases where pages are > > > guaranteed to > > > be contiguous, the extra page_to_pfn() checks should be optimized away by > > > the > > > compiler, i.e. there's no meaningful downside to the paranoia. > > To make sure I understand it correctly: > > There are 3 conditions: > > (1) Two struct pages aren't virtually contiguous, but there PFNs are > > contiguous. > > (2) Two struct pages are virtually contiguous but their PFNs aren't > > contiguous. > > (Looks this will not happen?) > > (3) Two struct pages are virtually contiguous, and their PFNs are > > contiguous, too. > > But they have different backends, e.g. > > PFN 1 and PFN 2 are contiguous, while PFN 1 belongs to RAM, and PFN 2 > > belongs to DEVMEM. > > > > I think you mean condition (3) is problematic, am I right? > Oh, I got it now. > You are saying about condition (2), with "CONFIG_SPARSEMEM=y && > CONFIG_SPARSEMEM_VMEMMAP=n". > Two struct pages are contiguous if one is at one section's tail and another at > another section's head, but the two sections aren't for contiguous PFNs. Yep, exactly.
Re: [Intel-gfx] [PATCH v2 05/27] drm/i915/gvt: Verify VFIO-pinned page is THP when shadowing 2M gtt entry
On Fri, Mar 17, 2023, Yan Zhao wrote:
> On Fri, Mar 10, 2023 at 04:22:36PM -0800, Sean Christopherson wrote:
> > When shadowing a GTT entry with a 2M page, explicitly verify that the
> > first page pinned by VFIO is a transparent hugepage instead of assuming
> > that page observed by is_2MB_gtt_possible() is the same page pinned by
> > vfio_pin_pages(). E.g. if userspace is doing something funky with the
> > guest's memslots, or if the page is demoted between is_2MB_gtt_possible()
> > and vfio_pin_pages().
> >
> > This is more of a performance optimization than a bug fix as the check
> > for contiguous struct pages should guard against incorrect mapping (even
> > though assuming struct pages are virtually contiguous is wrong).
> >
> > The real motivation for explicitly checking for a transparent hugepage
> > after pinning is that it will reduce the risk of introducing a bug in a
> > future fix for a page refcount leak (KVMGT doesn't put the reference
> > acquired by gfn_to_pfn()), and eventually will allow KVMGT to stop using
> > KVM's gfn_to_pfn() altogether.
> >
> > Signed-off-by: Sean Christopherson
> > ---
> > drivers/gpu/drm/i915/gvt/kvmgt.c | 18 --
> > 1 file changed, 16 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c
> > b/drivers/gpu/drm/i915/gvt/kvmgt.c
> > index 8ae7039b3683..90997cc385b4 100644
> > --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> > +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> > @@ -159,11 +159,25 @@ static int gvt_pin_guest_page(struct intel_vgpu
> > *vgpu, unsigned long gfn,
> > goto err;
> > }
> >
> > - if (npage == 0)
> > - base_page = cur_page;
> > + if (npage == 0) {
> > + /*
> > +* Bail immediately to avoid unnecessary pinning when
> > +* trying to shadow a 2M page and the host page isn't
> > +* a transparent hugepage.
> > +*
> > +* TODO: support other type hugepages, e.g. HugeTLB.
> > +*/
> > + if (size == I915_GTT_PAGE_SIZE_2M &&
> > + !PageTransHuge(cur_page))
> Maybe the checking of PageTransHuge(cur_page) and bailing out is not
> necessary.
> If a page is not transparent huge, but there are 512 contigous 4K
> pages, I think it's still good to map them in IOMMU in 2M.
> See vfio_pin_map_dma() who does similar things.
I agree that bailing isn't strictly necessary, and processing "blindly" should
Just Work for HugeTLB and other hugepage types. I was going to argue that it
would be safer to add this and then drop it at the end, but I think that's a
specious argument. If not checking the page type is unsafe, then the existing
code is buggy, and this changelog literally states that the check for contiguous
pages guards against any such problems.
I do think there's a (very, very theoretical) issue though. For
"CONFIG_SPARSEMEM=y
&& CONFIG_SPARSEMEM_VMEMMAP=n", struct pages aren't virtually contiguous with
respect
to their pfns, i.e. it's possible (again, very theoretically) that two struct
pages
could be virtually contiguous but physically discontiguous. I suspect I'm being
ridiculously paranoid, but for the efficient cases where pages are guaranteed to
be contiguous, the extra page_to_pfn() checks should be optimized away by the
compiler, i.e. there's no meaningful downside to the paranoia.
TL;DR: My plan is to drop this patch and instead harden the continuity check.
Re: [Intel-gfx] [PATCH v2 20/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Wed, Mar 15, 2023, Sean Christopherson wrote:
> On Wed, Mar 15, 2023, Yan Zhao wrote:
> > On Fri, Mar 10, 2023 at 04:22:51PM -0800, Sean Christopherson wrote:
> > > Disable the page-track notifier code at compile time if there are no
> > > external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
> > > now hooks emulated writes directly instead of relying on the page-track
> > > mechanism.
> > >
> > > Signed-off-by: Sean Christopherson
> > > ---
> > > arch/x86/include/asm/kvm_host.h | 2 ++
> > > arch/x86/include/asm/kvm_page_track.h | 2 ++
> > > arch/x86/kvm/mmu/page_track.c | 9 -
> > > arch/x86/kvm/mmu/page_track.h | 29 +++
> > > 4 files changed, 33 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/arch/x86/include/asm/kvm_host.h
> > > b/arch/x86/include/asm/kvm_host.h
> > > index 1a4225237564..a3423711e403 100644
> > > --- a/arch/x86/include/asm/kvm_host.h
> > > +++ b/arch/x86/include/asm/kvm_host.h
> > > @@ -1265,7 +1265,9 @@ struct kvm_arch {
> > >* create an NX huge page (without hanging the guest).
> > >*/
> > > struct list_head possible_nx_huge_pages;
> > > +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
> > > struct kvm_page_track_notifier_head track_notifier_head;
> > > +#endif
> > > /*
> > >* Protects marking pages unsync during page faults, as TDP MMU page
> > >* faults only take mmu_lock for read. For simplicity, the unsync
> > > diff --git a/arch/x86/include/asm/kvm_page_track.h
> > > b/arch/x86/include/asm/kvm_page_track.h
> > > index deece45936a5..53c2adb25a07 100644
> > > --- a/arch/x86/include/asm/kvm_page_track.h
> > > +++ b/arch/x86/include/asm/kvm_page_track.h
> > The "#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING" can be moved to the
> > front of this file?
> > All the structures are only exposed for external users now.
>
> Huh. I've no idea why I didn't do that. IIRC, the entire reason past me
> wrapped
> track_notifier_head in an #ifdef was to allow this change in kvm_page_track.h.
>
> I'll do this in the next version unless I discover an edge case I'm
> overlooking.
Ah, deja vu. I tried this first time around, and got yelled at by the kernel
test
robot. Unsuprisingly, my second attempt yielded the same result :-)
HDRTEST drivers/gpu/drm/i915/gvt/gvt.h
In file included from :
gpu/drivers/gpu/drm/i915/gvt/gvt.h:236:45: error: field ‘track_node’ has
incomplete type
236 | struct kvm_page_track_notifier_node track_node;
| ^~
The problem is direct header inclusion. Nothing in the kernel includes gvt.h
when CONFIG_DRM_I915_GVT=n, but the header include guard tests include headers
directly on the command line. I think I'll define a "stub" specifically to play
nice with this sort of testing. Guarding the guts of gvt.h with
CONFIG_DRM_I915_GVT
would just propagate the problem, and guarding the node definition in "struct
intel_vgpu" would be confusing since the guard would be dead code for all
intents
and purposes.
The obvious alternative would be to leave kvm_page_track_notifier_node outside
of
the #ifdef, but I really want to bury kvm_page_track_notifier_head for KVM's
sake,
and having "head" buried but not "node" would also be weird and confusing.
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 33f087437209..3d040741044b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -51,6 +51,12 @@ void kvm_page_track_unregister_notifier(struct kvm *kvm,
int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
+#else
+/*
+ * Allow defining a node in a structure even if page tracking is disabled, e.g.
+ * to play nice with testing headers via direct inclusion from the command
line.
+ */
+struct kvm_page_track_notifier_node {};
#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
Re: [Intel-gfx] [PATCH v2 25/27] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
Finally getting back to this series...
On Thu, Mar 23, 2023, Yan Zhao wrote:
> On Fri, Mar 17, 2023 at 04:28:56PM +0800, Yan Zhao wrote:
> > On Fri, Mar 10, 2023 at 04:22:56PM -0800, Sean Christopherson wrote:
> > ...
> > > +int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn)
> > > +{
> > > + struct kvm_memory_slot *slot;
> > > + int idx;
> > > +
> > > + idx = srcu_read_lock(>srcu);
> > > +
> > > + slot = gfn_to_memslot(kvm, gfn);
> > > + if (!slot) {
> > > + srcu_read_unlock(>srcu, idx);
> > > + return -EINVAL;
> > > + }
> > > +
> > Also fail if slot->flags & KVM_MEMSLOT_INVALID is true?
> > There should exist a window for external users to see an invalid slot
> > when a slot is about to get deleted/moved.
> > (It happens before MOVE is rejected in kvm_arch_prepare_memory_region()).
>
> Or using
> if (!kvm_is_visible_memslot(slot)) {
> srcu_read_unlock(>srcu, idx);
> return -EINVAL;
> }
Hrm. If the DELETE/MOVE succeeds, then the funky accounting is ok (by the end
of the series) as the tracking disappears on DELETE, KVMGT will reject MOVE, and
KVM proper zaps SPTEs and resets accounting on MOVE (account_shadowed() runs
under
mmu_lock and thus ensures all previous SPTEs are zapped before the "flush" from
kvm_arch_flush_shadow_memslot() can run).
If kvm_prepare_memory_region() fails though...
Ah, KVM itself is safe because of the aforementioned
kvm_arch_flush_shadow_memslot().
Any accounting done on a temporarily invalid memslot will be unwound when the
SPTEs
are zapped. So for KVM, ignoring invalid memslots is correct _and necessary_.
We could clean that up by having accounted_shadowed() use the @slot from the
fault,
which would close the window where the fault starts with a valid memslot but
then
sees an invalid memslot when accounting a new shadow page. But I don't think
there
is a bug there.
Right, and DELETE can't actually fail in the current code base, and we've
established
that MOVE can't possibly work. So even if this is problematic in theory, there
are
no _unknown_ bugs, and the known bugs are fixed by the end of the series.
And at the end of the series, KVMGT drops its tracking only when the DELETE is
committed. So I _think_ allowing external trackers to add and remove gfns for
write-tracking in an invalid slot is actually desirable/correct. I'm pretty
sure
removal should be allowed as that can lead to dangling write-protection in a
rollback scenario. And I can't think of anything that will break (in the
kernel)
if write-tracking a gfn in an invalid slot is allowed, so I don't see any harm
in
allowing the extremely theoretical case of KVMGT shadowing a gfn in a
to-be-deleted
memslot _and_ the deletion being rolled back.
Re: [Intel-gfx] [PATCH v2 16/27] KVM: x86: Add a new page-track hook to handle memslot deletion
On Fri, Mar 17, 2023, Yan Zhao wrote:
> On Fri, Mar 10, 2023 at 04:22:47PM -0800, Sean Christopherson wrote:
> > From: Yan Zhao
> >
> > Add a new page-track hook, track_remove_region(), that is called when a
> > memslot DELETE operation is about to be committed. The "remove" hook
> > will be used by KVMGT and will effectively replace the existing
> > track_flush_slot() altogether now that KVM itself doesn't rely on the
> > "flush" hook either.
> >
> > The "flush" hook is flawed as it's invoked before the memslot operation
> > is guaranteed to succeed, i.e. KVM might ultimately keep the existing
> > memslot without notifying external page track users, a.k.a. KVMGT. In
> > practice, this can't currently happen on x86, but there are no guarantees
> > that won't change in the future, not to mention that "flush" does a very
> > poor job of describing what is happening.
> >
> > Pass in the gfn+nr_pages instead of the slot itself so external users,
> > i.e. KVMGT, don't need to exposed to KVM internals (memslots). This will
> > help set the stage for additional cleanups to the page-track APIs.
> >
> > Cc: Zhenyu Wang
> > Signed-off-by: Yan Zhao
> > Co-developed-by: Sean Christopherson
> > Signed-off-by: Sean Christopherson
> ...
>
> > +void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot
> > *slot)
> > +{
> > + struct kvm_page_track_notifier_head *head;
> > + struct kvm_page_track_notifier_node *n;
> > + int idx;
> > +
> > + head = >arch.track_notifier_head;
> > +
> > + if (hlist_empty(>track_notifier_list))
> > + return;
> > +
> > + idx = srcu_read_lock(>track_srcu);
> > + hlist_for_each_entry_srcu(n, >track_notifier_list, node,
> > + srcu_read_lock_held(>track_srcu))
> Sorry, not sure why the alignment here is not right.
> Patchwork just sent me a mail to complain about it.
> Would you mind helping fix it in the next version?
Ah, it's off by two spaces, should be
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
srcu_read_lock_held(>track_srcu))
I'll get it fixed in the next version.
Re: [Intel-gfx] [PATCH v2 01/27] drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
On Wed, Mar 15, 2023, Andrzej Hajda wrote: > On 13.03.2023 16:37, Wang, Wei W wrote: > > On Saturday, March 11, 2023 8:23 AM, Sean Christopherson wrote: > > > diff --git a/drivers/gpu/drm/i915/gvt/gtt.c > > > b/drivers/gpu/drm/i915/gvt/gtt.c > > > index 4ec85308379a..58b9b316ae46 100644 > > > --- a/drivers/gpu/drm/i915/gvt/gtt.c > > > +++ b/drivers/gpu/drm/i915/gvt/gtt.c > > > @@ -1183,6 +1183,10 @@ static int is_2MB_gtt_possible(struct intel_vgpu > > > *vgpu, > > > pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry)); > > > if (is_error_noslot_pfn(pfn)) > > > return -EINVAL; > > > + > > > + if (!pfn_valid(pfn)) > > > + return -EINVAL; > > > + > > > > Merge the two errors in one "if" to have less LOC? > > i.e. > > if (is_error_noslot_pfn(pfn) || !pfn_valid(pfn)) > > return -EINVAL; > > you can just replace "if (is_error_noslot_pfn(pfn))" with "if > (!pfn_valid(pfn))", it covers both cases. Technically, yes, but the two checks are for very different things. Practically speaking, there can never be false negatives without KVM breaking horribly as overlap between struct page pfns and KVM's error/noslot would prevent mapping legal memory into a KVM guest. But I'd rather not hide the "did KVM find a valid mapping" in the "is this pfn backed by struct page" check, especially since this code goes away entirely by the end of the series.
Re: [Intel-gfx] [PATCH v2 20/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Wed, Mar 15, 2023, Yan Zhao wrote:
> On Fri, Mar 10, 2023 at 04:22:51PM -0800, Sean Christopherson wrote:
> > Disable the page-track notifier code at compile time if there are no
> > external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
> > now hooks emulated writes directly instead of relying on the page-track
> > mechanism.
> >
> > Signed-off-by: Sean Christopherson
> > ---
> > arch/x86/include/asm/kvm_host.h | 2 ++
> > arch/x86/include/asm/kvm_page_track.h | 2 ++
> > arch/x86/kvm/mmu/page_track.c | 9 -
> > arch/x86/kvm/mmu/page_track.h | 29 +++
> > 4 files changed, 33 insertions(+), 9 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/kvm_host.h
> > b/arch/x86/include/asm/kvm_host.h
> > index 1a4225237564..a3423711e403 100644
> > --- a/arch/x86/include/asm/kvm_host.h
> > +++ b/arch/x86/include/asm/kvm_host.h
> > @@ -1265,7 +1265,9 @@ struct kvm_arch {
> > * create an NX huge page (without hanging the guest).
> > */
> > struct list_head possible_nx_huge_pages;
> > +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
> > struct kvm_page_track_notifier_head track_notifier_head;
> > +#endif
> > /*
> > * Protects marking pages unsync during page faults, as TDP MMU page
> > * faults only take mmu_lock for read. For simplicity, the unsync
> > diff --git a/arch/x86/include/asm/kvm_page_track.h
> > b/arch/x86/include/asm/kvm_page_track.h
> > index deece45936a5..53c2adb25a07 100644
> > --- a/arch/x86/include/asm/kvm_page_track.h
> > +++ b/arch/x86/include/asm/kvm_page_track.h
> The "#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING" can be moved to the
> front of this file?
> All the structures are only exposed for external users now.
Huh. I've no idea why I didn't do that. IIRC, the entire reason past me
wrapped
track_notifier_head in an #ifdef was to allow this change in kvm_page_track.h.
I'll do this in the next version unless I discover an edge case I'm overlooking.
Thanks yet again!
Re: [Intel-gfx] [PATCH v2 20/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
On Wed, Mar 15, 2023, Yan Zhao wrote:
> Nit: there is a typo in the commit header: "iff" -> "if"
>
> > -void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
> > - int bytes)
> > +void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int
> > bytes)
> Line length is 81 characters. A little longer than 80 :)
>
> > +static inline bool kvm_page_track_has_external_user(struct kvm *kvm) {
> > return false; }
> This line is also too long.
The 80 character limit is a "soft" limit these days, e.g. checkpatch only
complains
if a line is 100+. In KVM x86, the preferred style is to treat the 80 char
limit
as "firm", for lack of a better word. E.g. let a line run over if it's just a
char or two and there's no other wrapping in the declaration, but don't create
long
lines just because checkpatch no longer yells.
There's obviously a fair bit of subjectivity, but the guideline has worked well
so far (hopefully I didn't just jinx us).
Re: [Intel-gfx] [PATCH v2 14/27] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
On Wed, Mar 15, 2023, Yan Zhao wrote:
> On Fri, Mar 10, 2023 at 04:22:45PM -0800, Sean Christopherson wrote:
> > Disallow moving memslots if the VM has external page-track users, i.e. if
> > KVMGT is being used to expose a virtual GPU to the guest, as KVM doesn't
> > correctly handle moving memory regions.
> >
> > Note, this is potential ABI breakage! E.g. userspace could move regions
> > that aren't shadowed by KVMGT without harming the guest. However, the
> > only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
> > regions. KVM's own support for moving memory regions was also broken for
> > multiple years (albeit for an edge case, but arguably moving RAM is
> > itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
> > new rmap and large page tracking when moving memslot").
> >
> > Signed-off-by: Sean Christopherson
> ...
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 29dd6c97d145..47ac9291cd43 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -12484,6 +12484,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
> >struct kvm_memory_slot *new,
> >enum kvm_mr_change change)
> > {
> > + /*
> > +* KVM doesn't support moving memslots when there are external page
> > +* trackers attached to the VM, i.e. if KVMGT is in use.
> > +*/
> > + if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
> > + return -EINVAL;
> Hmm, will page track work correctly on moving memslots when there's no
> external users?
>
> in case of KVM_MR_MOVE,
> kvm_prepare_memory_region(kvm, old, new, change)
> |->kvm_arch_prepare_memory_region(kvm, old, new, change)
>|->kvm_alloc_memslot_metadata(kvm, new)
> |->memset(>arch, 0, sizeof(slot->arch));
> |->kvm_page_track_create_memslot(kvm, slot, npages)
> The new->arch.arch.gfn_write_track will be fresh empty.
>
>
> kvm_arch_commit_memory_region(kvm, old, new, change);
> |->kvm_arch_free_memslot(kvm, old);
>|->kvm_page_track_free_memslot(slot);
> The old->arch.gfn_write_track is freed afterwards.
>
> So, in theory, the new GFNs are not write tracked though the old ones are.
>
> Is that acceptable for the internal page-track user?
It works because KVM zaps all SPTEs when a memslot is moved, i.e. the fact that
KVM loses the write-tracking counts is benign. I suspect no VMM actually does
does KVM_MR_MOVE in conjunction with shadow paging, but the ongoing maintenance
cost of supporting KVM_MR_MOVE is quite low at this point, so trying to rip it
out isn't worth the pain of having to deal with potential ABI breakage.
Though in hindsight I wish I had tried disallowed moving memslots instead of
fixing the various bugs a few years back. :-(
Re: [Intel-gfx] [PATCH v2 11/27] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
On Wed, Mar 15, 2023, Yan Zhao wrote:
> On Fri, Mar 10, 2023 at 04:22:42PM -0800, Sean Christopherson wrote:
> ...
> > -static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
> > - struct kvm_memory_slot *slot,
> > - struct kvm_page_track_notifier_node *node)
> > -{
> > - kvm_mmu_zap_all_fast(kvm);
> > -}
> > -
> > int kvm_mmu_init_vm(struct kvm *kvm)
> > {
> > struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
> > @@ -6110,7 +6103,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
> > }
> >
> > node->track_write = kvm_mmu_pte_write;
> > - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
> > kvm_page_track_register_notifier(kvm, node);
> >
> > kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index f706621c35b8..29dd6c97d145 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -12662,6 +12662,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
> > void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
> >struct kvm_memory_slot *slot)
> > {
> > + kvm_mmu_zap_all_fast(kvm);
> Could we still call kvm_mmu_invalidate_zap_pages_in_memslot() here?
> As I know, for TDX, its version of
> kvm_mmu_invalidate_zap_pages_in_memslot() is like
>
> static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
> struct kvm_memory_slot *slot,
> struct kvm_page_track_notifier_node *node)
> {
> if (kvm_gfn_shared_mask(kvm))
> kvm_mmu_zap_memslot(kvm, slot);
> else
> kvm_mmu_zap_all_fast(kvm);
> }
>
> Maybe this kind of judegment is better to be confined in mmu.c?
Hmm, yeah, I agree. The only reason I exposed kvm_mmu_zap_all_fast() is because
kvm_mmu_zap_all() is already exposed for kvm_arch_flush_shadow_all() and it felt
weird/wrong to split those. But that's the only usage of kvm_mmu_zap_all(), so
a better approach to maintain consistency would be to move
kvm_arch_flush_shadow_{all,memslot}() into mmu.c. I'll do that in the next
version.
Re: [Intel-gfx] [PATCH v2 19/27] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
On Wed, Mar 15, 2023, Yan Zhao wrote: > On Fri, Mar 10, 2023 at 04:22:50PM -0800, Sean Christopherson wrote: > > Bury the declaration of the page-track helpers that are intended only for > > internal KVM use in a "private" header. In addition to guarding against > > unwanted usage of the internal-only helpers, dropping their definitions > > avoids exposing other structures that should be KVM-internal, e.g. for > > memslots. This is a baby step toward making kvm_host.h a KVM-internal > > header in the very distant future. > > > > Signed-off-by: Sean Christopherson > > --- > > arch/x86/include/asm/kvm_page_track.h | 26 - > > arch/x86/kvm/mmu/mmu.c| 3 ++- > > arch/x86/kvm/mmu/page_track.c | 8 +-- > > arch/x86/kvm/mmu/page_track.h | 33 +++ > > arch/x86/kvm/x86.c| 1 + > > 5 files changed, 42 insertions(+), 29 deletions(-) > > create mode 100644 arch/x86/kvm/mmu/page_track.h > > > > diff --git a/arch/x86/include/asm/kvm_page_track.h > > b/arch/x86/include/asm/kvm_page_track.h > > index e5eb98ca4fce..deece45936a5 100644 > > --- a/arch/x86/include/asm/kvm_page_track.h > > +++ b/arch/x86/include/asm/kvm_page_track.h > > A curious question: > are arch/x86/include/asm/kvm_*.h all expected to be external accessible? Depends on what you mean by "expected". Currently, yes, everything in there is globally visible. But the vast majority of structs, defines, functions, etc. aren't intended for external non-KVM consumption, things ended up being globally visible largely through carelessness and/or a lack of a forcing function. E.g. there is absolutely no reason anything outside of KVM should need arch/x86/include/asm/kvm-x86-ops.h, but it landed in asm/ because, at the time it was added, nothing would be harmed by making kvm-x86-ops.h "public" and we didn't scrutinize the patches well enough. My primary motivation for this series is to (eventually) get to a state where only select symbols/defines/etc. are exposed by KVM to the outside world, and everything else is internal only. The end goal of tightly restricting KVM's global API is to allow concurrently loading multiple instances of kvm.ko so that userspace can upgrade/rollback KVM without needed to move VMs off the host, i.e. by performing intrahost migration between differenate instances of KVM on the same host. To do that safely, anything that is visible outside of KVM needs to be compatible across different instances of KVM, e.g. if kvm_vcpu is "public" then a KVM upgrade/rollback wouldn't be able to touch "struct kvm_vcpu" in any way. We'll definitely want to be able to modify things like the vCPU structures, thus the push to restrict the API. But even if we never realize that end goal, IMO drastically reducing KVM's "public" API surface is worthy goal in and of itself.
Re: [Intel-gfx] [PATCH v2 04/27] drm/i915/gvt: Incorporate KVM memslot info into check for 2MiB GTT entry
On Tue, Mar 14, 2023, Yan Zhao wrote: > On Fri, Mar 10, 2023 at 04:22:35PM -0800, Sean Christopherson wrote: > > Honor KVM's max allowed page size when determining whether or not a 2MiB > > GTT shadow page can be created for the guest. Querying KVM's max allowed > > size is somewhat odd as there's no strict requirement that KVM's memslots > > and VFIO's mappings are configured with the same gfn=>hva mapping, but > > the check will be accurate if userspace wants to have a functional guest, > > and at the very least checking KVM's memslots guarantees that the entire > > 2MiB range has been exposed to the guest. > > > hi Sean, > I remember in our last discussion, the conclusion was that > we can safely just use VFIO ABI (which is intel_gvt_dma_map_guest_page() > introduced in patch 7) to check max mapping size. [1][2] Gah, my apologies. I completely forgot about dropping KVM's mapping size check. I was pretty sure I was forgetting something, but couldn't figure out what I was forgetting. I'll drop this in the next version. Thanks!
[Intel-gfx] [PATCH v2 27/27] drm/i915/gvt: Drop final dependencies on KVM internal details
Open code gpa_to_gfn() in kvmgt_page_track_write() and drop KVMGT's dependency on kvm_host.h, i.e. include only kvm_page_track.h. KVMGT assumes "gfn == gpa >> PAGE_SHIFT" all over the place, including a few lines below in the same function with the same gpa, i.e. there's no reason to use KVM's helper for this one case. No functional change intended. Signed-off-by: Sean Christopherson --- drivers/gpu/drm/i915/gvt/gvt.h | 3 ++- drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index 2d65800d8e93..53a0a42a50db 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -34,10 +34,11 @@ #define _GVT_H_ #include -#include #include #include +#include + #include "i915_drv.h" #include "intel_gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index d16aced134b4..798d04481f03 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -1599,7 +1599,7 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len, mutex_lock(>vgpu_lock); - if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa))) + if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT)) intel_vgpu_page_track_handler(info, gpa, (void *)val, len); -- 2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 26/27] KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
Get/put references to KVM when a page-track notifier is (un)registered
instead of relying on the caller to do so. Forcing the caller to do the
bookkeeping is unnecessary and adds one more thing for users to get
wrong, e.g. see commit 9ed1fdee9ee3 ("drm/i915/gvt: Get reference to KVM
iff attachment to VM is successful").
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 10 --
arch/x86/kvm/mmu/page_track.c | 18 --
drivers/gpu/drm/i915/gvt/kvmgt.c | 17 +++--
3 files changed, 23 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 415537ce45b4..66a0d7c34311 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -47,12 +47,10 @@ struct kvm_page_track_notifier_node {
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
enum pg_level max_level);
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n);
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n);
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n);
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n);
int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 69b6431b394b..6ca644d3c926 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -157,17 +157,22 @@ int kvm_page_track_init(struct kvm *kvm)
* register the notifier so that event interception for the tracked guest
* pages can be received.
*/
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
-struct kvm_page_track_notifier_node *n)
+int kvm_page_track_register_notifier(struct kvm *kvm,
+struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
+ if (!kvm || kvm->mm != current->mm)
+ return -ESRCH;
+
+ kvm_get_kvm(kvm);
+
head = >arch.track_notifier_head;
write_lock(>mmu_lock);
hlist_add_head_rcu(>node, >track_notifier_list);
write_unlock(>mmu_lock);
+ return 0;
}
EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
@@ -175,9 +180,8 @@ EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
* stop receiving the event interception. It is the opposed operation of
* kvm_page_track_register_notifier().
*/
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n)
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
@@ -187,6 +191,8 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
hlist_del_rcu(>node);
write_unlock(>mmu_lock);
synchronize_srcu(>track_srcu);
+
+ kvm_put_kvm(kvm);
}
EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 898f1f1d308d..d16aced134b4 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -668,21 +668,19 @@ static bool __kvmgt_vgpu_exist(struct intel_vgpu *vgpu)
static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
{
struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
-
- if (!vgpu->vfio_device.kvm ||
- vgpu->vfio_device.kvm->mm != current->mm) {
- gvt_vgpu_err("KVM is required to use Intel vGPU\n");
- return -ESRCH;
- }
+ int ret;
if (__kvmgt_vgpu_exist(vgpu))
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
- kvm_get_kvm(vgpu->vfio_device.kvm);
- kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
->track_node);
+ ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
+ >track_node);
+ if (ret) {
+ gvt_vgpu_err("KVM is required to use Intel vGPU\n");
+ return ret;
+ }
set_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status);
@@ -717,7 +715,6 @@ static void intel_vgpu_close_device(s
[Intel-gfx] [PATCH v2 20/27] KVM: x86/mmu: Use page-track notifiers iff there are external users
Disable the page-track notifier code at compile time if there are no
external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
now hooks emulated writes directly instead of relying on the page-track
mechanism.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 2 ++
arch/x86/include/asm/kvm_page_track.h | 2 ++
arch/x86/kvm/mmu/page_track.c | 9 -
arch/x86/kvm/mmu/page_track.h | 29 +++
4 files changed, 33 insertions(+), 9 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1a4225237564..a3423711e403 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1265,7 +1265,9 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
struct kvm_page_track_notifier_head track_notifier_head;
+#endif
/*
* Protects marking pages unsync during page faults, as TDP MMU page
* faults only take mmu_lock for read. For simplicity, the unsync
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index deece45936a5..53c2adb25a07 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -55,6 +55,7 @@ void kvm_slot_page_track_remove_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
enum pg_level max_level);
@@ -64,5 +65,6 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index a21200df515d..619ec8e5fd32 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -194,6 +194,7 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
return !!READ_ONCE(slot->arch.gfn_track[mode][index]);
}
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
void kvm_page_track_cleanup(struct kvm *kvm)
{
struct kvm_page_track_notifier_head *head;
@@ -255,14 +256,13 @@ EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
* The node should figure out if the written page is the one that node is
* interested in by itself.
*/
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes)
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int
bytes)
{
struct kvm_page_track_notifier_head *head;
struct kvm_page_track_notifier_node *n;
int idx;
- head = >kvm->arch.track_notifier_head;
+ head = >arch.track_notifier_head;
if (hlist_empty(>track_notifier_list))
return;
@@ -273,8 +273,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
-
- kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
@@ -317,3 +315,4 @@ enum pg_level kvm_page_track_max_mapping_level(struct kvm
*kvm, gfn_t gfn,
return max_level;
}
EXPORT_SYMBOL_GPL(kvm_page_track_max_mapping_level);
+#endif
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
index 89712f123ad3..931b26b8fc8f 100644
--- a/arch/x86/kvm/mmu/page_track.h
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -6,8 +6,6 @@
#include
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
@@ -21,13 +19,36 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
const struct kvm_memory_slot *slot,
gfn_t gfn, enum kvm_page_track_mode mode);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int
bytes);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
static inline bool kvm_page_track_has_external_user(struct kvm *kvm)
{
return hlist_empty(>arch.track_notifier_head.track_notifier_list);
}
+#else
+static inline int kvm_page
[Intel-gfx] [PATCH v2 25/27] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
Refactor KVM's exported/external page-track, a.k.a. write-track, APIs
to take only the gfn and do the required memslot lookup in KVM proper.
Forcing users of the APIs to get the memslot unnecessarily bleeds
KVM internals into KVMGT and complicates usage of the APIs.
No functional change intended.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 8 +--
arch/x86/kvm/mmu/mmu.c| 4 +-
arch/x86/kvm/mmu/page_track.c | 86 ---
arch/x86/kvm/mmu/page_track.h | 5 ++
drivers/gpu/drm/i915/gvt/kvmgt.c | 37 +++-
5 files changed, 82 insertions(+), 58 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 20055064793a..415537ce45b4 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,11 +43,6 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void kvm_write_track_add_gfn(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn);
-
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
enum pg_level max_level);
@@ -58,6 +53,9 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+
+int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
+int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 3d1aad44c2ec..cf59b44de912 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -820,7 +820,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_add_gfn(kvm, slot, gfn);
+ return __kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -866,7 +866,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_remove_gfn(kvm, slot, gfn);
+ return __kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 327e73be62d6..69b6431b394b 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -74,16 +74,8 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
slot->arch.gfn_write_track[index] += count;
}
-/*
- * add guest page to the tracking pool so that corresponding access on that
- * page will be intercepted.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-gfn_t gfn)
+void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -104,18 +96,9 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct
kvm_memory_slot *slot,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
-/*
- * remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_remove_gfn(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void __kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn)
{
lockdep_assert_held_write(>mmu_lock);
@@ -133,7 +116,6 @@ void kvm_write_track_remove_gfn(struct kvm *kvm,
*/
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
/*
* check if the corresponding access on the specified guest page is tracked.
@@ -274,4 +256,64 @@ enum pg_level kvm_page_track_max_mapping_level(struct kvm
*kvm, gfn_t gfn,
return max_level;
}
EXPORT_SYMBOL_GPL(kvm_page_track_max_mapping_level);
+
+/*
+ * add guest page to the tracking pool so t
[Intel-gfx] [PATCH v2 22/27] KVM: x86/mmu: Rename page-track APIs to reflect the new reality
Rename the page-track APIs to capture that they're all about tracking
writes, now that the facade of supporting multiple modes is gone.
Opportunstically replace "slot" with "gfn" in anticipation of removing
the @slot param from the external APIs.
No functional change intended.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 8
arch/x86/kvm/mmu/mmu.c| 8
arch/x86/kvm/mmu/page_track.c | 21 +
arch/x86/kvm/mmu/page_track.h | 4 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 ++--
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 42a4ae451d36..20055064793a 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,10 +43,10 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_add_gfn(struct kvm *kvm,
+struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 7f21a1705438..3d1aad44c2ec 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -820,7 +820,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn);
+ return kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -866,7 +866,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_remove_page(kvm, slot, gfn);
+ return kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
@@ -2745,7 +2745,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct
kvm_memory_slot *slot,
* track machinery is used to write-protect upper-level shadow pages,
* i.e. this guards the role.level == 4K assertion below!
*/
- if (kvm_slot_page_track_is_active(kvm, slot, gfn))
+ if (kvm_gfn_is_write_tracked(kvm, slot, gfn))
return -EPERM;
/*
@@ -4153,7 +4153,7 @@ static bool page_fault_handle_page_track(struct kvm_vcpu
*vcpu,
* guest is writing the page which is write tracked which can
* not be fixed by page fault handler.
*/
- if (kvm_slot_page_track_is_active(vcpu->kvm, fault->slot, fault->gfn))
+ if (kvm_gfn_is_write_tracked(vcpu->kvm, fault->slot, fault->gfn))
return true;
return false;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index f8c89110f896..1993db4578e5 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -84,10 +84,9 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+gfn_t gfn)
{
-
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -102,12 +101,11 @@ void kvm_slot_page_track_add_page(struct kvm *kvm,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
+EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
/*
* remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page. It is the opposed operation of
- * kvm_slot_page_track_add_page().
+ * of corresponding access on that page.
*
* It should be called under the protection both of mmu-lock and kvm->srcu
* or kvm->slots_lock.
@@ -116,8 +114,8 @@ EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_remove_page(s
[Intel-gfx] [PATCH v2 24/27] KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
Bug the VM if something attempts to write-track a gfn, but write-tracking isn't enabled. The VM is doomed (and KVM has an egregious bug) if KVM or KVMGT wants to shadow guest page tables but can't because write-tracking isn't enabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/page_track.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index ffcd7ac66f9e..327e73be62d6 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -90,7 +90,7 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, 1); @@ -122,7 +122,7 @@ void kvm_write_track_remove_gfn(struct kvm *kvm, lockdep_assert_once(lockdep_is_held(>slots_lock) || srcu_read_lock_held(>srcu)); - if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm))) + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; update_gfn_write_track(slot, gfn, -1); -- 2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 23/27] KVM: x86/mmu: Assert that correct locks are held for page write-tracking
When adding/removing gfns to/from write-tracking, assert that mmu_lock
is held for write, and that either slots_lock or kvm->srcu is held.
mmu_lock must be held for write to protect gfn_write_track's refcount,
and SRCU or slots_lock must be held to protect the memslot itself.
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/page_track.c | 17 +++--
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 1993db4578e5..ffcd7ac66f9e 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -12,6 +12,7 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include
#include
#include
@@ -77,9 +78,6 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
* add guest page to the tracking pool so that corresponding access on that
* page will be intercepted.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -87,6 +85,11 @@ static void update_gfn_write_track(struct kvm_memory_slot
*slot, gfn_t gfn,
void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -107,9 +110,6 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
* remove the guest page from the tracking pool which stops the interception
* of corresponding access on that page.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -117,6 +117,11 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
void kvm_write_track_remove_gfn(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn)
{
+ lockdep_assert_held_write(>mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(>slots_lock) ||
+ srcu_read_lock_held(>srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 18/27] KVM: x86: Remove the unused page-track hook track_flush_slot()
From: Yan Zhao
Remove ->track_remove_slot(), there are no longer any users and it's
unlikely a "flush" hook will ever be the correct API to provide to an
external page-track user.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 11 ---
arch/x86/kvm/mmu/page_track.c | 26 --
arch/x86/kvm/x86.c| 2 --
3 files changed, 39 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 152c5e7d7868..e5eb98ca4fce 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -33,16 +33,6 @@ struct kvm_page_track_notifier_node {
*/
void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
struct kvm_page_track_notifier_node *node);
- /*
-* It is called when memory slot is being moved or removed
-* users can drop write-protection for the pages in that memory slot
-*
-* @kvm: the kvm where memory slot being moved or removed
-* @slot: the memory slot being moved or removed
-* @node: this node
-*/
- void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
/*
* Invoked when a memory region is removed from the guest. Or in KVM
@@ -87,7 +77,6 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index d4a8a995276a..907ab8abb452 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -278,32 +278,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t
gpa, const u8 *new,
kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
-/*
- * Notify the node that memory slot is being removed or moved so that it can
- * drop write-protection for the pages in the memory slot.
- *
- * The node should figure out it has any write-protected pages in this slot
- * by itself.
- */
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
-{
- struct kvm_page_track_notifier_head *head;
- struct kvm_page_track_notifier_node *n;
- int idx;
-
- head = >arch.track_notifier_head;
-
- if (hlist_empty(>track_notifier_list))
- return;
-
- idx = srcu_read_lock(>track_srcu);
- hlist_for_each_entry_srcu(n, >track_notifier_list, node,
- srcu_read_lock_held(>track_srcu))
- if (n->track_flush_slot)
- n->track_flush_slot(kvm, slot, n);
- srcu_read_unlock(>track_srcu, idx);
-}
-
/*
* Notify external page track nodes that a memory region is being removed from
* the VM, e.g. so that users can free any associated metadata.
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0da5ff007d20..59b02650cefc 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12673,8 +12673,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
kvm_mmu_zap_all_fast(kvm);
-
- kvm_page_track_flush_slot(kvm, slot);
}
static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 21/27] KVM: x86/mmu: Drop infrastructure for multiple page-track modes
Drop "support" for multiple page-track modes, as there is no evidence
that array-based and refcounted metadata is the optimal solution for
other modes, nor is there any evidence that other use cases, e.g. for
access-tracking, will be a good fit for the page-track machinery in
general.
E.g. one potential use case of access-tracking would be to prevent guest
access to poisoned memory (from the guest's perspective). In that case,
the number of poisoned pages is likely to be a very small percentage of
the guest memory, and there is no need to reference count the number of
access-tracking users, i.e. expanding gfn_track[] for a new mode would be
grossly inefficient. And for poisoned memory, host userspace would also
likely want to trap accesses, e.g. to inject #MC into the guest, and that
isn't currently supported by the page-track framework.
A better alternative for that poisoned page use case is likely a
variation of the proposed per-gfn attributes overlay (linked), which
would allow efficiently tracking the sparse set of poisoned pages, and by
default would exit to userspace on access.
Link: https://lore.kernel.org/all/y2wb48kd0j4vg...@google.com
Cc: Ben Gardon
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 12 +--
arch/x86/include/asm/kvm_page_track.h | 11 +--
arch/x86/kvm/mmu/mmu.c| 14 ++--
arch/x86/kvm/mmu/page_track.c | 111 --
arch/x86/kvm/mmu/page_track.h | 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 +-
6 files changed, 51 insertions(+), 104 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index a3423711e403..23567b851864 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -288,13 +288,13 @@ struct kvm_kernel_irq_routing_entry;
* kvm_mmu_page_role tracks the properties of a shadow page (where shadow page
* also includes TDP pages) to determine whether or not a page can be used in
* the given MMU context. This is a subset of the overall kvm_cpu_role to
- * minimize the size of kvm_memory_slot.arch.gfn_track, i.e. allows allocating
- * 2 bytes per gfn instead of 4 bytes per gfn.
+ * minimize the size of kvm_memory_slot.arch.gfn_write_track, i.e. allows
+ * allocating 2 bytes per gfn instead of 4 bytes per gfn.
*
* Upper-level shadow pages having gptes are tracked for write-protection via
- * gfn_track. As above, gfn_track is a 16 bit counter, so KVM must not create
- * more than 2^16-1 upper-level shadow pages at a single gfn, otherwise
- * gfn_track will overflow and explosions will ensure.
+ * gfn_write_track. As above, gfn_write_track is a 16 bit counter, so KVM must
+ * not create more than 2^16-1 upper-level shadow pages at a single gfn,
+ * otherwise gfn_write_track will overflow and explosions will ensue.
*
* A unique shadow page (SP) for a gfn is created if and only if an existing SP
* cannot be reused. The ability to reuse a SP is tracked by its role, which
@@ -1023,7 +1023,7 @@ struct kvm_lpage_info {
struct kvm_arch_memory_slot {
struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES];
struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1];
- unsigned short *gfn_track[KVM_PAGE_TRACK_MAX];
+ unsigned short *gfn_write_track;
};
/*
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 53c2adb25a07..42a4ae451d36 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,11 +4,6 @@
#include
-enum kvm_page_track_mode {
- KVM_PAGE_TRACK_WRITE,
- KVM_PAGE_TRACK_MAX,
-};
-
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
* the head which will be notified when guest is triggering the track event.
@@ -49,11 +44,9 @@ struct kvm_page_track_notifier_node {
};
void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
+ struct kvm_memory_slot *slot, gfn_t gfn);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
-struct kvm_memory_slot *slot, gfn_t gfn,
-enum kvm_page_track_mode mode);
+struct kvm_memory_slot *slot, gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index e192968340bf..7f21a1705438 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -820,8 +820,7 @@ static void account_shadowed(struct kvm *kvm, struct
kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_pag
[Intel-gfx] [PATCH v2 19/27] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
Bury the declaration of the page-track helpers that are intended only for
internal KVM use in a "private" header. In addition to guarding against
unwanted usage of the internal-only helpers, dropping their definitions
avoids exposing other structures that should be KVM-internal, e.g. for
memslots. This is a baby step toward making kvm_host.h a KVM-internal
header in the very distant future.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 26 -
arch/x86/kvm/mmu/mmu.c| 3 ++-
arch/x86/kvm/mmu/page_track.c | 8 +--
arch/x86/kvm/mmu/page_track.h | 33 +++
arch/x86/kvm/x86.c| 1 +
5 files changed, 42 insertions(+), 29 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index e5eb98ca4fce..deece45936a5 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -2,6 +2,8 @@
#ifndef _ASM_X86_KVM_PAGE_TRACK_H
#define _ASM_X86_KVM_PAGE_TRACK_H
+#include
+
enum kvm_page_track_mode {
KVM_PAGE_TRACK_WRITE,
KVM_PAGE_TRACK_MAX,
@@ -46,28 +48,15 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
-
-bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
-int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
-enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
- enum pg_level max_level);
-
-void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
-int kvm_page_track_create_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- unsigned long npages);
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot,
- gfn_t gfn, enum kvm_page_track_mode mode);
+
+enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
+ enum pg_level max_level);
void
kvm_page_track_register_notifier(struct kvm *kvm,
@@ -75,10 +64,5 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
-void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
-
-bool kvm_page_track_has_external_user(struct kvm *kvm);
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 4f2f83d8322e..e192968340bf 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -25,6 +25,7 @@
#include "kvm_cache_regs.h"
#include "smm.h"
#include "kvm_emulate.h"
+#include "page_track.h"
#include "cpuid.h"
#include "spte.h"
@@ -53,7 +54,7 @@
#include
#include
#include
-#include
+
#include "trace.h"
extern bool itlb_multihit_kvm_mitigation;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 907ab8abb452..a21200df515d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -15,10 +15,9 @@
#include
#include
-#include
-
#include "mmu.h"
#include "mmu_internal.h"
+#include "page_track.h"
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm)
{
@@ -318,8 +317,3 @@ enum pg_level kvm_page_track_max_mapping_level(struct kvm
*kvm, gfn_t gfn,
return max_level;
}
EXPORT_SYMBOL_GPL(kvm_page_track_max_mapping_level);
-
-bool kvm_page_track_has_external_user(struct kvm *kvm)
-{
- return hlist_empty(>arch.track_notifier_head.track_notifier_list);
-}
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
new file mode 100644
index ..89712f123ad3
--- /dev/null
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_PAGE_TRACK_H
+#define __KVM_X86_PAGE_TRACK_H
+
+#include
+
+#include
+
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+bool kvm_page_track_write_tracking_enabled(struct kvm *
[Intel-gfx] [PATCH v2 17/27] drm/i915/gvt: switch from ->track_flush_slot() to ->track_remove_region()
From: Yan Zhao
Switch from the poorly named and flawed ->track_flush_slot() to the newly
introduced ->track_remove_region(). From KVMGT's perspective, the two
hooks are functionally equivalent, the only difference being that
->track_remove_region() is called only when KVM is 100% certain the
memory region will be removed, i.e. is invoked slightly later in KVM's
memslot modification flow.
Cc: Zhenyu Wang
Suggested-by: Sean Christopherson
Signed-off-by: Yan Zhao
[sean: handle name change, massage changelog, rebase]
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 21 +
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 577712ea4893..9f188b6c3edf 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -108,9 +108,8 @@ struct gvt_dma {
static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
struct kvm_page_track_notifier_node *node);
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node);
static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf)
{
@@ -680,7 +679,7 @@ static int intel_vgpu_open_device(struct vfio_device
*vfio_dev)
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
- vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
+ vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
kvm_get_kvm(vgpu->vfio_device.kvm);
kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
>track_node);
@@ -1631,22 +1630,20 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8
*val, int len,
mutex_unlock(>vgpu_lock);
}
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node
*node)
{
unsigned long i;
- gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
mutex_lock(>vgpu_lock);
- for (i = 0; i < slot->npages; i++) {
- gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn))
- kvmgt_protect_table_del(info, gfn);
+ for (i = 0; i < nr_pages; i++) {
+ if (kvmgt_gfn_is_write_protected(info, gfn + i))
+ kvmgt_protect_table_del(info, gfn + i);
}
+
mutex_unlock(>vgpu_lock);
}
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 12/27] KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
Don't use the generic page-track mechanism to handle writes to guest PTEs
in KVM's MMU. KVM's MMU needs access to information that should not be
exposed to external page-track users, e.g. KVM needs (for some definitions
of "need") the vCPU to query the current paging mode, whereas external
users, i.e. KVMGT, have no ties to the current vCPU and so should never
need the vCPU.
Moving away from the page-track mechanism will allow dropping use of the
page-track mechanism for KVM's own MMU, and will also allow simplifying
and cleaning up the page-track APIs.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu.h | 2 ++
arch/x86/kvm/mmu/mmu.c | 13 ++---
arch/x86/kvm/mmu/page_track.c | 2 ++
4 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 17281d6825c9..1a4225237564 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1265,7 +1265,6 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
- struct kvm_page_track_notifier_node mmu_sp_tracker;
struct kvm_page_track_notifier_head track_notifier_head;
/*
* Protects marking pages unsync during page faults, as TDP MMU page
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 168c46fd8dd1..b8bde42f6037 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -119,6 +119,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes);
static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
{
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 409dabec69df..4f2f83d8322e 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5603,9 +5603,8 @@ static u64 *get_written_sptes(struct kvm_mmu_page *sp,
gpa_t gpa, int *nspte)
return spte;
}
-static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *new, int bytes,
- struct kvm_page_track_notifier_node *node)
+void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+int bytes)
{
gfn_t gfn = gpa >> PAGE_SHIFT;
struct kvm_mmu_page *sp;
@@ -6088,7 +6087,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
int kvm_mmu_init_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
int r;
INIT_LIST_HEAD(>arch.active_mmu_pages);
@@ -6102,9 +6100,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
return r;
}
- node->track_write = kvm_mmu_pte_write;
- kvm_page_track_register_notifier(kvm, node);
-
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6125,10 +6120,6 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
void kvm_mmu_uninit_vm(struct kvm *kvm)
{
- struct kvm_page_track_notifier_node *node = >arch.mmu_sp_tracker;
-
- kvm_page_track_unregister_notifier(kvm, node);
-
if (tdp_mmu_enabled)
kvm_mmu_uninit_tdp_mmu(kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e739dcc3375c..f39f190ad4ae 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -274,6 +274,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
if (n->track_write)
n->track_write(vcpu, gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
+
+ kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 16/27] KVM: x86: Add a new page-track hook to handle memslot deletion
From: Yan Zhao
Add a new page-track hook, track_remove_region(), that is called when a
memslot DELETE operation is about to be committed. The "remove" hook
will be used by KVMGT and will effectively replace the existing
track_flush_slot() altogether now that KVM itself doesn't rely on the
"flush" hook either.
The "flush" hook is flawed as it's invoked before the memslot operation
is guaranteed to succeed, i.e. KVM might ultimately keep the existing
memslot without notifying external page track users, a.k.a. KVMGT. In
practice, this can't currently happen on x86, but there are no guarantees
that won't change in the future, not to mention that "flush" does a very
poor job of describing what is happening.
Pass in the gfn+nr_pages instead of the slot itself so external users,
i.e. KVMGT, don't need to exposed to KVM internals (memslots). This will
help set the stage for additional cleanups to the page-track APIs.
Cc: Zhenyu Wang
Signed-off-by: Yan Zhao
Co-developed-by: Sean Christopherson
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 12
arch/x86/kvm/mmu/page_track.c | 23 +++
arch/x86/kvm/x86.c| 3 +++
3 files changed, 38 insertions(+)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 6a287bcbe8a9..152c5e7d7868 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,6 +43,17 @@ struct kvm_page_track_notifier_node {
*/
void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
+
+ /*
+* Invoked when a memory region is removed from the guest. Or in KVM
+* terms, when a memslot is deleted.
+*
+* @gfn: base gfn of the region being removed
+* @nr_pages: number of pages in the to-be-removed region
+* @node: this node
+*/
+ void (*track_remove_region)(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node);
};
int kvm_page_track_init(struct kvm *kvm);
@@ -77,6 +88,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 1cfc0a0ccc23..d4a8a995276a 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -304,6 +304,29 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct
kvm_memory_slot *slot)
srcu_read_unlock(>track_srcu, idx);
}
+/*
+ * Notify external page track nodes that a memory region is being removed from
+ * the VM, e.g. so that users can free any associated metadata.
+ */
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+ struct kvm_page_track_notifier_head *head;
+ struct kvm_page_track_notifier_node *n;
+ int idx;
+
+ head = >arch.track_notifier_head;
+
+ if (hlist_empty(>track_notifier_list))
+ return;
+
+ idx = srcu_read_lock(>track_srcu);
+ hlist_for_each_entry_srcu(n, >track_notifier_list, node,
+ srcu_read_lock_held(>track_srcu))
+ if (n->track_remove_region)
+ n->track_remove_region(slot->base_gfn, slot->npages, n);
+ srcu_read_unlock(>track_srcu, idx);
+}
+
enum pg_level kvm_page_track_max_mapping_level(struct kvm *kvm, gfn_t gfn,
enum pg_level max_level)
{
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 47ac9291cd43..0da5ff007d20 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12645,6 +12645,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ if (change == KVM_MR_DELETE)
+ kvm_page_track_delete_slot(kvm, old);
+
if (!kvm->arch.n_requested_mmu_pages &&
(change == KVM_MR_CREATE || change == KVM_MR_DELETE)) {
unsigned long nr_mmu_pages;
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 14/27] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
Disallow moving memslots if the VM has external page-track users, i.e. if
KVMGT is being used to expose a virtual GPU to the guest, as KVM doesn't
correctly handle moving memory regions.
Note, this is potential ABI breakage! E.g. userspace could move regions
that aren't shadowed by KVMGT without harming the guest. However, the
only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
regions. KVM's own support for moving memory regions was also broken for
multiple years (albeit for an edge case, but arguably moving RAM is
itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
new rmap and large page tracking when moving memslot").
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 3 +++
arch/x86/kvm/mmu/page_track.c | 5 +
arch/x86/kvm/x86.c| 7 +++
3 files changed, 15 insertions(+)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 0d65ae203fd6..6a287bcbe8a9 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -77,4 +77,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+
+bool kvm_page_track_has_external_user(struct kvm *kvm);
+
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 39a0863af8b4..1cfc0a0ccc23 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -321,3 +321,8 @@ enum pg_level kvm_page_track_max_mapping_level(struct kvm
*kvm, gfn_t gfn,
return max_level;
}
EXPORT_SYMBOL_GPL(kvm_page_track_max_mapping_level);
+
+bool kvm_page_track_has_external_user(struct kvm *kvm)
+{
+ return hlist_empty(>arch.track_notifier_head.track_notifier_list);
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 29dd6c97d145..47ac9291cd43 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12484,6 +12484,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ /*
+* KVM doesn't support moving memslots when there are external page
+* trackers attached to the VM, i.e. if KVMGT is in use.
+*/
+ if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
+ return -EINVAL;
+
if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
return -EINVAL;
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 15/27] drm/i915/gvt: Don't bother removing write-protection on to-be-deleted slot
When handling a slot "flush", don't call back into KVM to drop write
protection for gfns in the slot. Now that KVM rejects attempts to move
memory slots while KVMGT is attached, the only time a slot is "flushed"
is when it's being removed, i.e. the memslot and all its write-tracking
metadata is about to be deleted.
Signed-off-by: Sean Christopherson
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 8 +---
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 292750dc819f..577712ea4893 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1644,14 +1644,8 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn)) {
- write_lock(>mmu_lock);
- kvm_slot_page_track_remove_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
- write_unlock(>mmu_lock);
-
+ if (kvmgt_gfn_is_write_protected(info, gfn))
kvmgt_protect_table_del(info, gfn);
- }
}
mutex_unlock(>vgpu_lock);
}
--
2.40.0.rc1.284.g88254d51c5-goog
[Intel-gfx] [PATCH v2 13/27] KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
Drop @vcpu from KVM's ->track_write() hook provided for external users of
the page-track APIs now that KVM itself doesn't use the page-track
mechanism.
Signed-off-by: Sean Christopherson
---
arch/x86/include/asm/kvm_page_track.h | 5 ++---
arch/x86/kvm/mmu/page_track.c | 2 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 10 --
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h
b/arch/x86/include/asm/kvm_page_track.h
index 3f72c7a172fc..0d65ae203fd6 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -26,14 +26,13 @@ struct kvm_page_track_notifier_node {
* It is called when guest is writing the write-tracked page
* and write emulation is finished at that time.
*
-* @vcpu: the vcpu where the write access happened.
* @gpa: the physical address written by guest.
* @new: the data was written to the address.
* @bytes: the written length.
* @node: this node
*/
- void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes, struct kvm_page_track_notifier_node
*node);
+ void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
+ struct kvm_page_track_notifier_node *node);
/*
* It is called when memory slot is being moved or removed
* users can drop write-protection for the pages in that memory slot
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index f39f190ad4ae..39a0863af8b4 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -272,7 +272,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
const u8 *new,
hlist_for_each_entry_srcu(n, >track_notifier_list, node,
srcu_read_lock_held(>track_srcu))
if (n->track_write)
- n->track_write(vcpu, gpa, new, bytes, n);
+ n->track_write(gpa, new, bytes, n);
srcu_read_unlock(>track_srcu, idx);
kvm_mmu_track_write(vcpu, gpa, new, bytes);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 9824d075562e..292750dc819f 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -106,9 +106,8 @@ struct gvt_dma {
#define vfio_dev_to_vgpu(vfio_dev) \
container_of((vfio_dev), struct intel_vgpu, vfio_device)
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node);
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
@@ -1617,9 +1616,8 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info,
u64 gfn)
return 0;
}
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node)
{
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.40.0.rc1.284.g88254d51c5-goog
