This series implements the KVM stage 2 page table pre-faulting feature for arm64.
== Foundations == The series begins by establishing required foundations: 1. Updating kvm_s2_fault_desc to independently store the exception syndrome register (ESR) value, and updating all code paths to use this value exclusively. This is needed so we can later generate a synthetic fault to perform the pre-faulting - we need to be sure the code doesn't grab an incorrect ESR from elsewhere. 2. Updating kvm_s2_fault_desc to independently store the kvm_s2_mmu and updating all code paths to use this value exclusively. Similarly this is needed so we can generate a synthetic fault against the canonical stage 2 MMU (it would make no sense for it to touch nested shadow page tables) - we need to be sure that the code doesn't grab an incorrect MMU from elsewhere. 3. Updating the abort paths which consume kvm_s2_fault_desc to also return a kvm_s2_fault_result data structure. To perform pre-faulting the code must know the granule size of what was just walked. So the abort paths have to tell us what that was. 4. Pass walk flags to kvm_pgtable_get_leaf() to permit walking page tables under the MMU read lock. This is Jack's patch, verbatim, which allows the use of the KVM_PGTABLE_WALK_SHARED flag to walk page tables under the MMU read lock. Pre-faulting requires it to be able to work in parallel as specified by the API. The read lock precludes page tables being torn down behind our back. == Implementation == Pre-faulting is implemented in kvm_arch_vcpu_pre_fault_memory() whose job is to pre-fault the stage 2 page tables which map a specific GPA (which, for arm64, is the guest's IPA). This function is called by kvm_vcpu_pre_fault_memory() for each GPA in the range, which itself is ultimately invoked by userland via the KVM_PRE_FAULT_MEMORY ioctl. The implementation is simple - try to walk to the stage 2 page table mapping the GPA - if unmapped, fault it in through a synthetic page fault. pKVM is not supported regardless of whether the VM is protected or not. This is because pKVM instantiates vCPUs upon run, but pre-faulting is typically performed before a vCPU is run. It would be confusing and inconsistent to error out on non-running vCPUs but to pre-fault running ones. == Credits == This series is based, with gratitude, on Jack Thomson's series and their respins (links provided below) as well as the feedback he received. The series includes Jack's v5 "KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()" patch verbatim and two of his test patches (with minor fixups), plus a nested pre-fault test based on his. Link: https://patch.msgid.link/[email protected]/ Link: https://patch.msgid.link/[email protected]/ Link: https://patch.msgid.link/[email protected]/ Link: https://patch.msgid.link/[email protected]/ Link: https://patch.msgid.link/[email protected]/ == Reviewer Notes == I synced with maintainers on this who asked me to take a look, as there hadn't been progress on the series for some time. Signed-off-by: Lorenzo Stoakes (ARM) <[email protected]> --- v2: * Split series - Added 1/13 for the esr.h helpers, 2/13 for converting callers to use them, and move what was 1/8 to 3/13 to propagate s2fd->esr, as per Marc. * Made esr helpers __always_inline as per Marc. * Renamed esr_trap_get_class() -> esr_get_ec() + dropped churn as per Marc. * Updated 5/13 (was 3/8) to drop the kvm_s2_fault_result.mapped flag. * Also updated 5/13 so abort handlers return -EAGAIN instead of swallowing, as per Marc. * Split implementation patch further -> topup_mmu_memcache() change to 6/13, EHWPOISON change -> 7/13 and docs change -> 10/13, as per Marc. * No longer mkyoung existing walked ranges, updated docs to reflect, as per Marc. * Eliminated s2fd->pre_fault as per Marc. * Eliminated the redundant page table level in pre_fault_s2() as per Marc. * Put the commit message for 9/13 on a diet as per Marc. * Fixed up commit message for 11/13 (was 6/8) to correct reasoning about exit-reason assert, as per Fuad. * Simplified: nested test 13/13 (was 8/8), dropping the empty nested-S2 setup and using test_supports_el2() to handle NV=0 opt-out, as per Wei-Lin and Fuad. * Re-authored patch 13/13 to me as the changes are substantial enough to require it. Updated commit message to give Jack credit for original. v1: https://lore.kernel.org/r/[email protected] --- Jack Thomson (3): KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf() KVM: selftests: Enable pre_fault_memory_test for arm64 KVM: selftests: Add option for different backing in pre-fault tests Lorenzo Stoakes (ARM) (10): arm64: Add ESR fault helpers KVM: arm64: Use ESR helpers in guest abort handling KVM: arm64: Propagate and use esr in s2fd when handling guest aborts KVM: arm64: Propagate and use mmu in s2fd when handling guest aborts KVM: arm64: Propagate and use kvm_s2_fault_result on S2 fault KVM: arm64: Size the stage-2 memcache from the fault MMU KVM: arm64: Propagate EHWPOISON in kvm_s2_fault_pin_pfn() KVM: arm64: Implement KVM_PRE_FAULT_MEMORY Documentation: KVM: document arm64 KVM_PRE_FAULT_MEMORY KVM: selftests: Add nested pre-fault test for arm64 Documentation/virt/kvm/api.rst | 18 +- arch/arm64/include/asm/esr.h | 44 +++ arch/arm64/include/asm/kvm_emulate.h | 52 ++-- arch/arm64/include/asm/kvm_pgtable.h | 5 +- arch/arm64/include/asm/kvm_pkvm.h | 2 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 +- arch/arm64/kvm/hyp/pgtable.c | 5 +- arch/arm64/kvm/mmu.c | 300 +++++++++++++++++---- arch/arm64/kvm/nested.c | 2 +- tools/testing/selftests/kvm/Makefile.kvm | 2 + .../selftests/kvm/arm64/nv_pre_fault_memory_test.c | 158 +++++++++++ .../testing/selftests/kvm/pre_fault_memory_test.c | 152 +++++++++-- 14 files changed, 615 insertions(+), 137 deletions(-) --- base-commit: aa8e5dc6a7a2a1141ab40706a51010adcd0e57d2 change-id: 20260815-kvm-arm-prefault-9bb411b6897d Best regards, -- Lorenzo Stoakes (ARM) <[email protected]>
