Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-02-23 Thread Haibo Xu
On Thu, 18 Feb 2021 at 06:10, Marc Zyngier  wrote:
>
> On Thu, 04 Feb 2021 07:51:37 +,
> Haibo Xu  wrote:
> >
> > Kindly ping!
> >
> > On Thu, 21 Jan 2021 at 11:03, Haibo Xu  wrote:
> > >
> > > Re-send in case the previous email was blocked for the inlined hyper-link.
> > >
> > > Hi Marc,
> > >
> > > I have tried to enable the NV support in Qemu, and now I can
> > > successfully boot a L2 guest
> > > in Qemu KVM mode.
> > >
> > > This patch series looks good from the Qemu side except for two minor
> > > requirements:
> > > (1) Qemu will check whether a feature was supported by the KVM cap
> > > when the user tries to enable it in the command line, so a new
> > > capability was prefered for the NV(KVM_CAP_ARM_NV?).
>
> I have added KVM_CAP_ARM_EL2 (rather than NV) to that effect.
>
> > > (2) According to the Documentation/virt/kvm/api.rst, userspace can
> > > call KVM_ARM_VCPU_INIT multiple times for a given vcpu, but the
> > > kvm_vcpu_init_nested() do have some issue when called multiple
> > > times(please refer to the detailed comments in patch 63)
>
> This is now fixed, I believe.
>
> I have pushed out a branch [1] that addresses all the reported
> issues, though it currently lack some testing. Please let me know if
> it works for you.
>

Hi Marc,

I have verified the fix, and it works well with Qemu.

thanks,
Haibo

> Thanks,
>
> M.
>
> [1] 
> https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.12-WIP
>
> --
> Without deviation from the norm, progress is not possible.
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-02-17 Thread Marc Zyngier
On Thu, 04 Feb 2021 07:51:37 +,
Haibo Xu  wrote:
> 
> Kindly ping!
> 
> On Thu, 21 Jan 2021 at 11:03, Haibo Xu  wrote:
> >
> > Re-send in case the previous email was blocked for the inlined hyper-link.
> >
> > Hi Marc,
> >
> > I have tried to enable the NV support in Qemu, and now I can
> > successfully boot a L2 guest
> > in Qemu KVM mode.
> >
> > This patch series looks good from the Qemu side except for two minor
> > requirements:
> > (1) Qemu will check whether a feature was supported by the KVM cap
> > when the user tries to enable it in the command line, so a new
> > capability was prefered for the NV(KVM_CAP_ARM_NV?).

I have added KVM_CAP_ARM_EL2 (rather than NV) to that effect.

> > (2) According to the Documentation/virt/kvm/api.rst, userspace can
> > call KVM_ARM_VCPU_INIT multiple times for a given vcpu, but the
> > kvm_vcpu_init_nested() do have some issue when called multiple
> > times(please refer to the detailed comments in patch 63)

This is now fixed, I believe.

I have pushed out a branch [1] that addresses all the reported
issues, though it currently lack some testing. Please let me know if
it works for you.

Thanks,

M.

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.12-WIP

-- 
Without deviation from the norm, progress is not possible.
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-02-03 Thread Haibo Xu
Kindly ping!

On Thu, 21 Jan 2021 at 11:03, Haibo Xu  wrote:
>
> Re-send in case the previous email was blocked for the inlined hyper-link.
>
> Hi Marc,
>
> I have tried to enable the NV support in Qemu, and now I can
> successfully boot a L2 guest
> in Qemu KVM mode.
>
> This patch series looks good from the Qemu side except for two minor
> requirements:
> (1) Qemu will check whether a feature was supported by the KVM cap
> when the user tries
>  to enable it in the command line, so a new capability was
> prefered for the NV(KVM_CAP_ARM_NV?).
> (2) According to the Documentation/virt/kvm/api.rst, userspace can
> call KVM_ARM_VCPU_INIT
>  multiple times for a given vcpu, but the kvm_vcpu_init_nested()
> do have some issue when
>  called multiple times(please refer to the detailed comments in patch 63)
>
> Regards,
> Haibo
>
> On Fri, 11 Dec 2020 at 00:00, Marc Zyngier  wrote:
> >
> > This is a rework of the NV series that I posted 10 months ago[1], as a
> > lot of the KVM code has changed since, and the series apply anymore
> > (not that anybody really cares as the the HW is, as usual, made of
> > unobtainium...).
> >
> > From the previous version:
> >
> > - Integration with the new page-table code
> > - New exception injection code
> > - No more messing with the nVHE code
> > - No AArch32
> > - Rebased on v5.10-rc4 + kvmarm/next for 5.11
> >
> > From a functionality perspective, you can expect a L2 guest to work,
> > but don't even think of L3, as we only partially emulate the
> > ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
> > as well as anything that would require a Stage-1 PTW. What we want to
> > achieve is that with NV disabled, there is no performance overhead and
> > no regression.
> >
> > The series is roughly divided in 5 parts: exception handling, memory
> > virtualization, interrupts and timers for ARMv8.3, followed by the
> > ARMv8.4 support. There are of course some dependencies, but you'll
> > hopefully get the gist of it.
> >
> > For the most courageous of you, I've put out a branch[2]. Of course,
> > you'll need some userspace. Andre maintains a hacked version of
> > kvmtool[3] that takes a --nested option, allowing the guest to be
> > started at EL2. You can run the whole stack in the Foundation
> > model. Don't be in a hurry ;-).
> >
> > And to be clear: although Jintack and Christoffer have written tons of
> > the stuff originaly, I'm the one responsible for breaking it!
> >
> > [1] https://lore.kernel.org/r/20200211174938.27809-1-...@kernel.org
> > [2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git 
> > kvm-arm64/nv-5.11.-WIP
> > [3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5
> >
> > Andre Przywara (1):
> >   KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
> >
> > Christoffer Dall (15):
> >   KVM: arm64: nv: Introduce nested virtualization VCPU feature
> >   KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
> >   KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
> >   KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
> >   KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
> >   KVM: arm64: nv: Handle trapped ERET from virtual EL2
> >   KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
> >   KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
> >   KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
> > changes
> >   KVM: arm64: nv: Implement nested Stage-2 page table walk logic
> >   KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
> >   KVM: arm64: nv: arch_timer: Support hyp timer emulation
> >   KVM: arm64: nv: vgic: Emulate the HW bit in software
> >   KVM: arm64: nv: Add nested GICv3 tracepoints
> >   KVM: arm64: nv: Sync nested timer state with ARMv8.4
> >
> > Jintack Lim (19):
> >   arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
> >   KVM: arm64: nv: Handle HCR_EL2.NV system register traps
> >   KVM: arm64: nv: Support virtual EL2 exceptions
> >   KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
> >   KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
> >   KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
> >   KVM: arm64: nv: Handle PSCI call via smc from the guest
> >   KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
> >   KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
> >   KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
> >   KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
> >   KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
> >   KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
> >   KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
> >   KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
> >   KVM: arm64: nv: Set a handler for the system instruction traps
> >   KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
> >   KVM: arm64: nv: Trap and emulate TLBI instructions 

Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-01-20 Thread Haibo Xu
Re-send in case the previous email was blocked for the inlined hyper-link.

Hi Marc,

I have tried to enable the NV support in Qemu, and now I can
successfully boot a L2 guest
in Qemu KVM mode.

This patch series looks good from the Qemu side except for two minor
requirements:
(1) Qemu will check whether a feature was supported by the KVM cap
when the user tries
 to enable it in the command line, so a new capability was
prefered for the NV(KVM_CAP_ARM_NV?).
(2) According to the Documentation/virt/kvm/api.rst, userspace can
call KVM_ARM_VCPU_INIT
 multiple times for a given vcpu, but the kvm_vcpu_init_nested()
do have some issue when
 called multiple times(please refer to the detailed comments in patch 63)

Regards,
Haibo

On Fri, 11 Dec 2020 at 00:00, Marc Zyngier  wrote:
>
> This is a rework of the NV series that I posted 10 months ago[1], as a
> lot of the KVM code has changed since, and the series apply anymore
> (not that anybody really cares as the the HW is, as usual, made of
> unobtainium...).
>
> From the previous version:
>
> - Integration with the new page-table code
> - New exception injection code
> - No more messing with the nVHE code
> - No AArch32
> - Rebased on v5.10-rc4 + kvmarm/next for 5.11
>
> From a functionality perspective, you can expect a L2 guest to work,
> but don't even think of L3, as we only partially emulate the
> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
> as well as anything that would require a Stage-1 PTW. What we want to
> achieve is that with NV disabled, there is no performance overhead and
> no regression.
>
> The series is roughly divided in 5 parts: exception handling, memory
> virtualization, interrupts and timers for ARMv8.3, followed by the
> ARMv8.4 support. There are of course some dependencies, but you'll
> hopefully get the gist of it.
>
> For the most courageous of you, I've put out a branch[2]. Of course,
> you'll need some userspace. Andre maintains a hacked version of
> kvmtool[3] that takes a --nested option, allowing the guest to be
> started at EL2. You can run the whole stack in the Foundation
> model. Don't be in a hurry ;-).
>
> And to be clear: although Jintack and Christoffer have written tons of
> the stuff originaly, I'm the one responsible for breaking it!
>
> [1] https://lore.kernel.org/r/20200211174938.27809-1-...@kernel.org
> [2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git 
> kvm-arm64/nv-5.11.-WIP
> [3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5
>
> Andre Przywara (1):
>   KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
>
> Christoffer Dall (15):
>   KVM: arm64: nv: Introduce nested virtualization VCPU feature
>   KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
>   KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
>   KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
>   KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
>   KVM: arm64: nv: Handle trapped ERET from virtual EL2
>   KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
>   KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
>   KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
> changes
>   KVM: arm64: nv: Implement nested Stage-2 page table walk logic
>   KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
>   KVM: arm64: nv: arch_timer: Support hyp timer emulation
>   KVM: arm64: nv: vgic: Emulate the HW bit in software
>   KVM: arm64: nv: Add nested GICv3 tracepoints
>   KVM: arm64: nv: Sync nested timer state with ARMv8.4
>
> Jintack Lim (19):
>   arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
>   KVM: arm64: nv: Handle HCR_EL2.NV system register traps
>   KVM: arm64: nv: Support virtual EL2 exceptions
>   KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
>   KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
>   KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
>   KVM: arm64: nv: Handle PSCI call via smc from the guest
>   KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
>   KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
>   KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
>   KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
>   KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
>   KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
>   KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
>   KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
>   KVM: arm64: nv: Set a handler for the system instruction traps
>   KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
>   KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
>   KVM: arm64: nv: Nested GICv3 Support
>
> Marc Zyngier (31):
>   KVM: arm64: nv: Add EL2 system registers to vcpu context
>   KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
>   KVM: arm64: nv: Handle virtual EL2 registers in
> 

Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-01-20 Thread Haibo Xu
On Fri, 11 Dec 2020 at 00:00, Marc Zyngier  wrote:
>
> This is a rework of the NV series that I posted 10 months ago[1], as a
> lot of the KVM code has changed since, and the series apply anymore
> (not that anybody really cares as the the HW is, as usual, made of
> unobtainium...).
>
> From the previous version:
>
> - Integration with the new page-table code
> - New exception injection code
> - No more messing with the nVHE code
> - No AArch32
> - Rebased on v5.10-rc4 + kvmarm/next for 5.11
>
> From a functionality perspective, you can expect a L2 guest to work,
> but don't even think of L3, as we only partially emulate the
> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
> as well as anything that would require a Stage-1 PTW. What we want to
> achieve is that with NV disabled, there is no performance overhead and
> no regression.
>
> The series is roughly divided in 5 parts: exception handling, memory
> virtualization, interrupts and timers for ARMv8.3, followed by the
> ARMv8.4 support. There are of course some dependencies, but you'll
> hopefully get the gist of it.
>
> For the most courageous of you, I've put out a branch[2]. Of course,
> you'll need some userspace. Andre maintains a hacked version of
> kvmtool[3] that takes a --nested option, allowing the guest to be
> started at EL2. You can run the whole stack in the Foundation
> model. Don't be in a hurry ;-).
>
> And to be clear: although Jintack and Christoffer have written tons of
> the stuff originaly, I'm the one responsible for breaking it!
>
> [1] https://lore.kernel.org/r/20200211174938.27809-1-...@kernel.org
> [2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git
kvm-arm64/nv-5.11.-WIP
> [3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5

Hi Marc,

I have tried to enable the NV support in Qemu, and now I can successfully
boot a L2 guest
in Qemu KVM mode.

This patch series looks good from the Qemu side except for two minor
requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the
user tries
 to enable it in the command line, so a new capability was prefered for
the NV(KVM_CAP_ARM_NV?).
(2) According to the Documentation/virt/kvm/api.rst
,
userspace can call KVM_ARM_VCPU_INIT
 multiple times for a given vcpu, but the kvm_vcpu_init_nested() do
have some issue when
 called multiple times(please refer to the detailed comments in patch
63)

Regards,
Haibo
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-01-18 Thread André Przywara
On 10/12/2020 15:58, Marc Zyngier wrote:

Hi,

> This is a rework of the NV series that I posted 10 months ago[1], as a



> For the most courageous of you, I've put out a branch[2]. Of course,
> you'll need some userspace. Andre maintains a hacked version of
> kvmtool[3] that takes a --nested option, allowing the guest to be
> started at EL2.
I have updated the branch, as the headers changed as well:
https://gitlab.arm.com/linux-arm/kvmtool/-/commits/nv-5.11-WIP/

Cheers,
Andre
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-01-12 Thread Haibo Xu
On Mon, 11 Jan 2021 at 16:59, Marc Zyngier  wrote:
>
> Hi Haibo,
>
> On 2021-01-11 07:20, Haibo Xu wrote:
> > On Fri, 11 Dec 2020 at 00:00, Marc Zyngier  wrote:
> >>
> >> This is a rework of the NV series that I posted 10 months ago[1], as a
> >> lot of the KVM code has changed since, and the series apply anymore
> >> (not that anybody really cares as the the HW is, as usual, made of
> >> unobtainium...).
> >>
> >> From the previous version:
> >>
> >> - Integration with the new page-table code
> >> - New exception injection code
> >> - No more messing with the nVHE code
> >> - No AArch32
> >> - Rebased on v5.10-rc4 + kvmarm/next for 5.11
> >>
> >> From a functionality perspective, you can expect a L2 guest to work,
> >> but don't even think of L3, as we only partially emulate the
> >> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
> >> as well as anything that would require a Stage-1 PTW. What we want to
> >> achieve is that with NV disabled, there is no performance overhead and
> >> no regression.
> >>
> >> The series is roughly divided in 5 parts: exception handling, memory
> >> virtualization, interrupts and timers for ARMv8.3, followed by the
> >> ARMv8.4 support. There are of course some dependencies, but you'll
> >> hopefully get the gist of it.
> >>
> >> For the most courageous of you, I've put out a branch[2]. Of course,
> >> you'll need some userspace. Andre maintains a hacked version of
> >> kvmtool[3] that takes a --nested option, allowing the guest to be
> >> started at EL2. You can run the whole stack in the Foundation
> >> model. Don't be in a hurry ;-).
> >>
> >
> > Hi Marc,
> >
> > I got a kernel BUG message when booting the L2 guest kernel with the
> > kvmtool on a FVP setup.
> > Could you help have a look about the BUG message as well as my
> > environment configuration?
> > I think It probably caused by some local configurations of the FVP
> > setup.
>
> No, this is likely a bug in your L1 guest, which was fixed in -rc3:
>
> 2a5f1b67ec57 ("KVM: arm64: Don't access PMCR_EL0 when no PMU is
> available")
>
> and was found in the exact same circumstances. Alternatively, and if
> you don't want to change your L1 guest, you can just pass the --pmu
> option to kvmtool when starting the L1 guest.

After passing --pmu when starting a L1 guest, I can successfully run a
L2 guest now!
Thanks so much for the help!

Haibo

>
> Hope this helps,
>
>  M.
> --
> Jazz is not dead. It just smells funny...
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


Re: [PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2021-01-11 Thread Marc Zyngier

Hi Haibo,

On 2021-01-11 07:20, Haibo Xu wrote:

On Fri, 11 Dec 2020 at 00:00, Marc Zyngier  wrote:


This is a rework of the NV series that I posted 10 months ago[1], as a
lot of the KVM code has changed since, and the series apply anymore
(not that anybody really cares as the the HW is, as usual, made of
unobtainium...).

From the previous version:

- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32
- Rebased on v5.10-rc4 + kvmarm/next for 5.11

From a functionality perspective, you can expect a L2 guest to work,
but don't even think of L3, as we only partially emulate the
ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
as well as anything that would require a Stage-1 PTW. What we want to
achieve is that with NV disabled, there is no performance overhead and
no regression.

The series is roughly divided in 5 parts: exception handling, memory
virtualization, interrupts and timers for ARMv8.3, followed by the
ARMv8.4 support. There are of course some dependencies, but you'll
hopefully get the gist of it.

For the most courageous of you, I've put out a branch[2]. Of course,
you'll need some userspace. Andre maintains a hacked version of
kvmtool[3] that takes a --nested option, allowing the guest to be
started at EL2. You can run the whole stack in the Foundation
model. Don't be in a hurry ;-).



Hi Marc,

I got a kernel BUG message when booting the L2 guest kernel with the
kvmtool on a FVP setup.
Could you help have a look about the BUG message as well as my
environment configuration?
I think It probably caused by some local configurations of the FVP 
setup.


No, this is likely a bug in your L1 guest, which was fixed in -rc3:

2a5f1b67ec57 ("KVM: arm64: Don't access PMCR_EL0 when no PMU is 
available")


and was found in the exact same circumstances. Alternatively, and if
you don't want to change your L1 guest, you can just pass the --pmu
option to kvmtool when starting the L1 guest.

Hope this helps,

M.
--
Jazz is not dead. It just smells funny...
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm


[PATCH v3 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

2020-12-10 Thread Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a
lot of the KVM code has changed since, and the series apply anymore
(not that anybody really cares as the the HW is, as usual, made of
unobtainium...).

>From the previous version:

- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32
- Rebased on v5.10-rc4 + kvmarm/next for 5.11

>From a functionality perspective, you can expect a L2 guest to work,
but don't even think of L3, as we only partially emulate the
ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU,
as well as anything that would require a Stage-1 PTW. What we want to
achieve is that with NV disabled, there is no performance overhead and
no regression.

The series is roughly divided in 5 parts: exception handling, memory
virtualization, interrupts and timers for ARMv8.3, followed by the
ARMv8.4 support. There are of course some dependencies, but you'll
hopefully get the gist of it.

For the most courageous of you, I've put out a branch[2]. Of course,
you'll need some userspace. Andre maintains a hacked version of
kvmtool[3] that takes a --nested option, allowing the guest to be
started at EL2. You can run the whole stack in the Foundation
model. Don't be in a hurry ;-).

And to be clear: although Jintack and Christoffer have written tons of
the stuff originaly, I'm the one responsible for breaking it!

[1] https://lore.kernel.org/r/20200211174938.27809-1-...@kernel.org
[2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git 
kvm-arm64/nv-5.11.-WIP
[3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5

Andre Przywara (1):
  KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ

Christoffer Dall (15):
  KVM: arm64: nv: Introduce nested virtualization VCPU feature
  KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
  KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
  KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
  KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
  KVM: arm64: nv: Handle trapped ERET from virtual EL2
  KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
  KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
  KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
changes
  KVM: arm64: nv: Implement nested Stage-2 page table walk logic
  KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
  KVM: arm64: nv: arch_timer: Support hyp timer emulation
  KVM: arm64: nv: vgic: Emulate the HW bit in software
  KVM: arm64: nv: Add nested GICv3 tracepoints
  KVM: arm64: nv: Sync nested timer state with ARMv8.4

Jintack Lim (19):
  arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
  KVM: arm64: nv: Handle HCR_EL2.NV system register traps
  KVM: arm64: nv: Support virtual EL2 exceptions
  KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
  KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
  KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
  KVM: arm64: nv: Handle PSCI call via smc from the guest
  KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
  KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
  KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
  KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
  KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
  KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
  KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
  KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
  KVM: arm64: nv: Set a handler for the system instruction traps
  KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
  KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
  KVM: arm64: nv: Nested GICv3 Support

Marc Zyngier (31):
  KVM: arm64: nv: Add EL2 system registers to vcpu context
  KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
  KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
  KVM: arm64: nv: Handle SPSR_EL2 specially
  KVM: arm64: nv: Handle HCR_EL2.E2H specially
  KVM: arm64: nv: Save/Restore vEL2 sysregs
  KVM: arm64: nv: Forward debug traps to the nested guest
  KVM: arm64: nv: Filter out unsupported features from ID regs
  KVM: arm64: nv: Hide RAS from nested guests
  KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
  KVM: arm64: nv: Handle shadow stage 2 page faults
  KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
  KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's
  KVM: arm64: nv: Add handling of EL2-specific timer registers
  KVM: arm64: nv: Load timer before the GIC
  KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
  KVM: arm64: nv: Implement maintenance interrupt forwarding
  KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
  KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation