[PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>> >Hi,
>> >
>> >On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
>> >> >On 19/11/2018 09:10, Mark Rutland wrote:
>> >> >> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
>> >> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
>> >> >> Hi,
>> >> >>> When virtual machine starts, hang up.
>> >> >>
>> >> >> I take it you mean the *guest* hangs? Because it doesn't get a
>> >> >> timer
>> >> >> interrupt?
>> >> >>
>> >> >>> The kernel version of guest
>> >> >>> is 4.16. Host support vgic_v3.
>> >> >>
>> >> >> Your host kernel is something recent, I guess?
>> >> >>
>> >> >>> It was mainly due to the incorrect vgic_irq's(intid=27) group
>> >> >>> value
>> >> >>> during injection interruption. when kvm_vgic_vcpu_init is called,
>> >> >>> dist is not initialized at this time. Unable to get vgic V3 or V2
>> >> >>> correctly, so group is not set.
>> >> >>
>> >> >> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
>> >> >> version?) or some other userland tool?
>> >> >>
>> >> >
>> >> > QEMU emulator version 3.0.50 .
>> >> >
>> >> >>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>> >> >>> time.
>> >> >>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>> >> >>> interrupt injection failed.
>> >> >>>
>> >> >>> Signed-off-by: Peng Hao
>> >> >>> ---
>> >> >>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>> >> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> >> >>>
>> >> >>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>> b/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>> index 9c0dd23..d101000 100644
>> >> >>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu
>> >> >>> *vcpu,
>> >> >>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>> >> >>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>> >> >>>
>> >> >>> -if (irq->group)
>> >> >>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
[...]
>> >Could you try the following patch:
>> >
>> >diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
>> >index c0c0b88af1d5..6fa858c7a5a6 100644
>> >--- a/virt/kvm/arm/vgic/vgic-init.c
>> >+++ b/virt/kvm/arm/vgic/vgic-init.c
>> >@@ -231,13 +231,6 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
>> >irq->config = VGIC_CONFIG_LEVEL;
>> >}
>> >-/*
>> >- * GICv3 can only be created via the KVM_DEVICE_CREATE API and
>> >- * so we always know the emulation type at this point as it's
>> >- * either explicitly configured as GICv3, or explicitly
>> >- * configured as GICv2, or not configured yet which also
>> >- * implies GICv2.
>> >- */
>> >if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3)
>> >irq->group = 1;
>> >else
>> >@@ -298,6 +291,16 @@ int vgic_init(struct kvm *kvm)
>> >if (ret)
>> >goto out;
>> >+/* Initialize groups on CPUs created before the VGIC type was known */
>> >+kvm_for_each_vcpu(i, vcpu, kvm) {
>> >+struct vgic_cpu *vgic_cpu = >arch.vgic_cpu;
>> >+
>> >+for (i = 0; i < VGIC_NR_PRIVATE_IRQS; i++) {
>> >+struct vgic_irq *irq = _cpu->private_irqs[i];
>> >+irq->group = 1;
>> >+}
>> >+}
>> >+
>> >if (vgic_has_its(kvm)) {
>> >ret = vgic_v4_init(kvm);
>> >if (ret)
>> >
>> Sorry for the late reply.
>> I test the patch and it works. But I think there is randomness here.
>> 1.vgic_init: set intid/27 group=1
>> 2.vgic_mmio_write_group:## intid/27 group=0
>> 3.!!! receive a vtimer irq - it may exist.
>> 4.vgic_mmio_write_group:## intid/27 group=1
>>
>Are these writes from userspace or from the guest?
>(I assume they're from the guest.)
>
>>
>> I made an attempt before:
>>
>>Was this previous experiment using my patch or without my patch?
>>
>> 1. kvm_vgic_vcpu_init
>> according to if kvm_vgic_global_state.type == VGIC_V3, set group=1
>> 2.vgic_mmio_write_group:## intid/27 group=0
> >3.!!! receive a vtimer irq -I really receive a vtimer irq, then the
> >guest hangs.
>>
>
>If the guest you're using is flipping the group of the interrupt and yet
>enabling the device and interrupt while flipping the group, then it's
>not expected to work and it's a guest bug.
>
>My hunch here tells me that my patch is what we need for KVM, and either
>
>(1) You no longer have a problem but are tracing some guest
>initialization behavior which is fine, or
>
>(2) You still have a problem because you also have some broken guest
>code (guest UEFI or guest Linux) that also needs fixing.
>
>If you can demonstrate that you still have a problem, with my path
>applied to KVM, and with a recent guest Linux and guest UEFI, then I'll
>be happy to have another look. In that case, could you please provide
>the exact
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Fri, Nov 23, 2018 at 02:01:56PM +0800, peng.h...@zte.com.cn wrote:
> >Hi,
> >
> >On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
> >> >On 19/11/2018 09:10, Mark Rutland wrote:
> >> >> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
> >> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> >> >> Hi,
> >> >>> When virtual machine starts, hang up.
> >> >>
> >> >> I take it you mean the *guest* hangs? Because it doesn't get a timer
> >> >> interrupt?
> >> >>
> >> >>> The kernel version of guest
> >> >>> is 4.16. Host support vgic_v3.
> >> >>
> >> >> Your host kernel is something recent, I guess?
> >> >>
> >> >>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
> >> >>> during injection interruption. when kvm_vgic_vcpu_init is called,
> >> >>> dist is not initialized at this time. Unable to get vgic V3 or V2
> >> >>> correctly, so group is not set.
> >> >>
> >> >> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> >> >> version?) or some other userland tool?
> >> >>
> >> >
> >> > QEMU emulator version 3.0.50 .
> >> >
> >> >>> group is setted to 1 when vgic_mmio_write_group is invoked at some
> >> >>> time.
> >> >>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
> >> >>> interrupt injection failed.
> >> >>>
> >> >>> Signed-off-by: Peng Hao
> >> >>> ---
> >> >>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
> >> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
> >> >>>
> >> >>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c
> >> >>> b/virt/kvm/arm/vgic/vgic-v3.c
> >> >>> index 9c0dd23..d101000 100644
> >> >>> --- a/virt/kvm/arm/vgic/vgic-v3.c
> >> >>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> >> >>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
> >> >>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
> >> >>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
> >> >>>
> >> >>> -if (irq->group)
> >> >>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
> >> >>
> >> >> This is not the right fix, not only because it basically reverts the
> >> >> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> >> >> their configured group).
> >> >>
> >> >> Can you try to work out why kvm_vgic_vcpu_init() is apparently
> >> >> called
> >> >> before dist->vgic_model is set, also what value it has?
> >> >> If I understand the code correctly, that shouldn't happen for a
> >> >> GICv3.
> >> >>
> >> > Even if the value of group is correctly assigned in
> >> > kvm_vgic_vcpu_init, the group is then written 0 through
> >> > >vgic_mmio_write_group.
> >> > If the interrupt comes at this time, the interrupt injection fails.
> >>
> >> Does that mean that the guest is configuring its interrupts as Group0?
> >> That sounds wrong, Linux should configure all it's interrupts as
> >> non-secure group1.
> >> >>>
> >> >>> no, I think that uefi dose this, not linux.
> >> >>> 1. kvm_vgic_vcpu_init
> >> >>> 2. vgic_create
> >> >>> 3. kvm_vgic_dist_init
> >> >>> 4.vgic_mmio_write_group: uefi as guest, write group=0
> >> >>> 5.vgic_mmio_write_group: linux as guest, write group=1
> >> >>
> >> >> Is this the same issue fixed by EDK2 commit:
> >> >>
> >> >> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3
> >> >> interrupt acknowledge")
> >> >>
> >> >> ... where EDK2 would try to use IAR0 rather than IAR1?
> >> >>
> >> >> The commit messages notes this lead to a boot-time hang.
> >> >
> >> >I managed to trigger an issue with a really old EFI implementation that
> >> >doesn't configure its interrupts as Group1, and yet tries to ACK its
> >> >interrupts using the Group1 accessor. Guess what? It is not going to work.
> >> >
> >> >Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
> >> >to be the fix (I only assume it does, I haven't actually checked). A
> >> >recent build, as found in Debian Buster, works perfectly (tested with
> >> >both QEMU v2.12 and tip of tree).
> >> >
> >> >Now, I really don't get what you're saying about Linux not getting
> >> >interrupts. How do you get to booting Linux if EFI is not making any
> >> >forward progress? Are you trying them independently?
> >> >
> >> I start linux with bypassing uefi, the print info is the same.
> >> [507107.748908] vgic_mmio_write_group:## intid/27 group=0
> >> [507107.752185] vgic_mmio_write_group:## intid/27 group=0
> >> [507107.899566] vgic_mmio_write_group:## intid/27 group=1
> >> [507107.907370] vgic_mmio_write_group:## intid/27 group=1
> >> the command line is like this:
> >> /home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
> >> virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
> >> /home/kernelboot/vmlinuz-4.16.0+
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>Hi,
>
>On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
>> >On 19/11/2018 09:10, Mark Rutland wrote:
>> >> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
>> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
>> >> Hi,
>> >>> When virtual machine starts, hang up.
>> >>
>> >> I take it you mean the *guest* hangs? Because it doesn't get a timer
>> >> interrupt?
>> >>
>> >>> The kernel version of guest
>> >>> is 4.16. Host support vgic_v3.
>> >>
>> >> Your host kernel is something recent, I guess?
>> >>
>> >>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>> >>> during injection interruption. when kvm_vgic_vcpu_init is called,
>> >>> dist is not initialized at this time. Unable to get vgic V3 or V2
>> >>> correctly, so group is not set.
>> >>
>> >> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
>> >> version?) or some other userland tool?
>> >>
>> >
>> > QEMU emulator version 3.0.50 .
>> >
>> >>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>> >>> time.
>> >>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>> >>> interrupt injection failed.
>> >>>
>> >>> Signed-off-by: Peng Hao
>> >>> ---
>> >>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> >>>
>> >>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c
>> >>> b/virt/kvm/arm/vgic/vgic-v3.c
>> >>> index 9c0dd23..d101000 100644
>> >>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> >>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> >>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>> >>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>> >>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>> >>>
>> >>> -if (irq->group)
>> >>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>> >>
>> >> This is not the right fix, not only because it basically reverts the
>> >> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
>> >> their configured group).
>> >>
>> >> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
>> >> before dist->vgic_model is set, also what value it has?
>> >> If I understand the code correctly, that shouldn't happen for a GICv3.
>> >>
>> > Even if the value of group is correctly assigned in
>> > kvm_vgic_vcpu_init, the group is then written 0 through
>> > >vgic_mmio_write_group.
>> > If the interrupt comes at this time, the interrupt injection fails.
>>
>> Does that mean that the guest is configuring its interrupts as Group0?
>> That sounds wrong, Linux should configure all it's interrupts as
>> non-secure group1.
>> >>>
>> >>> no, I think that uefi dose this, not linux.
>> >>> 1. kvm_vgic_vcpu_init
>> >>> 2. vgic_create
>> >>> 3. kvm_vgic_dist_init
>> >>> 4.vgic_mmio_write_group: uefi as guest, write group=0
>> >>> 5.vgic_mmio_write_group: linux as guest, write group=1
>> >>
>> >> Is this the same issue fixed by EDK2 commit:
>> >>
>> >> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
>> >> acknowledge")
>> >>
>> >> ... where EDK2 would try to use IAR0 rather than IAR1?
>> >>
>> >> The commit messages notes this lead to a boot-time hang.
>> >
>> >I managed to trigger an issue with a really old EFI implementation that
>> >doesn't configure its interrupts as Group1, and yet tries to ACK its
>> >interrupts using the Group1 accessor. Guess what? It is not going to work.
>> >
>> >Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
>> >to be the fix (I only assume it does, I haven't actually checked). A
>> >recent build, as found in Debian Buster, works perfectly (tested with
>> >both QEMU v2.12 and tip of tree).
>> >
>> >Now, I really don't get what you're saying about Linux not getting
>> >interrupts. How do you get to booting Linux if EFI is not making any
>> >forward progress? Are you trying them independently?
>> >
>> I start linux with bypassing uefi, the print info is the same.
>> [507107.748908] vgic_mmio_write_group:## intid/27 group=0
>> [507107.752185] vgic_mmio_write_group:## intid/27 group=0
>> [507107.899566] vgic_mmio_write_group:## intid/27 group=1
>> [507107.907370] vgic_mmio_write_group:## intid/27 group=1
>> the command line is like this:
>> /home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
>> virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
>> /home/kernelboot/vmlinuz-4.16.0+ -initrd
>> /home/kernelboot/initramfs-4.16.0+.img -append root=/dev/mapper/cla-root ro
>> crashkernel=auto rd.lvm.lv=cla/root rd.lvm.lv=cla/swap.UTF-8 -drive
>> file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
>> -device
>>
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Wed, Nov 21, 2018 at 03:53:03PM +, Julien Thierry wrote:
>
>
> On 21/11/18 15:24, Christoffer Dall wrote:
> >On Wed, Nov 21, 2018 at 12:17:45PM +, Julien Thierry wrote:
> >>
> >>
> >>On 21/11/18 11:06, Christoffer Dall wrote:
> >>>Hi,
> >>>
> >>>On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
> >On 19/11/2018 09:10, Mark Rutland wrote:
> >>On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> >>Hi,
> >>>When virtual machine starts, hang up.
> >>
> >>I take it you mean the *guest* hangs? Because it doesn't get a timer
> >>interrupt?
> >>
> >>>The kernel version of guest
> >>>is 4.16. Host support vgic_v3.
> >>
> >>Your host kernel is something recent, I guess?
> >>
> >>>It was mainly due to the incorrect vgic_irq's(intid=27) group value
> >>>during injection interruption. when kvm_vgic_vcpu_init is called,
> >>>dist is not initialized at this time. Unable to get vgic V3 or V2
> >>>correctly, so group is not set.
> >>
> >>Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> >>version?) or some other userland tool?
> >>
> >
> >QEMU emulator version 3.0.50 .
> >
> >>>group is setted to 1 when vgic_mmio_write_group is invoked at some
> >>>time.
> >>>when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
> >>>interrupt injection failed.
> >>>
> >>>Signed-off-by: Peng Hao
> >>>---
> >>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>>diff --git a/virt/kvm/arm/vgic/vgic-v3.c
> >>>b/virt/kvm/arm/vgic/vgic-v3.c
> >>>index 9c0dd23..d101000 100644
> >>>--- a/virt/kvm/arm/vgic/vgic-v3.c
> >>>+++ b/virt/kvm/arm/vgic/vgic-v3.c
> >>>@@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
> >>>struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
> >>>(val & ICH_LR_PENDING_BIT)) irq->line_level = false;
> >>>
> >>>-if (irq->group)
> >>>+if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
> >>
> >>This is not the right fix, not only because it basically reverts the
> >>GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> >>their configured group).
> >>
> >>Can you try to work out why kvm_vgic_vcpu_init() is apparently
> >>called
> >>before dist->vgic_model is set, also what value it has?
> >>If I understand the code correctly, that shouldn't happen for a
> >>GICv3.
> >>
> >Even if the value of group is correctly assigned in
> >kvm_vgic_vcpu_init, the group is then written 0 through
> >vgic_mmio_write_group.
> > If the interrupt comes at this time, the interrupt injection
> > fails.
>
> Does that mean that the guest is configuring its interrupts as Group0?
> That sounds wrong, Linux should configure all it's interrupts as
> non-secure group1.
> >>>
> >>>no, I think that uefi dose this, not linux.
> >>>1. kvm_vgic_vcpu_init
> >>>2. vgic_create
> >>>3. kvm_vgic_dist_init
> >>>4.vgic_mmio_write_group: uefi as guest, write group=0
> >>>5.vgic_mmio_write_group: linux as guest, write group=1
> >>
> >>Is this the same issue fixed by EDK2 commit:
> >>
> >>66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3
> >>interrupt acknowledge")
> >>
> >>... where EDK2 would try to use IAR0 rather than IAR1?
> >>
> >>The commit messages notes this lead to a boot-time hang.
> >
> >I managed to trigger an issue with a really old EFI implementation that
> >doesn't configure its interrupts as Group1, and yet tries to ACK its
> >interrupts using the Group1 accessor. Guess what? It is not going to
> >work.
> >
> >Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
> >to be the fix (I only assume it does, I haven't actually checked). A
> >recent build, as found in Debian Buster, works perfectly (tested with
> >both QEMU v2.12 and tip of tree).
> >
> >Now, I really don't get what you're saying about Linux not getting
> >interrupts. How do you get to booting Linux if EFI is not making any
> >forward progress? Are you trying them independently?
> >
> I start linux with bypassing uefi, the print info is the same.
> [507107.748908] vgic_mmio_write_group:## intid/27 group=0
> [507107.752185] vgic_mmio_write_group:## intid/27 group=0
> [507107.899566] vgic_mmio_write_group:## intid/27 group=1
> [507107.907370] vgic_mmio_write_group:## intid/27
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 21/11/18 15:24, Christoffer Dall wrote:
On Wed, Nov 21, 2018 at 12:17:45PM +, Julien Thierry wrote:
On 21/11/18 11:06, Christoffer Dall wrote:
Hi,
On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
On 19/11/2018 09:10, Mark Rutland wrote:
On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
Hi,
When virtual machine starts, hang up.
I take it you mean the *guest* hangs? Because it doesn't get a timer
interrupt?
The kernel version of guest
is 4.16. Host support vgic_v3.
Your host kernel is something recent, I guess?
It was mainly due to the incorrect vgic_irq's(intid=27) group value
during injection interruption. when kvm_vgic_vcpu_init is called,
dist is not initialized at this time. Unable to get vgic V3 or V2
correctly, so group is not set.
Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
version?) or some other userland tool?
QEMU emulator version 3.0.50 .
group is setted to 1 when vgic_mmio_write_group is invoked at some
time.
when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
interrupt injection failed.
Signed-off-by: Peng Hao
---
virt/kvm/arm/vgic/vgic-v3.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 9c0dd23..d101000 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
(val & ICH_LR_PENDING_BIT)) irq->line_level = false;
-if (irq->group)
+if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
This is not the right fix, not only because it basically reverts the
GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
their configured group).
Can you try to work out why kvm_vgic_vcpu_init() is apparently called
before dist->vgic_model is set, also what value it has?
If I understand the code correctly, that shouldn't happen for a GICv3.
Even if the value of group is correctly assigned in kvm_vgic_vcpu_init, the
group is then written 0 through vgic_mmio_write_group.
If the interrupt comes at this time, the interrupt injection fails.
Does that mean that the guest is configuring its interrupts as Group0?
That sounds wrong, Linux should configure all it's interrupts as
non-secure group1.
no, I think that uefi dose this, not linux.
1. kvm_vgic_vcpu_init
2. vgic_create
3. kvm_vgic_dist_init
4.vgic_mmio_write_group: uefi as guest, write group=0
5.vgic_mmio_write_group: linux as guest, write group=1
Is this the same issue fixed by EDK2 commit:
66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
acknowledge")
... where EDK2 would try to use IAR0 rather than IAR1?
The commit messages notes this lead to a boot-time hang.
I managed to trigger an issue with a really old EFI implementation that
doesn't configure its interrupts as Group1, and yet tries to ACK its
interrupts using the Group1 accessor. Guess what? It is not going to work.
Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
to be the fix (I only assume it does, I haven't actually checked). A
recent build, as found in Debian Buster, works perfectly (tested with
both QEMU v2.12 and tip of tree).
Now, I really don't get what you're saying about Linux not getting
interrupts. How do you get to booting Linux if EFI is not making any
forward progress? Are you trying them independently?
I start linux with bypassing uefi, the print info is the same.
[507107.748908] vgic_mmio_write_group:## intid/27 group=0
[507107.752185] vgic_mmio_write_group:## intid/27 group=0
[507107.899566] vgic_mmio_write_group:## intid/27 group=1
[507107.907370] vgic_mmio_write_group:## intid/27 group=1
the command line is like this:
/home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
/home/kernelboot/vmlinuz-4.16.0+ -initrd /home/kernelboot/initramfs-4.16.0+.img
-append root=/dev/mapper/cla-root ro crashkernel=auto rd.lvm.lv=cla/root
rd.lvm.lv=cla/swap.UTF-8 -drive
file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-vnc 0.0.0.0:0 -k en-us -device
virtio-gpu-pci,id=video0,max_outputs=1,bus=pci.3,addr=0x0 -device pvpanic-mmio
-msg timestamp=on
This is strange. That's probably the Linux 4.16 kernel setting group to 0, and
I'll continue to track in guest.
Could you try the following patch:
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index c0c0b88af1d5..6fa858c7a5a6 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -231,13 +231,6 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
irq->config =
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Wed, Nov 21, 2018 at 12:17:45PM +, Julien Thierry wrote:
>
>
> On 21/11/18 11:06, Christoffer Dall wrote:
> >Hi,
> >
> >On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
> >>>On 19/11/2018 09:10, Mark Rutland wrote:
> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
> >>On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> Hi,
> >When virtual machine starts, hang up.
>
> I take it you mean the *guest* hangs? Because it doesn't get a timer
> interrupt?
>
> >The kernel version of guest
> >is 4.16. Host support vgic_v3.
>
> Your host kernel is something recent, I guess?
>
> >It was mainly due to the incorrect vgic_irq's(intid=27) group value
> >during injection interruption. when kvm_vgic_vcpu_init is called,
> >dist is not initialized at this time. Unable to get vgic V3 or V2
> >correctly, so group is not set.
>
> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> version?) or some other userland tool?
>
> >>>
> >>>QEMU emulator version 3.0.50 .
> >>>
> >group is setted to 1 when vgic_mmio_write_group is invoked at some
> >time.
> >when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
> >interrupt injection failed.
> >
> >Signed-off-by: Peng Hao
> >---
> > virt/kvm/arm/vgic/vgic-v3.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >diff --git a/virt/kvm/arm/vgic/vgic-v3.c
> >b/virt/kvm/arm/vgic/vgic-v3.c
> >index 9c0dd23..d101000 100644
> >--- a/virt/kvm/arm/vgic/vgic-v3.c
> >+++ b/virt/kvm/arm/vgic/vgic-v3.c
> >@@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
> >struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
> >(val & ICH_LR_PENDING_BIT)) irq->line_level = false;
> >
> >-if (irq->group)
> >+if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>
> This is not the right fix, not only because it basically reverts the
> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> their configured group).
>
> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
> before dist->vgic_model is set, also what value it has?
> If I understand the code correctly, that shouldn't happen for a GICv3.
>
> >>>Even if the value of group is correctly assigned in
> >>>kvm_vgic_vcpu_init, the group is then written 0 through
> >>>vgic_mmio_write_group.
> >>> If the interrupt comes at this time, the interrupt injection fails.
> >>
> >>Does that mean that the guest is configuring its interrupts as Group0?
> >>That sounds wrong, Linux should configure all it's interrupts as
> >>non-secure group1.
> >
> >no, I think that uefi dose this, not linux.
> >1. kvm_vgic_vcpu_init
> >2. vgic_create
> >3. kvm_vgic_dist_init
> >4.vgic_mmio_write_group: uefi as guest, write group=0
> >5.vgic_mmio_write_group: linux as guest, write group=1
>
> Is this the same issue fixed by EDK2 commit:
>
> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
> acknowledge")
>
> ... where EDK2 would try to use IAR0 rather than IAR1?
>
> The commit messages notes this lead to a boot-time hang.
> >>>
> >>>I managed to trigger an issue with a really old EFI implementation that
> >>>doesn't configure its interrupts as Group1, and yet tries to ACK its
> >>>interrupts using the Group1 accessor. Guess what? It is not going to work.
> >>>
> >>>Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
> >>>to be the fix (I only assume it does, I haven't actually checked). A
> >>>recent build, as found in Debian Buster, works perfectly (tested with
> >>>both QEMU v2.12 and tip of tree).
> >>>
> >>>Now, I really don't get what you're saying about Linux not getting
> >>>interrupts. How do you get to booting Linux if EFI is not making any
> >>>forward progress? Are you trying them independently?
> >>>
> >>I start linux with bypassing uefi, the print info is the same.
> >>[507107.748908] vgic_mmio_write_group:## intid/27 group=0
> >>[507107.752185] vgic_mmio_write_group:## intid/27 group=0
> >>[507107.899566] vgic_mmio_write_group:## intid/27 group=1
> >>[507107.907370] vgic_mmio_write_group:## intid/27 group=1
> >>the command line is like this:
> >>/home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
> >>virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
> >>/home/kernelboot/vmlinuz-4.16.0+ -initrd
> >>/home/kernelboot/initramfs-4.16.0+.img -append root=/dev/mapper/cla-root ro
> >>crashkernel=auto rd.lvm.lv=cla/root
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 21/11/18 11:06, Christoffer Dall wrote:
Hi,
On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
On 19/11/2018 09:10, Mark Rutland wrote:
On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
Hi,
When virtual machine starts, hang up.
I take it you mean the *guest* hangs? Because it doesn't get a timer
interrupt?
The kernel version of guest
is 4.16. Host support vgic_v3.
Your host kernel is something recent, I guess?
It was mainly due to the incorrect vgic_irq's(intid=27) group value
during injection interruption. when kvm_vgic_vcpu_init is called,
dist is not initialized at this time. Unable to get vgic V3 or V2
correctly, so group is not set.
Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
version?) or some other userland tool?
QEMU emulator version 3.0.50 .
group is setted to 1 when vgic_mmio_write_group is invoked at some
time.
when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
interrupt injection failed.
Signed-off-by: Peng Hao
---
virt/kvm/arm/vgic/vgic-v3.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 9c0dd23..d101000 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
(val & ICH_LR_PENDING_BIT)) irq->line_level = false;
-if (irq->group)
+if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
This is not the right fix, not only because it basically reverts the
GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
their configured group).
Can you try to work out why kvm_vgic_vcpu_init() is apparently called
before dist->vgic_model is set, also what value it has?
If I understand the code correctly, that shouldn't happen for a GICv3.
Even if the value of group is correctly assigned in kvm_vgic_vcpu_init, the
group is then written 0 through vgic_mmio_write_group.
If the interrupt comes at this time, the interrupt injection fails.
Does that mean that the guest is configuring its interrupts as Group0?
That sounds wrong, Linux should configure all it's interrupts as
non-secure group1.
no, I think that uefi dose this, not linux.
1. kvm_vgic_vcpu_init
2. vgic_create
3. kvm_vgic_dist_init
4.vgic_mmio_write_group: uefi as guest, write group=0
5.vgic_mmio_write_group: linux as guest, write group=1
Is this the same issue fixed by EDK2 commit:
66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
acknowledge")
... where EDK2 would try to use IAR0 rather than IAR1?
The commit messages notes this lead to a boot-time hang.
I managed to trigger an issue with a really old EFI implementation that
doesn't configure its interrupts as Group1, and yet tries to ACK its
interrupts using the Group1 accessor. Guess what? It is not going to work.
Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
to be the fix (I only assume it does, I haven't actually checked). A
recent build, as found in Debian Buster, works perfectly (tested with
both QEMU v2.12 and tip of tree).
Now, I really don't get what you're saying about Linux not getting
interrupts. How do you get to booting Linux if EFI is not making any
forward progress? Are you trying them independently?
I start linux with bypassing uefi, the print info is the same.
[507107.748908] vgic_mmio_write_group:## intid/27 group=0
[507107.752185] vgic_mmio_write_group:## intid/27 group=0
[507107.899566] vgic_mmio_write_group:## intid/27 group=1
[507107.907370] vgic_mmio_write_group:## intid/27 group=1
the command line is like this:
/home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
/home/kernelboot/vmlinuz-4.16.0+ -initrd /home/kernelboot/initramfs-4.16.0+.img
-append root=/dev/mapper/cla-root ro crashkernel=auto rd.lvm.lv=cla/root
rd.lvm.lv=cla/swap.UTF-8 -drive
file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-vnc 0.0.0.0:0 -k en-us -device
virtio-gpu-pci,id=video0,max_outputs=1,bus=pci.3,addr=0x0 -device pvpanic-mmio
-msg timestamp=on
This is strange. That's probably the Linux 4.16 kernel setting group to 0, and
I'll continue to track in guest.
Could you try the following patch:
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index c0c0b88af1d5..6fa858c7a5a6 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -231,13 +231,6 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
irq->config = VGIC_CONFIG_LEVEL;
}
- /*
-* GICv3 can only be created via the KVM_DEVICE_CREATE API and
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
Hi,
On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.h...@zte.com.cn wrote:
> >On 19/11/2018 09:10, Mark Rutland wrote:
> >> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> >> Hi,
> >>> When virtual machine starts, hang up.
> >>
> >> I take it you mean the *guest* hangs? Because it doesn't get a timer
> >> interrupt?
> >>
> >>> The kernel version of guest
> >>> is 4.16. Host support vgic_v3.
> >>
> >> Your host kernel is something recent, I guess?
> >>
> >>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
> >>> during injection interruption. when kvm_vgic_vcpu_init is called,
> >>> dist is not initialized at this time. Unable to get vgic V3 or V2
> >>> correctly, so group is not set.
> >>
> >> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> >> version?) or some other userland tool?
> >>
> >
> > QEMU emulator version 3.0.50 .
> >
> >>> group is setted to 1 when vgic_mmio_write_group is invoked at some
> >>> time.
> >>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
> >>> interrupt injection failed.
> >>>
> >>> Signed-off-by: Peng Hao
> >>> ---
> >>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> >>> index 9c0dd23..d101000 100644
> >>> --- a/virt/kvm/arm/vgic/vgic-v3.c
> >>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> >>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
> >>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
> >>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
> >>>
> >>> -if (irq->group)
> >>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
> >>
> >> This is not the right fix, not only because it basically reverts the
> >> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> >> their configured group).
> >>
> >> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
> >> before dist->vgic_model is set, also what value it has?
> >> If I understand the code correctly, that shouldn't happen for a GICv3.
> >>
> > Even if the value of group is correctly assigned in
> > kvm_vgic_vcpu_init, the group is then written 0 through
> > vgic_mmio_write_group.
> > If the interrupt comes at this time, the interrupt injection fails.
>
> Does that mean that the guest is configuring its interrupts as Group0?
> That sounds wrong, Linux should configure all it's interrupts as
> non-secure group1.
> >>>
> >>> no, I think that uefi dose this, not linux.
> >>> 1. kvm_vgic_vcpu_init
> >>> 2. vgic_create
> >>> 3. kvm_vgic_dist_init
> >>> 4.vgic_mmio_write_group: uefi as guest, write group=0
> >>> 5.vgic_mmio_write_group: linux as guest, write group=1
> >>
> >> Is this the same issue fixed by EDK2 commit:
> >>
> >> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
> >> acknowledge")
> >>
> >> ... where EDK2 would try to use IAR0 rather than IAR1?
> >>
> >> The commit messages notes this lead to a boot-time hang.
> >
> >I managed to trigger an issue with a really old EFI implementation that
> >doesn't configure its interrupts as Group1, and yet tries to ACK its
> >interrupts using the Group1 accessor. Guess what? It is not going to work.
> >
> >Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
> >to be the fix (I only assume it does, I haven't actually checked). A
> >recent build, as found in Debian Buster, works perfectly (tested with
> >both QEMU v2.12 and tip of tree).
> >
> >Now, I really don't get what you're saying about Linux not getting
> >interrupts. How do you get to booting Linux if EFI is not making any
> >forward progress? Are you trying them independently?
> >
> I start linux with bypassing uefi, the print info is the same.
> [507107.748908] vgic_mmio_write_group:## intid/27 group=0
> [507107.752185] vgic_mmio_write_group:## intid/27 group=0
> [507107.899566] vgic_mmio_write_group:## intid/27 group=1
> [507107.907370] vgic_mmio_write_group:## intid/27 group=1
> the command line is like this:
> /home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
> virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
> /home/kernelboot/vmlinuz-4.16.0+ -initrd
> /home/kernelboot/initramfs-4.16.0+.img -append root=/dev/mapper/cla-root ro
> crashkernel=auto rd.lvm.lv=cla/root rd.lvm.lv=cla/swap.UTF-8 -drive
> file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
> -device
> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -vnc 0.0.0.0:0 -k en-us -device
>
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>On 19/11/2018 09:10, Mark Rutland wrote:
>> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
>> Hi,
>>> When virtual machine starts, hang up.
>>
>> I take it you mean the *guest* hangs? Because it doesn't get a timer
>> interrupt?
>>
>>> The kernel version of guest
>>> is 4.16. Host support vgic_v3.
>>
>> Your host kernel is something recent, I guess?
>>
>>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>>> during injection interruption. when kvm_vgic_vcpu_init is called,
>>> dist is not initialized at this time. Unable to get vgic V3 or V2
>>> correctly, so group is not set.
>>
>> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
>> version?) or some other userland tool?
>>
>
> QEMU emulator version 3.0.50 .
>
>>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>>> time.
>>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>>> interrupt injection failed.
>>>
>>> Signed-off-by: Peng Hao
>>> ---
>>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>>> index 9c0dd23..d101000 100644
>>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>>>
>>> -if (irq->group)
>>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>>
>> This is not the right fix, not only because it basically reverts the
>> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
>> their configured group).
>>
>> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
>> before dist->vgic_model is set, also what value it has?
>> If I understand the code correctly, that shouldn't happen for a GICv3.
>>
> Even if the value of group is correctly assigned in kvm_vgic_vcpu_init,
> the group is then written 0 through vgic_mmio_write_group.
> If the interrupt comes at this time, the interrupt injection fails.
Does that mean that the guest is configuring its interrupts as Group0?
That sounds wrong, Linux should configure all it's interrupts as
non-secure group1.
>>>
>>> no, I think that uefi dose this, not linux.
>>> 1. kvm_vgic_vcpu_init
>>> 2. vgic_create
>>> 3. kvm_vgic_dist_init
>>> 4.vgic_mmio_write_group: uefi as guest, write group=0
>>> 5.vgic_mmio_write_group: linux as guest, write group=1
>>
>> Is this the same issue fixed by EDK2 commit:
>>
>> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
>> acknowledge")
>>
>> ... where EDK2 would try to use IAR0 rather than IAR1?
>>
>> The commit messages notes this lead to a boot-time hang.
>
>I managed to trigger an issue with a really old EFI implementation that
>doesn't configure its interrupts as Group1, and yet tries to ACK its
>interrupts using the Group1 accessor. Guess what? It is not going to work.
>
>Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
>to be the fix (I only assume it does, I haven't actually checked). A
>recent build, as found in Debian Buster, works perfectly (tested with
>both QEMU v2.12 and tip of tree).
>
>Now, I really don't get what you're saying about Linux not getting
>interrupts. How do you get to booting Linux if EFI is not making any
>forward progress? Are you trying them independently?
>
I start linux with bypassing uefi, the print info is the same.
[507107.748908] vgic_mmio_write_group:## intid/27 group=0
[507107.752185] vgic_mmio_write_group:## intid/27 group=0
[507107.899566] vgic_mmio_write_group:## intid/27 group=1
[507107.907370] vgic_mmio_write_group:## intid/27 group=1
the command line is like this:
/home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -machine
virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -kernel
/home/kernelboot/vmlinuz-4.16.0+ -initrd /home/kernelboot/initramfs-4.16.0+.img
-append root=/dev/mapper/cla-root ro crashkernel=auto rd.lvm.lv=cla/root
rd.lvm.lv=cla/swap.UTF-8 -drive
file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-vnc 0.0.0.0:0 -k en-us -device
virtio-gpu-pci,id=video0,max_outputs=1,bus=pci.3,addr=0x0 -device pvpanic-mmio
-msg timestamp=on
This is strange. That's probably the Linux 4.16 kernel setting group to 0, and
I'll continue to track in guest.
Thanks.
>Thanks,
>
>M.
>--
>Jazz is not dead. It just smells funny...___
kvmarm
Re:Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>On 19/11/2018 09:10, Mark Rutland wrote:
>> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
>> Hi,
>>> When virtual machine starts, hang up.
>>
>> I take it you mean the *guest* hangs? Because it doesn't get a timer
>> interrupt?
>>
>>> The kernel version of guest
>>> is 4.16. Host support vgic_v3.
>>
>> Your host kernel is something recent, I guess?
>>
>>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>>> during injection interruption. when kvm_vgic_vcpu_init is called,
>>> dist is not initialized at this time. Unable to get vgic V3 or V2
>>> correctly, so group is not set.
>>
>> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
>> version?) or some other userland tool?
>>
>
> QEMU emulator version 3.0.50 .
>
>>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>>> time.
>>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>>> interrupt injection failed.
>>>
>>> Signed-off-by: Peng Hao
>>> ---
>>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>>> index 9c0dd23..d101000 100644
>>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>>>
>>> -if (irq->group)
>>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>>
>> This is not the right fix, not only because it basically reverts the
>> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
>> their configured group).
>>
>> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
>> before dist->vgic_model is set, also what value it has?
>> If I understand the code correctly, that shouldn't happen for a GICv3.
>>
> Even if the value of group is correctly assigned in kvm_vgic_vcpu_init,
> the group is then written 0 through vgic_mmio_write_group.
> If the interrupt comes at this time, the interrupt injection fails.
Does that mean that the guest is configuring its interrupts as Group0?
That sounds wrong, Linux should configure all it's interrupts as
non-secure group1.
>>>
>>> no, I think that uefi dose this, not linux.
>>> 1. kvm_vgic_vcpu_init
>>> 2. vgic_create
>>> 3. kvm_vgic_dist_init
>>> 4.vgic_mmio_write_group: uefi as guest, write group=0
>>> 5.vgic_mmio_write_group: linux as guest, write group=1
>>
>> Is this the same issue fixed by EDK2 commit:
>>
>> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
>> acknowledge")
>>
>> ... where EDK2 would try to use IAR0 rather than IAR1?
>>
>> The commit messages notes this lead to a boot-time hang.
>
>I managed to trigger an issue with a really old EFI implementation that
>doesn't configure its interrupts as Group1, and yet tries to ACK its
>interrupts using the Group1 accessor. Guess what? It is not going to work.
>
>Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
>to be the fix (I only assume it does, I haven't actually checked). A
>recent build, as found in Debian Buster, works perfectly (tested with
>both QEMU v2.12 and tip of tree).
>
>Now, I really don't get what you're saying about Linux not getting
>interrupts. How do you get to booting Linux if EFI is not making any
>forward progress? Are you trying them independently?
>
I will update the UEFI version in recent days and test it later.
Thanks.
>Thanks,
>
>M.
>--
>Jazz is not dead. It just smells funny...___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 19/11/2018 09:10, Mark Rutland wrote:
> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
>>> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> Hi,
>> When virtual machine starts, hang up.
>
> I take it you mean the *guest* hangs? Because it doesn't get a timer
> interrupt?
>
>> The kernel version of guest
>> is 4.16. Host support vgic_v3.
>
> Your host kernel is something recent, I guess?
>
>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>> during injection interruption. when kvm_vgic_vcpu_init is called,
>> dist is not initialized at this time. Unable to get vgic V3 or V2
>> correctly, so group is not set.
>
> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> version?) or some other userland tool?
>
QEMU emulator version 3.0.50 .
>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>> time.
>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>> interrupt injection failed.
>>
>> Signed-off-by: Peng Hao
>> ---
>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> index 9c0dd23..d101000 100644
>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>>
>> -if (irq->group)
>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>
> This is not the right fix, not only because it basically reverts the
> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> their configured group).
>
> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
> before dist->vgic_model is set, also what value it has?
> If I understand the code correctly, that shouldn't happen for a GICv3.
>
Even if the value of group is correctly assigned in kvm_vgic_vcpu_init,
the group is then written 0 through vgic_mmio_write_group.
If the interrupt comes at this time, the interrupt injection fails.
>>>
>>> Does that mean that the guest is configuring its interrupts as Group0?
>>> That sounds wrong, Linux should configure all it's interrupts as
>>> non-secure group1.
>>
>> no, I think that uefi dose this, not linux.
>> 1. kvm_vgic_vcpu_init
>> 2. vgic_create
>> 3. kvm_vgic_dist_init
>> 4.vgic_mmio_write_group: uefi as guest, write group=0
>> 5.vgic_mmio_write_group: linux as guest, write group=1
>
> Is this the same issue fixed by EDK2 commit:
>
> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
> acknowledge")
>
> ... where EDK2 would try to use IAR0 rather than IAR1?
>
> The commit messages notes this lead to a boot-time hang.
I managed to trigger an issue with a really old EFI implementation that
doesn't configure its interrupts as Group1, and yet tries to ACK its
interrupts using the Group1 accessor. Guess what? It is not going to work.
Commit c7fefb690661f2e38afcb8200bd318ecf38ab961 in the edk2 tree seems
to be the fix (I only assume it does, I haven't actually checked). A
recent build, as found in Debian Buster, works perfectly (tested with
both QEMU v2.12 and tip of tree).
Now, I really don't get what you're saying about Linux not getting
interrupts. How do you get to booting Linux if EFI is not making any
forward progress? Are you trying them independently?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 19/11/2018 09:10, Mark Rutland wrote:
> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
>>> On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> Hi,
>> When virtual machine starts, hang up.
>
> I take it you mean the *guest* hangs? Because it doesn't get a timer
> interrupt?
>
>> The kernel version of guest
>> is 4.16. Host support vgic_v3.
>
> Your host kernel is something recent, I guess?
>
>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>> during injection interruption. when kvm_vgic_vcpu_init is called,
>> dist is not initialized at this time. Unable to get vgic V3 or V2
>> correctly, so group is not set.
>
> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> version?) or some other userland tool?
>
QEMU emulator version 3.0.50 .
>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>> time.
>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>> interrupt injection failed.
>>
>> Signed-off-by: Peng Hao
>> ---
>> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> index 9c0dd23..d101000 100644
>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>>
>> -if (irq->group)
>> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
>
> This is not the right fix, not only because it basically reverts the
> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> their configured group).
>
> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
> before dist->vgic_model is set, also what value it has?
> If I understand the code correctly, that shouldn't happen for a GICv3.
>
Even if the value of group is correctly assigned in kvm_vgic_vcpu_init,
the group is then written 0 through vgic_mmio_write_group.
If the interrupt comes at this time, the interrupt injection fails.
>>>
>>> Does that mean that the guest is configuring its interrupts as Group0?
>>> That sounds wrong, Linux should configure all it's interrupts as
>>> non-secure group1.
>>
>> no, I think that uefi dose this, not linux.
>> 1. kvm_vgic_vcpu_init
>> 2. vgic_create
>> 3. kvm_vgic_dist_init
>> 4.vgic_mmio_write_group: uefi as guest, write group=0
>> 5.vgic_mmio_write_group: linux as guest, write group=1
>
> Is this the same issue fixed by EDK2 commit:
>
> 66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
> acknowledge")
>
> ... where EDK2 would try to use IAR0 rather than IAR1?
>
> The commit messages notes this lead to a boot-time hang.
I doubt it. The EDK2 bug resulted in no interrupt being delivered due to
the group being set to 1, and the interrupt being acknowledged through
the Group0 accessor. Also, this was a 32bit special, and 64bit EDK2 was
just fine.
What Peng describes here is the guest writing the group as 0, which
doesn't make much sense (I certainly don't see EDK2 doing that).
Thanks,
M.
--
Jazz is not dead. It just smells funny...
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.h...@zte.com.cn wrote:
> >On 16/11/18 00:23, peng.h...@zte.com.cn wrote:
> >>> Hi,
> When virtual machine starts, hang up.
> >>>
> >>> I take it you mean the *guest* hangs? Because it doesn't get a timer
> >>> interrupt?
> >>>
> The kernel version of guest
> is 4.16. Host support vgic_v3.
> >>>
> >>> Your host kernel is something recent, I guess?
> >>>
> It was mainly due to the incorrect vgic_irq's(intid=27) group value
> during injection interruption. when kvm_vgic_vcpu_init is called,
> dist is not initialized at this time. Unable to get vgic V3 or V2
> correctly, so group is not set.
> >>>
> >>> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
> >>> version?) or some other userland tool?
> >>>
> >>
> >> QEMU emulator version 3.0.50 .
> >>
> group is setted to 1 when vgic_mmio_write_group is invoked at some
> time.
> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
> interrupt injection failed.
>
> Signed-off-by: Peng Hao
> ---
> virt/kvm/arm/vgic/vgic-v3.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> index 9c0dd23..d101000 100644
> --- a/virt/kvm/arm/vgic/vgic-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>
> -if (irq->group)
> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3)
> >>>
> >>> This is not the right fix, not only because it basically reverts the
> >>> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using
> >>> their configured group).
> >>>
> >>> Can you try to work out why kvm_vgic_vcpu_init() is apparently called
> >>> before dist->vgic_model is set, also what value it has?
> >>> If I understand the code correctly, that shouldn't happen for a GICv3.
> >>>
> >> Even if the value of group is correctly assigned in kvm_vgic_vcpu_init,
> >> the group is then written 0 through vgic_mmio_write_group.
> >> If the interrupt comes at this time, the interrupt injection fails.
> >
> >Does that mean that the guest is configuring its interrupts as Group0?
> >That sounds wrong, Linux should configure all it's interrupts as
> >non-secure group1.
>
> no, I think that uefi dose this, not linux.
> 1. kvm_vgic_vcpu_init
> 2. vgic_create
> 3. kvm_vgic_dist_init
> 4.vgic_mmio_write_group: uefi as guest, write group=0
> 5.vgic_mmio_write_group: linux as guest, write group=1
Is this the same issue fixed by EDK2 commit:
66127011a544b90e ("ArmPkg/ArmGicDxe ARM: fix encoding for GICv3 interrupt
acknowledge")
... where EDK2 would try to use IAR0 rather than IAR1?
The commit messages notes this lead to a boot-time hang.
Thanks,
Mark.
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re:Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>On 16/11/18 00:23, peng.h...@zte.com.cn wrote: >>> Hi, When virtual machine starts, hang up. >>> >>> I take it you mean the *guest* hangs? Because it doesn't get a timer >>> interrupt? >>> The kernel version of guest is 4.16. Host support vgic_v3. >>> >>> Your host kernel is something recent, I guess? >>> It was mainly due to the incorrect vgic_irq's(intid=27) group value during injection interruption. when kvm_vgic_vcpu_init is called, dist is not initialized at this time. Unable to get vgic V3 or V2 correctly, so group is not set. >>> >>> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which >>> version?) or some other userland tool? >>> >> >> QEMU emulator version 3.0.50 . >> group is setted to 1 when vgic_mmio_write_group is invoked at some time. when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and interrupt injection failed. Signed-off-by: Peng Hao --- virt/kvm/arm/vgic/vgic-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index 9c0dd23..d101000 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) irq->line_level = false; -if (irq->group) +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) >>> >>> This is not the right fix, not only because it basically reverts the >>> GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using >>> their configured group). >>> >>> Can you try to work out why kvm_vgic_vcpu_init() is apparently called >>> before dist->vgic_model is set, also what value it has? >>> If I understand the code correctly, that shouldn't happen for a GICv3. >>> >> Even if the value of group is correctly assigned in kvm_vgic_vcpu_init, the >> group is then written 0 through vgic_mmio_write_group. >> If the interrupt comes at this time, the interrupt injection fails. >> > >Does that mean that the guest is configuring its interrupts as Group0? >That sounds wrong, Linux should configure all it's interrupts as >non-secure group1. no, I think that uefi dose this, not linux. 1. kvm_vgic_vcpu_init 2. vgic_create 3. kvm_vgic_dist_init 4.vgic_mmio_write_group: uefi as guest, write group=0 5.vgic_mmio_write_group: linux as guest, write group=1 > >-- >Julien Thierry___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 16/11/18 00:23, peng.h...@zte.com.cn wrote: Hi, When virtual machine starts, hang up. I take it you mean the *guest* hangs? Because it doesn't get a timer interrupt? The kernel version of guest is 4.16. Host support vgic_v3. Your host kernel is something recent, I guess? It was mainly due to the incorrect vgic_irq's(intid=27) group value during injection interruption. when kvm_vgic_vcpu_init is called, dist is not initialized at this time. Unable to get vgic V3 or V2 correctly, so group is not set. Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which version?) or some other userland tool? QEMU emulator version 3.0.50 . group is setted to 1 when vgic_mmio_write_group is invoked at some time. when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and interrupt injection failed. Signed-off-by: Peng Hao --- virt/kvm/arm/vgic/vgic-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index 9c0dd23..d101000 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) irq->line_level = false; -if (irq->group) +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) This is not the right fix, not only because it basically reverts the GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using their configured group). Can you try to work out why kvm_vgic_vcpu_init() is apparently called before dist->vgic_model is set, also what value it has? If I understand the code correctly, that shouldn't happen for a GICv3. Even if the value of group is correctly assigned in kvm_vgic_vcpu_init, the group is then written 0 through vgic_mmio_write_group. If the interrupt comes at this time, the interrupt injection fails. Does that mean that the guest is configuring its interrupts as Group0? That sounds wrong, Linux should configure all it's interrupts as non-secure group1. -- Julien Thierry ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re:Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>> >> When virtual machine starts, hang up. The kernel version of guest >> is 4.16. Host support vgic_v3. >> It was mainly due to the incorrect vgic_irq's(intid=27) group value >> during injection interruption. when kvm_vgic_vcpu_init is called, >> dist is not initialized at this time. Unable to get vgic V3 or V2 >> correctly, so group is not set. >> group is setted to 1 when vgic_mmio_write_group is invoked at some >> time. >> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and >> interrupt injection failed. >> >> Signed-off-by: Peng Hao >> --- >> virt/kvm/arm/vgic/vgic-v3.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c >> index 9c0dd23..d101000 100644 >> --- a/virt/kvm/arm/vgic/vgic-v3.c >> +++ b/virt/kvm/arm/vgic/vgic-v3.c >> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct >> vgic_irq *irq, int lr) >> if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) >> irq->line_level = false; >> >> -if (irq->group) >> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) >> val |= ICH_LR_GROUP; >> >> val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; > >This patch makes exactly zero sense. The group is a property of the >interrupt, and has nothing to do with the model used for the emulation > in kvm_vgic_vcpu_init, the value of group is dependent on vgic_model. >Which version of the kernel is this against? What userspace are you >using? How can I reproduce this issue? QEMU emulator version 3.0.50 . host kernel: 4.19.0-rc4+ guest kernel :4.16.0 when starting the virtual machine, it will happen. Thanks. > >Thanks, > >M. > >-- >Jazz is not dead, it just smell funny.___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re:Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>Hi, >> When virtual machine starts, hang up. > >I take it you mean the *guest* hangs? Because it doesn't get a timer >interrupt? > >> The kernel version of guest >> is 4.16. Host support vgic_v3. > >Your host kernel is something recent, I guess? > >> It was mainly due to the incorrect vgic_irq's(intid=27) group value >> during injection interruption. when kvm_vgic_vcpu_init is called, >> dist is not initialized at this time. Unable to get vgic V3 or V2 >> correctly, so group is not set. > >Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which >version?) or some other userland tool? > QEMU emulator version 3.0.50 . >> group is setted to 1 when vgic_mmio_write_group is invoked at some >> time. >> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and >> interrupt injection failed. >> >> Signed-off-by: Peng Hao >> --- >> virt/kvm/arm/vgic/vgic-v3.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c >> index 9c0dd23..d101000 100644 >> --- a/virt/kvm/arm/vgic/vgic-v3.c >> +++ b/virt/kvm/arm/vgic/vgic-v3.c >> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, >> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && >> (val & ICH_LR_PENDING_BIT)) irq->line_level = false; >> >> -if (irq->group) >> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) > >This is not the right fix, not only because it basically reverts the >GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using >their configured group). > >Can you try to work out why kvm_vgic_vcpu_init() is apparently called >before dist->vgic_model is set, also what value it has? >If I understand the code correctly, that shouldn't happen for a GICv3. > Even if the value of group is correctly assigned in kvm_vgic_vcpu_init, the group is then written 0 through vgic_mmio_write_group. If the interrupt comes at this time, the interrupt injection fails. >Cheers, >Andre. > >> val |= ICH_LR_GROUP; >> >> val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT;___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Thu, 15 Nov 2018 15:14:08 +, Peng Hao wrote: > > When virtual machine starts, hang up. The kernel version of guest > is 4.16. Host support vgic_v3. > It was mainly due to the incorrect vgic_irq's(intid=27) group value > during injection interruption. when kvm_vgic_vcpu_init is called, > dist is not initialized at this time. Unable to get vgic V3 or V2 > correctly, so group is not set. > group is setted to 1 when vgic_mmio_write_group is invoked at some > time. > when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and > interrupt injection failed. > > Signed-off-by: Peng Hao > --- > virt/kvm/arm/vgic/vgic-v3.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c > index 9c0dd23..d101000 100644 > --- a/virt/kvm/arm/vgic/vgic-v3.c > +++ b/virt/kvm/arm/vgic/vgic-v3.c > @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct > vgic_irq *irq, int lr) > if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) > irq->line_level = false; > > - if (irq->group) > + if (model == KVM_DEV_TYPE_ARM_VGIC_V3) > val |= ICH_LR_GROUP; > > val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; This patch makes exactly zero sense. The group is a property of the interrupt, and has nothing to do with the model used for the emulation Which version of the kernel is this against? What userspace are you using? How can I reproduce this issue? Thanks, M. -- Jazz is not dead, it just smell funny. ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On Thu, 15 Nov 2018 23:14:08 +0800 Peng Hao wrote: Hi, > When virtual machine starts, hang up. I take it you mean the *guest* hangs? Because it doesn't get a timer interrupt? > The kernel version of guest > is 4.16. Host support vgic_v3. Your host kernel is something recent, I guess? > It was mainly due to the incorrect vgic_irq's(intid=27) group value > during injection interruption. when kvm_vgic_vcpu_init is called, > dist is not initialized at this time. Unable to get vgic V3 or V2 > correctly, so group is not set. Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which version?) or some other userland tool? > group is setted to 1 when vgic_mmio_write_group is invoked at some > time. > when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and > interrupt injection failed. > > Signed-off-by: Peng Hao > --- > virt/kvm/arm/vgic/vgic-v3.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c > index 9c0dd23..d101000 100644 > --- a/virt/kvm/arm/vgic/vgic-v3.c > +++ b/virt/kvm/arm/vgic/vgic-v3.c > @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, > struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && > (val & ICH_LR_PENDING_BIT)) irq->line_level = false; > > - if (irq->group) > + if (model == KVM_DEV_TYPE_ARM_VGIC_V3) This is not the right fix, not only because it basically reverts the GICv3 part of 87322099052 (KVM: arm/arm64: vgic: Signal IRQs using their configured group). Can you try to work out why kvm_vgic_vcpu_init() is apparently called before dist->vgic_model is set, also what value it has? If I understand the code correctly, that shouldn't happen for a GICv3. Cheers, Andre. > val |= ICH_LR_GROUP; > > val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
On 15/11/18 10:22, peng.h...@zte.com.cn wrote: Hi Peng, On 15/11/18 15:14, Peng Hao wrote: When virtual machine starts, hang up. The kernel version of guest is 4.16. Host support vgic_v3. I don't understand the issue here. the vgic_irq->group field does not exist in 4.16 (neither in the first nor the last version). Nor does the line you are modifying. Guest's kernel version is 4.16 and Host's kernel version is mainline. vgic_irq is a structure of Host's kvm. Both exist in mainline, but what you are mentioning shouldn't be an issue in mainline since vgic_irq->group gets initialized in kvm_vgic_vcpu_init for SGIs and PPIs (i.e. intid < 32) which includes your interrupt. Initialization sequence of several components is like this; 1. kvm_vgic_vcpu_init here, dist is not initialized at this time. and the value of irq->group depends on dist->vgic_model. but now dist->vgic_model = 0 and irq->group=0. 2. vgic_create - assign to dist->vgic_model. 3. kvm_vgic_dist_init I see, thanks for explaining. This means the initialization in kvm_vgic_vcpu_init depends on whether userland creates the GIC device first or the vcpus first... I don't think the issue lies in vgic_v3_populate_lr, but in kvm_vgic_vcpu_init which clearly expects dist->vgic_model to be initialized in two places. However, the comment at the top of vgic-init.c states: * CPU Interface: * * - kvm_vgic_vcpu_init(): initialization of static data that * doesn't depend on any sizing information or emulation type. No * allocation is allowed there. So I don't think we're supposed to check for dist->vgic_model here. Those two checks should probably be moved elsewhere (maybe in kvm_vgic_vcpu_enable?) once the gic type has been set. Thanks, thanks. Cheers, It was mainly due to the incorrect vgic_irq's(intid=27) group value during injection interruption. when kvm_vgic_vcpu_init is called, dist is not initialized at this time. Unable to get vgic V3 or V2 correctly, so group is not set. group is setted to 1 when vgic_mmio_write_group is invoked at some time. when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and interrupt injection failed. Signed-off-by: Peng Hao --- virt/kvm/arm/vgic/vgic-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index 9c0dd23..d101000 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) irq->line_level = false; -if (irq->group) +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) val |= ICH_LR_GROUP; val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; -- Julien Thierry -- Julien Thierry ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re:Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
>Hi Peng, > >On 15/11/18 15:14, Peng Hao wrote: >> When virtual machine starts, hang up. The kernel version of guest >> is 4.16. Host support vgic_v3. > >I don't understand the issue here. the vgic_irq->group field does not >exist in 4.16 (neither in the first nor the last version). Nor does the >line you are modifying. Guest's kernel version is 4.16 and Host's kernel version is mainline. vgic_irq is a structure of Host's kvm. > >Both exist in mainline, but what you are mentioning shouldn't be an >issue in mainline since vgic_irq->group gets initialized in >kvm_vgic_vcpu_init for SGIs and PPIs (i.e. intid < 32) which includes >your interrupt. Initialization sequence of several components is like this; 1. kvm_vgic_vcpu_init here, dist is not initialized at this time. and the value of irq->group depends on dist->vgic_model. but now dist->vgic_model = 0 and irq->group=0. 2. vgic_create - assign to dist->vgic_model. 3. kvm_vgic_dist_init thanks. > >Cheers, > >> It was mainly due to the incorrect vgic_irq's(intid=27) group value >> during injection interruption. when kvm_vgic_vcpu_init is called, >> dist is not initialized at this time. Unable to get vgic V3 or V2 >> correctly, so group is not set. >> group is setted to 1 when vgic_mmio_write_group is invoked at some >> time. >> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and >> interrupt injection failed. >> >> Signed-off-by: Peng Hao >> --- >> virt/kvm/arm/vgic/vgic-v3.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c >> index 9c0dd23..d101000 100644 >> --- a/virt/kvm/arm/vgic/vgic-v3.c >> +++ b/virt/kvm/arm/vgic/vgic-v3.c >> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct >> vgic_irq *irq, int lr) >> if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) >> irq->line_level = false; >> >> -if (irq->group) >> +if (model == KVM_DEV_TYPE_ARM_VGIC_V3) >> val |= ICH_LR_GROUP; >> >> val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; >> > >-- >Julien Thierry___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] kvm: arm/arm64 : fix vm's hanging at startup time
Hi Peng, On 15/11/18 15:14, Peng Hao wrote: When virtual machine starts, hang up. The kernel version of guest is 4.16. Host support vgic_v3. I don't understand the issue here. the vgic_irq->group field does not exist in 4.16 (neither in the first nor the last version). Nor does the line you are modifying. Both exist in mainline, but what you are mentioning shouldn't be an issue in mainline since vgic_irq->group gets initialized in kvm_vgic_vcpu_init for SGIs and PPIs (i.e. intid < 32) which includes your interrupt. Cheers, It was mainly due to the incorrect vgic_irq's(intid=27) group value during injection interruption. when kvm_vgic_vcpu_init is called, dist is not initialized at this time. Unable to get vgic V3 or V2 correctly, so group is not set. group is setted to 1 when vgic_mmio_write_group is invoked at some time. when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and interrupt injection failed. Signed-off-by: Peng Hao --- virt/kvm/arm/vgic/vgic-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index 9c0dd23..d101000 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT)) irq->line_level = false; - if (irq->group) + if (model == KVM_DEV_TYPE_ARM_VGIC_V3) val |= ICH_LR_GROUP; val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT; -- Julien Thierry ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
