Re: [PATCH] KVM: Add system call KVM_VERIFY_MSI to verify MSI vector
Hi Marc, 在 2022/11/10 18:28, Marc Zyngier 写道: On Wed, 09 Nov 2022 06:21:18 +, "chenxiang (M)" wrote: Hi Marc, 在 2022/11/8 20:47, Marc Zyngier 写道: On Tue, 08 Nov 2022 08:08:57 +, chenxiang wrote: From: Xiang Chen Currently the numbers of MSI vectors come from register PCI_MSI_FLAGS which should be power-of-2, but in some scenaries it is not the same as the number that driver requires in guest, for example, a PCI driver wants to allocate 6 MSI vecotrs in guest, but as the limitation, it will allocate 8 MSI vectors. So it requires 8 MSI vectors in qemu while the driver in guest only wants to allocate 6 MSI vectors. When GICv4.1 is enabled, we can see some exception print as following for above scenaro: vfio-pci :3a:00.1: irq bypass producer (token 8f08224d) registration fails:66311 In order to verify whether a MSI vector is valid, add KVM_VERIFY_MSI to do that. If there is a mapping, return 0, otherwise return negative value. This is the kernel part of adding system call KVM_VERIFY_MSI. Exposing something that is an internal implementation detail to userspace feels like the absolute wrong way to solve this issue. Can you please characterise the issue you're having? Is it that vfio tries to enable an interrupt for which there is no virtual ITS mapping? Shouldn't we instead try and manage this in the kernel? Before i reported the issue to community, you gave a suggestion about the issue, but not sure whether i misundertood your meaning. You can refer to the link for more details about the issue. https://lkml.kernel.org/lkml/87cze9lcut.wl-...@kernel.org/T/ Right. It would have been helpful to mention this earlier. Anyway, I would really like this to be done without involving userspace at all. But first, can you please confirm that the VM works as expected despite the message? Yes, it works well except the message. If that's the case, we only need to handle the case where this is a multi-MSI setup, and I think this can be done in VFIO, without involving userspace. It seems we can verify every kvm_msi for multi-MSI setup in function vfio_pci_set_msi_trigger(). If it is a invalid MSI vector, then we can decrease the numer of MSI vectors before calling vfio_msi_set_block(). Thanks, M.
Re: [PATCH] KVM: Add system call KVM_VERIFY_MSI to verify MSI vector
On Wed, 09 Nov 2022 06:21:18 +, "chenxiang (M)" wrote: > > Hi Marc, > > > 在 2022/11/8 20:47, Marc Zyngier 写道: > > On Tue, 08 Nov 2022 08:08:57 +, > > chenxiang wrote: > >> From: Xiang Chen > >> > >> Currently the numbers of MSI vectors come from register PCI_MSI_FLAGS > >> which should be power-of-2, but in some scenaries it is not the same as > >> the number that driver requires in guest, for example, a PCI driver wants > >> to allocate 6 MSI vecotrs in guest, but as the limitation, it will allocate > >> 8 MSI vectors. So it requires 8 MSI vectors in qemu while the driver in > >> guest only wants to allocate 6 MSI vectors. > >> > >> When GICv4.1 is enabled, we can see some exception print as following for > >> above scenaro: > >> vfio-pci :3a:00.1: irq bypass producer (token 8f08224d) > >> registration fails:66311 > >> > >> In order to verify whether a MSI vector is valid, add KVM_VERIFY_MSI to do > >> that. If there is a mapping, return 0, otherwise return negative value. > >> > >> This is the kernel part of adding system call KVM_VERIFY_MSI. > > Exposing something that is an internal implementation detail to > > userspace feels like the absolute wrong way to solve this issue. > > > > Can you please characterise the issue you're having? Is it that vfio > > tries to enable an interrupt for which there is no virtual ITS > > mapping? Shouldn't we instead try and manage this in the kernel? > > Before i reported the issue to community, you gave a suggestion about > the issue, but not sure whether i misundertood your meaning. > You can refer to the link for more details about the issue. > https://lkml.kernel.org/lkml/87cze9lcut.wl-...@kernel.org/T/ Right. It would have been helpful to mention this earlier. Anyway, I would really like this to be done without involving userspace at all. But first, can you please confirm that the VM works as expected despite the message? If that's the case, we only need to handle the case where this is a multi-MSI setup, and I think this can be done in VFIO, without involving userspace. Thanks, M. -- Without deviation from the norm, progress is not possible.
Re: [PATCH] KVM: Add system call KVM_VERIFY_MSI to verify MSI vector
Hi Marc, 在 2022/11/8 20:47, Marc Zyngier 写道: On Tue, 08 Nov 2022 08:08:57 +, chenxiang wrote: From: Xiang Chen Currently the numbers of MSI vectors come from register PCI_MSI_FLAGS which should be power-of-2, but in some scenaries it is not the same as the number that driver requires in guest, for example, a PCI driver wants to allocate 6 MSI vecotrs in guest, but as the limitation, it will allocate 8 MSI vectors. So it requires 8 MSI vectors in qemu while the driver in guest only wants to allocate 6 MSI vectors. When GICv4.1 is enabled, we can see some exception print as following for above scenaro: vfio-pci :3a:00.1: irq bypass producer (token 8f08224d) registration fails:66311 In order to verify whether a MSI vector is valid, add KVM_VERIFY_MSI to do that. If there is a mapping, return 0, otherwise return negative value. This is the kernel part of adding system call KVM_VERIFY_MSI. Exposing something that is an internal implementation detail to userspace feels like the absolute wrong way to solve this issue. Can you please characterise the issue you're having? Is it that vfio tries to enable an interrupt for which there is no virtual ITS mapping? Shouldn't we instead try and manage this in the kernel? Before i reported the issue to community, you gave a suggestion about the issue, but not sure whether i misundertood your meaning. You can refer to the link for more details about the issue. https://lkml.kernel.org/lkml/87cze9lcut.wl-...@kernel.org/T/ Best regards, Xiang
Re: [PATCH] KVM: Add system call KVM_VERIFY_MSI to verify MSI vector
On Tue, 08 Nov 2022 08:08:57 +, chenxiang wrote: > > From: Xiang Chen > > Currently the numbers of MSI vectors come from register PCI_MSI_FLAGS > which should be power-of-2, but in some scenaries it is not the same as > the number that driver requires in guest, for example, a PCI driver wants > to allocate 6 MSI vecotrs in guest, but as the limitation, it will allocate > 8 MSI vectors. So it requires 8 MSI vectors in qemu while the driver in > guest only wants to allocate 6 MSI vectors. > > When GICv4.1 is enabled, we can see some exception print as following for > above scenaro: > vfio-pci :3a:00.1: irq bypass producer (token 8f08224d) > registration fails:66311 > > In order to verify whether a MSI vector is valid, add KVM_VERIFY_MSI to do > that. If there is a mapping, return 0, otherwise return negative value. > > This is the kernel part of adding system call KVM_VERIFY_MSI. Exposing something that is an internal implementation detail to userspace feels like the absolute wrong way to solve this issue. Can you please characterise the issue you're having? Is it that vfio tries to enable an interrupt for which there is no virtual ITS mapping? Shouldn't we instead try and manage this in the kernel? Thanks, M. -- Without deviation from the norm, progress is not possible.
[PATCH] KVM: Add system call KVM_VERIFY_MSI to verify MSI vector
From: Xiang Chen
Currently the numbers of MSI vectors come from register PCI_MSI_FLAGS
which should be power-of-2, but in some scenaries it is not the same as
the number that driver requires in guest, for example, a PCI driver wants
to allocate 6 MSI vecotrs in guest, but as the limitation, it will allocate
8 MSI vectors. So it requires 8 MSI vectors in qemu while the driver in
guest only wants to allocate 6 MSI vectors.
When GICv4.1 is enabled, we can see some exception print as following for
above scenaro:
vfio-pci :3a:00.1: irq bypass producer (token 8f08224d)
registration fails:66311
In order to verify whether a MSI vector is valid, add KVM_VERIFY_MSI to do
that. If there is a mapping, return 0, otherwise return negative value.
This is the kernel part of adding system call KVM_VERIFY_MSI.
Signed-off-by: Xiang Chen
---
arch/arm64/kvm/vgic/vgic-irqfd.c | 5 +
arch/arm64/kvm/vgic/vgic-its.c | 36
arch/arm64/kvm/vgic/vgic.h | 1 +
include/linux/kvm_host.h | 2 +-
include/uapi/linux/kvm.h | 2 ++
virt/kvm/kvm_main.c | 9 +
6 files changed, 54 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/kvm/vgic/vgic-irqfd.c b/arch/arm64/kvm/vgic/vgic-irqfd.c
index 475059b..2312da6 100644
--- a/arch/arm64/kvm/vgic/vgic-irqfd.c
+++ b/arch/arm64/kvm/vgic/vgic-irqfd.c
@@ -98,6 +98,11 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
return vgic_its_inject_msi(kvm, &msi);
}
+int kvm_verify_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+ return vgic_its_verify_msi(kvm, msi);
+}
+
/**
* kvm_arch_set_irq_inatomic: fast-path for irqfd injection
*/
diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c
index 24d7778..cae6183 100644
--- a/arch/arm64/kvm/vgic/vgic-its.c
+++ b/arch/arm64/kvm/vgic/vgic-its.c
@@ -767,6 +767,42 @@ int vgic_its_inject_cached_translation(struct kvm *kvm,
struct kvm_msi *msi)
return 0;
}
+int vgic_its_verify_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+ struct vgic_its *its;
+ struct its_ite *ite;
+ struct kvm_vcpu *vcpu;
+ int ret = 0;
+
+ if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
+ return -EINVAL;
+
+ if (!vgic_has_its(kvm))
+ return -ENODEV;
+
+ its = vgic_msi_to_its(kvm, msi);
+ if (IS_ERR(its))
+ return PTR_ERR(its);
+
+ mutex_lock(&its->its_lock);
+ if (!its->enabled) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+ ite = find_ite(its, msi->devid, msi->data);
+ if (!ite || !its_is_collection_mapped(ite->collection)) {
+ ret = -E_ITS_INT_UNMAPPED_INTERRUPT;
+ goto unlock;
+ }
+
+ vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
+ if (!vcpu)
+ ret = -E_ITS_INT_UNMAPPED_INTERRUPT;
+unlock:
+ mutex_unlock(&its->its_lock);
+ return ret;
+}
+
/*
* Queries the KVM IO bus framework to get the ITS pointer from the given
* doorbell address.
diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h
index 0c8da72..d452150 100644
--- a/arch/arm64/kvm/vgic/vgic.h
+++ b/arch/arm64/kvm/vgic/vgic.h
@@ -240,6 +240,7 @@ int kvm_vgic_register_its_device(void);
void vgic_enable_lpis(struct kvm_vcpu *vcpu);
void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu);
int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
+int vgic_its_verify_msi(struct kvm *kvm, struct kvm_msi *msi);
int vgic_v3_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr
*attr);
int vgic_v3_dist_uaccess(struct kvm_vcpu *vcpu, bool is_write,
int offset, u32 *val);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 32f259f..7923352 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1597,7 +1597,7 @@ void kvm_unregister_irq_ack_notifier(struct kvm *kvm,
int kvm_request_irq_source_id(struct kvm *kvm);
void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id);
bool kvm_arch_irqfd_allowed(struct kvm *kvm, struct kvm_irqfd *args);
-
+int kvm_verify_msi(struct kvm *kvm, struct kvm_msi *msi);
/*
* Returns a pointer to the memslot if it contains gfn.
* Otherwise returns NULL.
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 0d5d441..72b28f8 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1543,6 +1543,8 @@ struct kvm_s390_ucas_mapping {
#define KVM_PPC_SVM_OFF _IO(KVMIO, 0xb3)
#define KVM_ARM_MTE_COPY_TAGS_IOR(KVMIO, 0xb4, struct
kvm_arm_copy_mte_tags)
+#define KVM_VERIFY_MSI_IOW(KVMIO, 0xb5, struct kvm_msi)
+
/* ioctl for vm fd */
#define KVM_CREATE_DEVICE_IOWR(KVMIO, 0xe0, struct kvm_create_device)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index e30f1b4..439bdd7 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_m
