Hello,
I would like to paravirtualize several instances of a minimal Linux solution
with xen. Guest OS have a PCI passthrough access on the ethernet interfaces.
I use a pvops kernel 4.4.122 , xen 4.11.2 and an x86 platform. My hardware
platform doesn't support IOMMU but I doesn't need hard segmentation between OS
with this platform.
I got a bug when the Ethernet driver configures the MSI for a domU from a domU.
pciback 0000:00:14.3: xen map irq failed -19 for 1 domain
Please find the bug description and maybe a solution :
1) dom1 is configured to use 0000:00:14.3
in file /etc/xen/dom1 :
pci=['00:14.3,permissive=1,seize=1,msitranslate=1']
and kernel bootparam include "iommu=soft"
2) xen-pciback is loaded in dom0 :
> modprobe xen-pciback passthrough=1 hide='(0000:00:14.3)'
> xl pci-assignable-list
0000:00:14.3
(xen-pciback is not loaded in dom1)
3) Xen debug terminal print that all the PCI devices are linked to dom0 before
and after xen-pciback probing
...
(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
...
4) no error are detected when I start dom1
> xl create /etc/xen/dom1
or with
> xl pci-attach dom1 00:14.3
5) For each Ethernet device I see the following call to the hypervisor
- xc_domain_iomem_permission
- xc_domain_ioport_permission
- xc_domain_iomem_permission
- xc_physdev_map_pirq
- xc_domain_irq_permission
- xc_assign_device
none of them update the pdev->domain or the pdev->domain->domain_id of the pdev
element linked with the bfd 0000:00:14.3 (devfn 163)
6) Xen debug terminal prints again that all the PCI devices are linked to dom0
...
(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
...
I guess that this is normal because Dom0 will allow dom1 to configure the
device (IRQ & DMA) thanks to xen-pciback. Could you confirm me this assertion ?
7) When the Ethernet driver (igb) of dom1 configure MSI. xen failed to map MSI
irqs because map_domain_pirq call pci_get_pdev_by_domain with domain_id 1 and
the device is linked with domain_id 0
> modprobe igb
igb: Copyright (c) 2007-2014 Intel Corporation.
igb 0000:00:14.3: enabling device (0000 -> 0002)
igb 0000:00:14.3: Xen PCI mapped GSI23 to IRQ25
en_initdom_setup_msi_irqs:xen map irq map_irq.domid 3, map_irq.type 3
map_irq.index -1 , map_irq.pirq -1
xen_initdom_setup_msi_irqs:xen map irq map_irq.bus 0, map_irq.devfn 163
pciback 0000:00:14.3: PCI_CAP_ID_MSIX
pci_seg_supported
There,HYPERVISOR_physdev_op and PHYSDEVOP_map_pirq call map_domain_pirq :
(XEN) map_domain_pirq--> call pci_get_pdev_by_domain
(XEN) pci_get_pdev_by_domain-->
(XEN) domain->domain_id 1
(XEN) seg 0 bus 0 devfn 163
...
Unfortunately the device is already mapped to domain 0 and
pci_get_pdev_by_domain don't found the device ... So map_domain_pirq &
HYPERVISOR_physdev_op return -19 :
pciback 0000:00:14.3: xen map irq failed -19 for 2 domain
I don't find any hypervisor syscall able to assign my device to a domU.
xc_assign_device is cancelled because iommu is disabled.
libxl_pci.c:1137:do_pci_add: Domain 3:do_pci_add --> call xc_assign_device ->
hypervisor call arch_do_domctl--> iommu_do_domctl--> CANCELLED
To solve my issue , I just need to replace in map_domain_pirq in
xen/arch/x86/irq.c
pdev = pci_get_pdev_by_domain(d, msi->seg, msi->bus, msi->devfn);
by
pdev = pci_get_pdev(msi->seg, msi->bus, msi->devfn);
With this "fix" in Xen, dom1 can use the ethernet device and its MSI that I see
in /proc/interrupt
I use a customized cross-compilation of xen and xen-tools for a customized
mininal linux solution thanks to buildroot. This solution has been built from
scratch. So I'm not sure that all the xen components work perfectly. However,
everything seem to work with my modifications (CPU, ethernet, memory, disk,
interrupt ...)
So is it really a bug ? or just a configuration error ? maybe a parameter has
been forgotten in /etc/xen/domU ?
Thank you for your help
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
