Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
Am 28.07.2015 um 03:08 schrieb Andy Lutomirski: > On Mon, Sep 1, 2014 at 10:39 AM, Andy Lutomirski wrote: >> This fixes virtio on Xen guests as well as on any other platform >> that uses virtio_pci on which physical addresses don't match bus >> addresses. >> >> This can be tested with: >> >> virtme-run --xen xen --kimg arch/x86/boot/bzImage --console >> >> using virtme from here: >> >> https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git >> >> Without these patches, the guest hangs forever. With these patches, >> everything works. >> > > Dusting off an ancient thread. > > Now that the dust has accumulated^Wsettled, is it worth pursuing this? > I think the situation is considerably worse than it was when I > originally wrote these patches: I think that QEMU now supports a nasty > mode in which the guest's PCI bus appears to be behind an IOMMU but > the virtio devices on that bus punch straight through that IOMMU. > > I have a half-hearted port to modern kernels here: > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=virtio_ring_xen > > I didn't implement DMA API access for virtio_pci_modern, and I have no > idea what to do about detecting whether a given virtio device honors > its IOMMU or not. I think its really tricky. Looking at where virtio came from, the virtio ring was always native access without IOMMU. This was true for the early lguest things and then the early s390 transport, (which is quite close to the lguest interface). virtio-pci used the same scheme - ignoring all iommu considerations. I understand that for PCI we actually might want to follow iommu restrictions from a correctness and security point of view and from the ccw point of view we do not. No idea about virtio-mmio. I think the proper way of handling this is to take this into the TC for virtio - dont know what would be the right thing to do. A feature bit, always iommu for pci, something else? Michael, Conny, do you agree? Christian ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 28/07/2015 03:08, Andy Lutomirski wrote: > On Mon, Sep 1, 2014 at 10:39 AM, Andy Lutomirski wrote: >> This fixes virtio on Xen guests as well as on any other platform >> that uses virtio_pci on which physical addresses don't match bus >> addresses. >> >> This can be tested with: >> >> virtme-run --xen xen --kimg arch/x86/boot/bzImage --console >> >> using virtme from here: >> >> https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git >> >> Without these patches, the guest hangs forever. With these patches, >> everything works. >> > > Dusting off an ancient thread. > > Now that the dust has accumulated^Wsettled, is it worth pursuing this? > I think the situation is considerably worse than it was when I > originally wrote these patches: I think that QEMU now supports a nasty > mode in which the guest's PCI bus appears to be behind an IOMMU but > the virtio devices on that bus punch straight through that IOMMU. That is an experimental feature (it's x-iommu), so it can change. The plan was: - for PPC, virtio never honors IOMMU - for non-PPC, either have virtio always honor IOMMU, or enforce that virtio is not under IOMMU. Paolo > I have a half-hearted port to modern kernels here: > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=virtio_ring_xen > > I didn't implement DMA API access for virtio_pci_modern, and I have no > idea what to do about detecting whether a given virtio device honors > its IOMMU or not. > > --Andy > ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PULL] vhost: cleanups and fixes
The following changes since commit cbfe8fa6cd672011c755c3cd85c9ffd4e2d10a6f: Linux 4.2-rc4 (2015-07-26 12:26:21 -0700) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git tags/for_linus for you to fetch changes up to 1e0994730f772580ff98754eb5595190cdf371ef: vhost: fix error handling for memory region alloc (2015-07-27 18:05:05 +0300) vhost: fixes for 4.2 Two bugfixes only here. Signed-off-by: Michael S. Tsirkin Igor Mammedov (1): vhost: fix error handling for memory region alloc Marc-André Lureau (1): vhost: actually track log eventfd file drivers/vhost/vhost.c | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] virtio_mmio: add ACPI probing
On Tue, Jul 28, 2015 at 10:44:02AM +0100, Graeme Gregory wrote:
> Added the match table and pointers for ACPI probing to the driver.
>
> This uses the same identifier for virt devices as being used for qemu
> ARM64 ACPI support.
>
> http://git.linaro.org/people/shannon.zhao/qemu.git/commit/d0bf1955a3ecbab4b51d46f8c5dda02b7e14a17e
>
> Signed-off-by: Graeme Gregory
> ---
> drivers/virtio/virtio_mmio.c | 10 ++
> 1 file changed, 10 insertions(+)
>
> diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
> index 10189b5..f499d9d 100644
> --- a/drivers/virtio/virtio_mmio.c
> +++ b/drivers/virtio/virtio_mmio.c
> @@ -58,6 +58,7 @@
>
> #define pr_fmt(fmt) "virtio-mmio: " fmt
>
> +#include
> #include
> #include
> #include
> @@ -732,12 +733,21 @@ static struct of_device_id virtio_mmio_match[] = {
> };
> MODULE_DEVICE_TABLE(of, virtio_mmio_match);
>
> +#ifdef CONFIG_ACPI
> +static const struct acpi_device_id virtio_mmio_acpi_match[] = {
> + { "LNRO0005", },
> + { }
> +};
Hmm - we have reserved QEMU in ASWG explicitly for this purpose.
Pater - do you think it's a good idea to change this before QEMU 2.4
is released?
> +MODULE_DEVICE_TABLE(acpi, virtio_mmio_acpi_match);
> +#endif
> +
> static struct platform_driver virtio_mmio_driver = {
> .probe = virtio_mmio_probe,
> .remove = virtio_mmio_remove,
> .driver = {
> .name = "virtio-mmio",
> .of_match_table = virtio_mmio_match,
> + .acpi_match_table = ACPI_PTR(virtio_mmio_acpi_match),
> },
> };
>
> --
> 2.1.4
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] virtio_mmio: add ACPI probing
On 28 July 2015 at 11:08, Michael S. Tsirkin wrote:
> On Tue, Jul 28, 2015 at 10:44:02AM +0100, Graeme Gregory wrote:
>> Added the match table and pointers for ACPI probing to the driver.
>>
>> This uses the same identifier for virt devices as being used for qemu
>> ARM64 ACPI support.
>>
>> http://git.linaro.org/people/shannon.zhao/qemu.git/commit/d0bf1955a3ecbab4b51d46f8c5dda02b7e14a17e
>>
>> Signed-off-by: Graeme Gregory
>> ---
>> drivers/virtio/virtio_mmio.c | 10 ++
>> 1 file changed, 10 insertions(+)
>>
>> diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
>> index 10189b5..f499d9d 100644
>> --- a/drivers/virtio/virtio_mmio.c
>> +++ b/drivers/virtio/virtio_mmio.c
>> @@ -58,6 +58,7 @@
>>
>> #define pr_fmt(fmt) "virtio-mmio: " fmt
>>
>> +#include
>> #include
>> #include
>> #include
>> @@ -732,12 +733,21 @@ static struct of_device_id virtio_mmio_match[] = {
>> };
>> MODULE_DEVICE_TABLE(of, virtio_mmio_match);
>>
>> +#ifdef CONFIG_ACPI
>> +static const struct acpi_device_id virtio_mmio_acpi_match[] = {
>> + { "LNRO0005", },
>> + { }
>> +};
>
> Hmm - we have reserved QEMU in ASWG explicitly for this purpose.
>
> Pater - do you think it's a good idea to change this before QEMU 2.4
> is released?
Shannon's call, I guess. I don't know enough about ACPI to say.
I thought these ACPI IDs were already fixed because they were
what the kernel was looking for...
-- PMM
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] virtio_mmio: add ACPI probing
On Tue, Jul 28, 2015 at 11:12:33AM +0100, Peter Maydell wrote:
> On 28 July 2015 at 11:08, Michael S. Tsirkin wrote:
> > On Tue, Jul 28, 2015 at 10:44:02AM +0100, Graeme Gregory wrote:
> >> Added the match table and pointers for ACPI probing to the driver.
> >>
> >> This uses the same identifier for virt devices as being used for qemu
> >> ARM64 ACPI support.
> >>
> >> http://git.linaro.org/people/shannon.zhao/qemu.git/commit/d0bf1955a3ecbab4b51d46f8c5dda02b7e14a17e
> >>
> >> Signed-off-by: Graeme Gregory
> >> ---
> >> drivers/virtio/virtio_mmio.c | 10 ++
> >> 1 file changed, 10 insertions(+)
> >>
> >> diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
> >> index 10189b5..f499d9d 100644
> >> --- a/drivers/virtio/virtio_mmio.c
> >> +++ b/drivers/virtio/virtio_mmio.c
> >> @@ -58,6 +58,7 @@
> >>
> >> #define pr_fmt(fmt) "virtio-mmio: " fmt
> >>
> >> +#include
> >> #include
> >> #include
> >> #include
> >> @@ -732,12 +733,21 @@ static struct of_device_id virtio_mmio_match[] = {
> >> };
> >> MODULE_DEVICE_TABLE(of, virtio_mmio_match);
> >>
> >> +#ifdef CONFIG_ACPI
> >> +static const struct acpi_device_id virtio_mmio_acpi_match[] = {
> >> + { "LNRO0005", },
> >> + { }
> >> +};
> >
> > Hmm - we have reserved QEMU in ASWG explicitly for this purpose.
> >
> > Pater - do you think it's a good idea to change this before QEMU 2.4
> > is released?
>
> Shannon's call, I guess. I don't know enough about ACPI to say.
> I thought these ACPI IDs were already fixed because they were
> what the kernel was looking for...
>
> -- PMM
Apparently not :)
--
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, 2015-07-28 at 10:16 +0200, Paolo Bonzini wrote: > > On 28/07/2015 03:08, Andy Lutomirski wrote: > > On Mon, Sep 1, 2014 at 10:39 AM, Andy Lutomirski > > wrote: > >> This fixes virtio on Xen guests as well as on any other platform > >> that uses virtio_pci on which physical addresses don't match bus > >> addresses. > >> > >> This can be tested with: > >> > >> virtme-run --xen xen --kimg arch/x86/boot/bzImage --console > >> > >> using virtme from here: > >> > >> https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git > >> > >> Without these patches, the guest hangs forever. With these patches, > >> everything works. > >> > > > > Dusting off an ancient thread. > > > > Now that the dust has accumulated^Wsettled, is it worth pursuing this? > > I think the situation is considerably worse than it was when I > > originally wrote these patches: I think that QEMU now supports a nasty > > mode in which the guest's PCI bus appears to be behind an IOMMU but > > the virtio devices on that bus punch straight through that IOMMU. > > That is an experimental feature (it's x-iommu), so it can change. > > The plan was: > > - for PPC, virtio never honors IOMMU > > - for non-PPC, either have virtio always honor IOMMU, or enforce that > virtio is not under IOMMU. > I dislike having PPC special cased. In fact, today x86 guests also assume that virtio bypasses IOMMU I believe. In fact *all* guests do. I would much prefer if the information as to whether it honors or not gets passed to the guest somewhat. My preference goes for passing it via the virtio config space but there were objections that it should be a bus property (which is tricky to do with PCI and doesn't properly reflect the fact that in qemu you can mix & match IOMMU-honoring devices and bypassing-virtio on the same bus). Ben. > Paolo > > > I have a half-hearted port to modern kernels here: > > > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=virtio_ring_xen > > > > I didn't implement DMA API access for virtio_pci_modern, and I have no > > idea what to do about detecting whether a given virtio device honors > > its IOMMU or not. > > > > --Andy > > ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 28/07/2015 12:12, Benjamin Herrenschmidt wrote: >> > That is an experimental feature (it's x-iommu), so it can change. >> > >> > The plan was: >> > >> > - for PPC, virtio never honors IOMMU >> > >> > - for non-PPC, either have virtio always honor IOMMU, or enforce that >> > virtio is not under IOMMU. >> > > I dislike having PPC special cased. > > In fact, today x86 guests also assume that virtio bypasses IOMMU I > believe. In fact *all* guests do. This doesn't matter much, since the only guests that implement an IOMMU in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind of stability. > I would much prefer if the information as to whether it honors or not > gets passed to the guest somewhat. My preference goes for passing it via > the virtio config space but there were objections that it should be a > bus property (which is tricky to do with PCI and doesn't properly > reflect the fact that in qemu you can mix & match IOMMU-honoring devices > and bypassing-virtio on the same bus). Yes, for example on x86 it must be passed through the DMAR table. virtio-pci device must have a separate DRHD for them. In QEMU, you could add an "under-iommu" property to PCI bridges, and walk the hierarchy of bridges to build the DRHDs. Paolo ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 02:46:20PM +0200, Paolo Bonzini wrote: > > > On 28/07/2015 12:12, Benjamin Herrenschmidt wrote: > >> > That is an experimental feature (it's x-iommu), so it can change. > >> > > >> > The plan was: > >> > > >> > - for PPC, virtio never honors IOMMU > >> > > >> > - for non-PPC, either have virtio always honor IOMMU, or enforce that > >> > virtio is not under IOMMU. > >> > > > I dislike having PPC special cased. > > > > In fact, today x86 guests also assume that virtio bypasses IOMMU I > > believe. In fact *all* guests do. > > This doesn't matter much, since the only guests that implement an IOMMU > in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind > of stability. Hmm I think Jan (cc) said it was already used out there. > > I would much prefer if the information as to whether it honors or not > > gets passed to the guest somewhat. My preference goes for passing it via > > the virtio config space but there were objections that it should be a > > bus property (which is tricky to do with PCI and doesn't properly > > reflect the fact that in qemu you can mix & match IOMMU-honoring devices > > and bypassing-virtio on the same bus). > > Yes, for example on x86 it must be passed through the DMAR table. > virtio-pci device must have a separate DRHD for them. In QEMU, you > could add an "under-iommu" property to PCI bridges, and walk the > hierarchy of bridges to build the DRHDs. > > Paolo -- MST ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Mon, Jul 27, 2015 at 06:08:59PM -0700, Andy Lutomirski wrote: > On Mon, Sep 1, 2014 at 10:39 AM, Andy Lutomirski wrote: > > This fixes virtio on Xen guests as well as on any other platform > > that uses virtio_pci on which physical addresses don't match bus > > addresses. > > > > This can be tested with: > > > > virtme-run --xen xen --kimg arch/x86/boot/bzImage --console > > > > using virtme from here: > > > > https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git > > > > Without these patches, the guest hangs forever. With these patches, > > everything works. > > > > Dusting off an ancient thread. > > Now that the dust has accumulated^Wsettled, is it worth pursuing this? > I think the situation is considerably worse than it was when I > originally wrote these patches: I think that QEMU now supports a nasty > mode in which the guest's PCI bus appears to be behind an IOMMU but > the virtio devices on that bus punch straight through that IOMMU. > > I have a half-hearted port to modern kernels here: > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=virtio_ring_xen > > I didn't implement DMA API access for virtio_pci_modern, and I have no > idea what to do about detecting whether a given virtio device honors > its IOMMU or not. > > --Andy It's worth thinking about. I'll need to measure what's the overhead of supporting both modes - probably after I'm back from the KVM forum. -- MST ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 15:06, Michael S. Tsirkin wrote: > On Tue, Jul 28, 2015 at 02:46:20PM +0200, Paolo Bonzini wrote: >> >> >> On 28/07/2015 12:12, Benjamin Herrenschmidt wrote: > That is an experimental feature (it's x-iommu), so it can change. > > The plan was: > > - for PPC, virtio never honors IOMMU > > - for non-PPC, either have virtio always honor IOMMU, or enforce that > virtio is not under IOMMU. > >>> I dislike having PPC special cased. >>> >>> In fact, today x86 guests also assume that virtio bypasses IOMMU I >>> believe. In fact *all* guests do. >> >> This doesn't matter much, since the only guests that implement an IOMMU >> in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind >> of stability. > > Hmm I think Jan (cc) said it was already used out there. Yes, no known issues with vt-d emulation for almost a year now. Error reporting could be improved, and interrupt remapping is still missing, but those are minor issues in this context. In my testing setups, I also have virtio devices in use, passed through to an L2 guest, but only in 1:1 mapping so that their broken IOMMU support causes no practical problems. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Jul 28, 2015 6:11 AM, "Jan Kiszka" wrote: > > On 2015-07-28 15:06, Michael S. Tsirkin wrote: > > On Tue, Jul 28, 2015 at 02:46:20PM +0200, Paolo Bonzini wrote: > >> > >> > >> On 28/07/2015 12:12, Benjamin Herrenschmidt wrote: > > That is an experimental feature (it's x-iommu), so it can change. > > > > The plan was: > > > > - for PPC, virtio never honors IOMMU > > > > - for non-PPC, either have virtio always honor IOMMU, or enforce that > > virtio is not under IOMMU. > > > >>> I dislike having PPC special cased. > >>> > >>> In fact, today x86 guests also assume that virtio bypasses IOMMU I > >>> believe. In fact *all* guests do. > >> > >> This doesn't matter much, since the only guests that implement an IOMMU > >> in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind > >> of stability. > > > > Hmm I think Jan (cc) said it was already used out there. > > Yes, no known issues with vt-d emulation for almost a year now. Error > reporting could be improved, and interrupt remapping is still missing, > but those are minor issues in this context. > > In my testing setups, I also have virtio devices in use, passed through > to an L2 guest, but only in 1:1 mapping so that their broken IOMMU > support causes no practical problems. > How are you getting 1:1 to work? Is it something that L0 QEMU can advertise to L1? If so, can we just do that unconditionally, which would make my patch work? I have no objection to 1:1 devices in general. It's only devices that the PCI code on the guest identifies as not 1:1 but that are nonetheless 1:1 that cause problems. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 28/07/2015 15:11, Jan Kiszka wrote:
>>> >>
>>> >> This doesn't matter much, since the only guests that implement an IOMMU
>>> >> in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind
>>> >> of stability.
>> >
>> > Hmm I think Jan (cc) said it was already used out there.
> Yes, no known issues with vt-d emulation for almost a year now. Error
> reporting could be improved, and interrupt remapping is still missing,
> but those are minor issues in this context.
On the other hand interrupt remapping is absolutely necessary for
production use, hence my point that x86 does not promise API stability.
("Any kind of stability" actually didn't include crashes; those are not
expected :))
The Google patches for userspace PIC and IOAPIC are proceeding well, so
hopefully we can have interrupt remapping soon.
Paolo
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 18:36, Paolo Bonzini wrote:
> On 28/07/2015 15:11, Jan Kiszka wrote:
>>
>> This doesn't matter much, since the only guests that implement an IOMMU
>> in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind
>> of stability.
Hmm I think Jan (cc) said it was already used out there.
>> Yes, no known issues with vt-d emulation for almost a year now. Error
>> reporting could be improved, and interrupt remapping is still missing,
>> but those are minor issues in this context.
>
> On the other hand interrupt remapping is absolutely necessary for
> production use, hence my point that x86 does not promise API stability.
Well, we currently implement the features that the Q35 used to expose.
Adding interrupt remapping will require a new chipset and/or a hack
switch to ignore compatibility.
>
> ("Any kind of stability" actually didn't include crashes; those are not
> expected :))
>
> The Google patches for userspace PIC and IOAPIC are proceeding well, so
> hopefully we can have interrupt remapping soon.
If the day had 48 hours... I'd love to look into this, first adding QEMU
support for the new irqchip architecture.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 18:11, Andy Lutomirski wrote: > On Jul 28, 2015 6:11 AM, "Jan Kiszka" wrote: >> >> On 2015-07-28 15:06, Michael S. Tsirkin wrote: >>> On Tue, Jul 28, 2015 at 02:46:20PM +0200, Paolo Bonzini wrote: On 28/07/2015 12:12, Benjamin Herrenschmidt wrote: >>> That is an experimental feature (it's x-iommu), so it can change. >>> >>> The plan was: >>> >>> - for PPC, virtio never honors IOMMU >>> >>> - for non-PPC, either have virtio always honor IOMMU, or enforce that >>> virtio is not under IOMMU. >>> > I dislike having PPC special cased. > > In fact, today x86 guests also assume that virtio bypasses IOMMU I > believe. In fact *all* guests do. This doesn't matter much, since the only guests that implement an IOMMU in QEMU are (afaik) PPC and x86, and x86 does not yet promise any kind of stability. >>> >>> Hmm I think Jan (cc) said it was already used out there. >> >> Yes, no known issues with vt-d emulation for almost a year now. Error >> reporting could be improved, and interrupt remapping is still missing, >> but those are minor issues in this context. >> >> In my testing setups, I also have virtio devices in use, passed through >> to an L2 guest, but only in 1:1 mapping so that their broken IOMMU >> support causes no practical problems. >> > > How are you getting 1:1 to work? Is it something that L0 QEMU can > advertise to L1? If so, can we just do that unconditionally, which > would make my patch work? The guest hypervisor is Jailhouse and the guest is the root cell that loaded the hypervisor, thus continues with identity mappings. You usually don't have 1:1 mapping with other setups - maybe with some Xen configuration? Dunno. > > I have no objection to 1:1 devices in general. It's only devices that > the PCI code on the guest identifies as not 1:1 but that are > nonetheless 1:1 that cause problems. The ability to have virtio on systems with IOMMU in place makes testing much more efficient for us. Ideally, we would have it in non-identity mapping scenarios as well, e.g. to start secondary Linux instances in the test VMs, giving them their own virtio devices. And we will eventually have this need on ARM as well. Virtio needs to be backward compatible, so the change to put these devices under IOMMU control could be advertised during feature negotiations and controlled on QEMU side via a device property. Newer guest drivers would have to acknowledge that they support virtio via IOMMUs. Older ones would refuse to work, and the admin could instead spawn VMs with this feature disabled. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 9:44 AM, Jan Kiszka wrote: > The ability to have virtio on systems with IOMMU in place makes testing > much more efficient for us. Ideally, we would have it in non-identity > mapping scenarios as well, e.g. to start secondary Linux instances in > the test VMs, giving them their own virtio devices. And we will > eventually have this need on ARM as well. > > Virtio needs to be backward compatible, so the change to put these > devices under IOMMU control could be advertised during feature > negotiations and controlled on QEMU side via a device property. Newer > guest drivers would have to acknowledge that they support virtio via > IOMMUs. Older ones would refuse to work, and the admin could instead > spawn VMs with this feature disabled. > The trouble is that this is really a property of the bus and not of the device. If you build a virtio device that physically plugs into a PCIe slot, the device has no concept of an IOMMU in the first place. Similarly, if you take an L0-provided IOMMU-supporting device and pass it through to L2 using current QEMU on L1 (with Q35 emulation and iommu enabled), then, from L2's perspective, the device is 1:1 no matter what the device thinks. IOW, I think the original design was wrong and now we have to deal with it. I think the best solution would be to teach QEMU to fix its ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 28/07/2015 18:42, Jan Kiszka wrote:
> > On the other hand interrupt remapping is absolutely necessary for
> > production use, hence my point that x86 does not promise API stability.
>
> Well, we currently implement the features that the Q35 used to expose.
> Adding interrupt remapping will require a new chipset and/or a hack
> switch to ignore compatibility.
Isn't the VT-d register space separate from other Q35 features and
backwards-compatible? You could even add it to PIIX in theory just by
adding a DMAR.
It's not like for example SMRAM, where the registers are in the
northbridge configuration space and move around in every chipset generation.
> > ("Any kind of stability" actually didn't include crashes; those are not
> > expected :))
> >
> > The Google patches for userspace PIC and IOAPIC are proceeding well, so
> > hopefully we can have interrupt remapping soon.
>
> If the day had 48 hours... I'd love to look into this, first adding QEMU
> support for the new irqchip architecture.
I hope I can squeeze in some time for that... Google also had an intern
that was looking at it.
Paolo
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 19:10, Andy Lutomirski wrote: > On Tue, Jul 28, 2015 at 9:44 AM, Jan Kiszka wrote: >> The ability to have virtio on systems with IOMMU in place makes testing >> much more efficient for us. Ideally, we would have it in non-identity >> mapping scenarios as well, e.g. to start secondary Linux instances in >> the test VMs, giving them their own virtio devices. And we will >> eventually have this need on ARM as well. >> >> Virtio needs to be backward compatible, so the change to put these >> devices under IOMMU control could be advertised during feature >> negotiations and controlled on QEMU side via a device property. Newer >> guest drivers would have to acknowledge that they support virtio via >> IOMMUs. Older ones would refuse to work, and the admin could instead >> spawn VMs with this feature disabled. >> > > The trouble is that this is really a property of the bus and not of > the device. If you build a virtio device that physically plugs into a > PCIe slot, the device has no concept of an IOMMU in the first place. If one would build a real virtio device today, it would be broken because every IOMMU would start to translate its requests. Already from that POV, we really need to introduce a feature flag "I will be IOMMU-translated" so that a potential physical implementation can carry it unconditionally. > Similarly, if you take an L0-provided IOMMU-supporting device and pass > it through to L2 using current QEMU on L1 (with Q35 emulation and > iommu enabled), then, from L2's perspective, the device is 1:1 no > matter what the device thinks. > > IOW, I think the original design was wrong and now we have to deal > with it. I think the best solution would be to teach QEMU to fix its > ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. Only the current drivers are broken. And we can easily tell them apart from newer ones via feature flags. Sorry, don't get the problem. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 19:15, Paolo Bonzini wrote:
>
>
> On 28/07/2015 18:42, Jan Kiszka wrote:
>>> On the other hand interrupt remapping is absolutely necessary for
>>> production use, hence my point that x86 does not promise API stability.
>>
>> Well, we currently implement the features that the Q35 used to expose.
>> Adding interrupt remapping will require a new chipset and/or a hack
>> switch to ignore compatibility.
>
> Isn't the VT-d register space separate from other Q35 features and
> backwards-compatible? You could even add it to PIIX in theory just by
> adding a DMAR.
Yes, it's practically working, but it's not accurate /wrt how that
hardware looked like in reality.
>
> It's not like for example SMRAM, where the registers are in the
> northbridge configuration space and move around in every chipset generation.
>
>>> ("Any kind of stability" actually didn't include crashes; those are not
>>> expected :))
>>>
>>> The Google patches for userspace PIC and IOAPIC are proceeding well, so
>>> hopefully we can have interrupt remapping soon.
>>
>> If the day had 48 hours... I'd love to look into this, first adding QEMU
>> support for the new irqchip architecture.
>
> I hope I can squeeze in some time for that... Google also had an intern
> that was looking at it.
Great!
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 28/07/2015 19:19, Jan Kiszka wrote: > On 2015-07-28 19:15, Paolo Bonzini wrote: >> >> >> On 28/07/2015 18:42, Jan Kiszka wrote: On the other hand interrupt remapping is absolutely necessary for production use, hence my point that x86 does not promise API stability. >>> >>> Well, we currently implement the features that the Q35 used to expose. >>> Adding interrupt remapping will require a new chipset and/or a hack >>> switch to ignore compatibility. >> >> Isn't the VT-d register space separate from other Q35 features and >> backwards-compatible? You could even add it to PIIX in theory just by >> adding a DMAR. > > Yes, it's practically working, but it's not accurate /wrt how that > hardware looked like in reality. We've done that for a long time. Real PIIX3 didn't have ACPI too, for example (and it had a USB UHCI that is optional in QEMU). Of course I'm not advocating adding the IOMMU to PIIX (assuming that would work even just practically)... but I don't think adding interrupt remapping to Q35 is a big deal. It would be optional, just in case you want to debug something without interrupt remapping, but it can be added. The Google patches for userspace PIC and IOAPIC are proceeding well, so hopefully we can have interrupt remapping soon. >>> >>> If the day had 48 hours... I'd love to look into this, first adding QEMU >>> support for the new irqchip architecture. >> >> I hope I can squeeze in some time for that... Google also had an intern >> that was looking at it. > > Great! In theory it's easy with the latest series. All you need is support for converting IOAPIC routes to KVM routes (and of course the glue code to enable the capability and create the userspace devices); everything else should work just by reusing the -machine kernel_irqchip=on code. In theory... Paolo ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 10:17 AM, Jan Kiszka wrote: > On 2015-07-28 19:10, Andy Lutomirski wrote: >> The trouble is that this is really a property of the bus and not of >> the device. If you build a virtio device that physically plugs into a >> PCIe slot, the device has no concept of an IOMMU in the first place. > > If one would build a real virtio device today, it would be broken > because every IOMMU would start to translate its requests. Already from > that POV, we really need to introduce a feature flag "I will be > IOMMU-translated" so that a potential physical implementation can carry > it unconditionally. > Except that, with my patches, it would work correctly. ISTM the thing that's broken right now is QEMU and the virtio_pci driver. My patches fix the driver. Last year that would have been the end of the story except for PPC. Now we have to deal with QEMU. >> Similarly, if you take an L0-provided IOMMU-supporting device and pass >> it through to L2 using current QEMU on L1 (with Q35 emulation and >> iommu enabled), then, from L2's perspective, the device is 1:1 no >> matter what the device thinks. >> >> IOW, I think the original design was wrong and now we have to deal >> with it. I think the best solution would be to teach QEMU to fix its >> ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. > > Only the current drivers are broken. And we can easily tell them apart > from newer ones via feature flags. Sorry, don't get the problem. I still don't see how feature flags solve the problem. Suppose we added a feature flag meaning "respects IOMMU". Bad case 1: Build a malicious device that advertises non-IOMMU-respecting virtio. Plug it in behind an IOMMU. Host starts leaking physical addresses to the device (and the device doesn't work, of course). Maybe that's only barely a security problem, but still... Bad case 2: Use current QEMU w/ IOMMU enabled. Assign a virtio device provided by L0 QEMU to L2. L1 crashes. I consider *that* to be a security problem, although in practice no one will configure their system that way because it has zero chance of actually working. Nonetheless, the device does work if L1 accesses it directly? The issue is vfio doesn't notice that the device doesn't respect the IOMMU because "respects-IOMMU" is a property of the PCI bus and the platform IOMMU, and vfio assumes it works correctly. Bad case 2: Some hypothetical well-behaved new QEMU provides a virtio device that *does* respect the IOMMU and sets the feature flag. They emulate Q35 with an IOMMU. They boot Linux 4.1. Data corruption in the guest. We could make the rule that *all* virtio-pci devices (except on PPC) respect the bus rules. We'd have to fix QEMU so that virtio devices on Q35 iommu=on systems set up a PCI topology where the devices *aren't* behind the IOMMU or are protected by RMRRs or whatever. Then old kernels would work correctly on new hosts, new kernels would work correctly except on old iommu-providing hosts, and Xen would work. In fact, on Xen, it's impossible without colossal hacks to support non-IOMMU-respecting virtio devices because Xen acts as an intermediate IOMMU between the Linux dom0 guest and the actual host. The QEMU host doesn't even know that Xen is involved. This is why Xen and virtio don't currently work together (without my patches): the device thinks it doesn't respect the IOMMU, the driver thinks the device doesn't respect the IOMMU, and they're both wrong. TL;DR: I think there are only two cases. Either a device respects the IOMMU or a device doesn't know whether it respects the IOMMU. The latter case is problematic. --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 20:22, Andy Lutomirski wrote: > On Tue, Jul 28, 2015 at 10:17 AM, Jan Kiszka wrote: >> On 2015-07-28 19:10, Andy Lutomirski wrote: >>> The trouble is that this is really a property of the bus and not of >>> the device. If you build a virtio device that physically plugs into a >>> PCIe slot, the device has no concept of an IOMMU in the first place. >> >> If one would build a real virtio device today, it would be broken >> because every IOMMU would start to translate its requests. Already from >> that POV, we really need to introduce a feature flag "I will be >> IOMMU-translated" so that a potential physical implementation can carry >> it unconditionally. >> > > Except that, with my patches, it would work correctly. ISTM the thing I haven't looked at your patches yet - they make the virtio PCI driver in Linux IOMMU-compatible? Perfect - except for a compatibility check, right? > that's broken right now is QEMU and the virtio_pci driver. My patches > fix the driver. Last year that would have been the end of the story > except for PPC. Now we have to deal with QEMU. > >>> Similarly, if you take an L0-provided IOMMU-supporting device and pass >>> it through to L2 using current QEMU on L1 (with Q35 emulation and >>> iommu enabled), then, from L2's perspective, the device is 1:1 no >>> matter what the device thinks. >>> >>> IOW, I think the original design was wrong and now we have to deal >>> with it. I think the best solution would be to teach QEMU to fix its >>> ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. >> >> Only the current drivers are broken. And we can easily tell them apart >> from newer ones via feature flags. Sorry, don't get the problem. > > I still don't see how feature flags solve the problem. Suppose we > added a feature flag meaning "respects IOMMU". > > Bad case 1: Build a malicious device that advertises > non-IOMMU-respecting virtio. Plug it in behind an IOMMU. Host starts > leaking physical addresses to the device (and the device doesn't work, > of course). Maybe that's only barely a security problem, but still... I don't see right now how critical such a hypothetical case could be. But the OS / its drivers could still decide to refuse talking to such a device. > > Bad case 2: Use current QEMU w/ IOMMU enabled. Assign a virtio > device provided by L0 QEMU to L2. L1 crashes. I consider *that* to > be a security problem, although in practice no one will configure > their system that way because it has zero chance of actually working. > Nonetheless, the device does work if L1 accesses it directly? The > issue is vfio doesn't notice that the device doesn't respect the IOMMU > because "respects-IOMMU" is a property of the PCI bus and the platform > IOMMU, and vfio assumes it works correctly. I would have no problem with rejecting configurations in future QEMU that try to expose unconfined virtio devices in the presence of IOMMU emulation. Once we can do better, it's just about letting the guest know about the difference. The current situation is indeed just broken, we don't need to discuss this as we can't change history to prevent this. > > Bad case 2: Some hypothetical well-behaved new QEMU provides a virtio > device that *does* respect the IOMMU and sets the feature flag. They > emulate Q35 with an IOMMU. They boot Linux 4.1. Data corruption in > the guest. No. In that case, the feature negotiation of "virtio-with-iommu-support" would have failed for older drivers, and the device would have never been used by the guest. > > We could make the rule that *all* virtio-pci devices (except on PPC) > respect the bus rules. We'd have to fix QEMU so that virtio devices > on Q35 iommu=on systems set up a PCI topology where the devices > *aren't* behind the IOMMU or are protected by RMRRs or whatever. Then > old kernels would work correctly on new hosts, new kernels would work > correctly except on old iommu-providing hosts, and Xen would work. I don't see a point in doing anything about old QEMU with IOMMU enabled and virtio devices plugged except declaring such setups broken. No one should have configured this for production purposes, only for test setups (like we, with the knowledge about the limitations). > > In fact, on Xen, it's impossible without colossal hacks to support > non-IOMMU-respecting virtio devices because Xen acts as an > intermediate IOMMU between the Linux dom0 guest and the actual host. > The QEMU host doesn't even know that Xen is involved. This is why Xen > and virtio don't currently work together (without my patches): the > device thinks it doesn't respect the IOMMU, the driver thinks the > device doesn't respect the IOMMU, and they're both wrong. > > TL;DR: I think there are only two cases. Either a device respects the > IOMMU or a device doesn't know whether it respects the IOMMU. The > latter case is problematic. See above, the latter is only problematic on setups that actually use an IOMMU. If
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 12:06 PM, Jan Kiszka wrote: > On 2015-07-28 20:22, Andy Lutomirski wrote: >> On Tue, Jul 28, 2015 at 10:17 AM, Jan Kiszka wrote: >>> On 2015-07-28 19:10, Andy Lutomirski wrote: The trouble is that this is really a property of the bus and not of the device. If you build a virtio device that physically plugs into a PCIe slot, the device has no concept of an IOMMU in the first place. >>> >>> If one would build a real virtio device today, it would be broken >>> because every IOMMU would start to translate its requests. Already from >>> that POV, we really need to introduce a feature flag "I will be >>> IOMMU-translated" so that a potential physical implementation can carry >>> it unconditionally. >>> >> >> Except that, with my patches, it would work correctly. ISTM the thing > > I haven't looked at your patches yet - they make the virtio PCI driver > in Linux IOMMU-compatible? Perfect - except for a compatibility check, > right? Yes. (virtio_pci_legacy, anyway. Presumably virtio_pci_modern is easy to adapt, too.) > >> that's broken right now is QEMU and the virtio_pci driver. My patches >> fix the driver. Last year that would have been the end of the story >> except for PPC. Now we have to deal with QEMU. >> Similarly, if you take an L0-provided IOMMU-supporting device and pass it through to L2 using current QEMU on L1 (with Q35 emulation and iommu enabled), then, from L2's perspective, the device is 1:1 no matter what the device thinks. IOW, I think the original design was wrong and now we have to deal with it. I think the best solution would be to teach QEMU to fix its ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. >>> >>> Only the current drivers are broken. And we can easily tell them apart >>> from newer ones via feature flags. Sorry, don't get the problem. >> >> I still don't see how feature flags solve the problem. Suppose we >> added a feature flag meaning "respects IOMMU". >> >> Bad case 1: Build a malicious device that advertises >> non-IOMMU-respecting virtio. Plug it in behind an IOMMU. Host starts >> leaking physical addresses to the device (and the device doesn't work, >> of course). Maybe that's only barely a security problem, but still... > > I don't see right now how critical such a hypothetical case could be. > But the OS / its drivers could still decide to refuse talking to such a > device. > How does OS know it's such a device as opposed to a QEMU-supplied thing? >> >> Bad case 2: Some hypothetical well-behaved new QEMU provides a virtio >> device that *does* respect the IOMMU and sets the feature flag. They >> emulate Q35 with an IOMMU. They boot Linux 4.1. Data corruption in >> the guest. > > No. In that case, the feature negotiation of "virtio-with-iommu-support" > would have failed for older drivers, and the device would have never > been used by the guest. So are you suggesting that newer virtio devices always provide this feature flag and, if supplied by QEMU with iommu=on, simply refuse to operate of the driver doesn't support that flag? That could work as long as QEMU with the current (broken?) iommu=on never exposes such a device. > >> >> We could make the rule that *all* virtio-pci devices (except on PPC) >> respect the bus rules. We'd have to fix QEMU so that virtio devices >> on Q35 iommu=on systems set up a PCI topology where the devices >> *aren't* behind the IOMMU or are protected by RMRRs or whatever. Then >> old kernels would work correctly on new hosts, new kernels would work >> correctly except on old iommu-providing hosts, and Xen would work. > > I don't see a point in doing anything about old QEMU with IOMMU enabled > and virtio devices plugged except declaring such setups broken. No one > should have configured this for production purposes, only for test > setups (like we, with the knowledge about the limitations). > I'm fine with that. In fact, I proposed these patches before QEMU had this feature in the first place. >> >> In fact, on Xen, it's impossible without colossal hacks to support >> non-IOMMU-respecting virtio devices because Xen acts as an >> intermediate IOMMU between the Linux dom0 guest and the actual host. >> The QEMU host doesn't even know that Xen is involved. This is why Xen >> and virtio don't currently work together (without my patches): the >> device thinks it doesn't respect the IOMMU, the driver thinks the >> device doesn't respect the IOMMU, and they're both wrong. >> >> TL;DR: I think there are only two cases. Either a device respects the >> IOMMU or a device doesn't know whether it respects the IOMMU. The >> latter case is problematic. > > See above, the latter is only problematic on setups that actually use an > IOMMU. If that includes Xen, then no one should use it until virtio can > declare itself IOMMU compatible, and drivers exist that process this. Xen works right now with my patches on standard QEMU (as l
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On 2015-07-28 21:24, Andy Lutomirski wrote: > On Tue, Jul 28, 2015 at 12:06 PM, Jan Kiszka wrote: >> On 2015-07-28 20:22, Andy Lutomirski wrote: >>> On Tue, Jul 28, 2015 at 10:17 AM, Jan Kiszka wrote: On 2015-07-28 19:10, Andy Lutomirski wrote: > The trouble is that this is really a property of the bus and not of > the device. If you build a virtio device that physically plugs into a > PCIe slot, the device has no concept of an IOMMU in the first place. If one would build a real virtio device today, it would be broken because every IOMMU would start to translate its requests. Already from that POV, we really need to introduce a feature flag "I will be IOMMU-translated" so that a potential physical implementation can carry it unconditionally. >>> >>> Except that, with my patches, it would work correctly. ISTM the thing >> >> I haven't looked at your patches yet - they make the virtio PCI driver >> in Linux IOMMU-compatible? Perfect - except for a compatibility check, >> right? > > Yes. (virtio_pci_legacy, anyway. Presumably virtio_pci_modern is > easy to adapt, too.) > >> >>> that's broken right now is QEMU and the virtio_pci driver. My patches >>> fix the driver. Last year that would have been the end of the story >>> except for PPC. Now we have to deal with QEMU. >>> > Similarly, if you take an L0-provided IOMMU-supporting device and pass > it through to L2 using current QEMU on L1 (with Q35 emulation and > iommu enabled), then, from L2's perspective, the device is 1:1 no > matter what the device thinks. > > IOW, I think the original design was wrong and now we have to deal > with it. I think the best solution would be to teach QEMU to fix its > ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. Only the current drivers are broken. And we can easily tell them apart from newer ones via feature flags. Sorry, don't get the problem. >>> >>> I still don't see how feature flags solve the problem. Suppose we >>> added a feature flag meaning "respects IOMMU". >>> >>> Bad case 1: Build a malicious device that advertises >>> non-IOMMU-respecting virtio. Plug it in behind an IOMMU. Host starts >>> leaking physical addresses to the device (and the device doesn't work, >>> of course). Maybe that's only barely a security problem, but still... >> >> I don't see right now how critical such a hypothetical case could be. >> But the OS / its drivers could still decide to refuse talking to such a >> device. >> > > How does OS know it's such a device as opposed to a QEMU-supplied thing? It can restrict itself to virtio devices exposing the feature if it feels uncomfortable that it might be talking to some evil piece of silicon (instead of the hypervisor, which has to be trusted anyway). > >>> >>> Bad case 2: Some hypothetical well-behaved new QEMU provides a virtio >>> device that *does* respect the IOMMU and sets the feature flag. They >>> emulate Q35 with an IOMMU. They boot Linux 4.1. Data corruption in >>> the guest. >> >> No. In that case, the feature negotiation of "virtio-with-iommu-support" >> would have failed for older drivers, and the device would have never >> been used by the guest. > > So are you suggesting that newer virtio devices always provide this > feature flag and, if supplied by QEMU with iommu=on, simply refuse to > operate of the driver doesn't support that flag? Exactly. > > That could work as long as QEMU with the current (broken?) iommu=on > never exposes such a device. QEMU would have to be adjusted first so that all its virtio-pci device models take IOMMUs into account - if they exist or not. Only then it could expose the feature and expect the guest to acknowledge it. For compat reasons, QEMU should still be able to expose virtio devices without the flag set - but then without any IOMMU emulation enabled as well. That would prevent the current setup we are using today, but it's trivial to update the guest kernel to a newer virtio driver which would restore our scenario again. > >> >>> >>> We could make the rule that *all* virtio-pci devices (except on PPC) >>> respect the bus rules. We'd have to fix QEMU so that virtio devices >>> on Q35 iommu=on systems set up a PCI topology where the devices >>> *aren't* behind the IOMMU or are protected by RMRRs or whatever. Then >>> old kernels would work correctly on new hosts, new kernels would work >>> correctly except on old iommu-providing hosts, and Xen would work. >> >> I don't see a point in doing anything about old QEMU with IOMMU enabled >> and virtio devices plugged except declaring such setups broken. No one >> should have configured this for production purposes, only for test >> setups (like we, with the knowledge about the limitations). >> > > I'm fine with that. In fact, I proposed these patches before QEMU had > this feature in the first place. > >>> >>> In fact, on Xen, it's impossible withou
Re: [PATCH] virtio_mmio: add ACPI probing
On 28 July 2015 at 11:27, Michael S. Tsirkin wrote:
> On Tue, Jul 28, 2015 at 11:12:33AM +0100, Peter Maydell wrote:
>> On 28 July 2015 at 11:08, Michael S. Tsirkin wrote:
>> > On Tue, Jul 28, 2015 at 10:44:02AM +0100, Graeme Gregory wrote:
>> >> Added the match table and pointers for ACPI probing to the driver.
>> >>
>> >> This uses the same identifier for virt devices as being used for qemu
>> >> ARM64 ACPI support.
>> >>
>> >> http://git.linaro.org/people/shannon.zhao/qemu.git/commit/d0bf1955a3ecbab4b51d46f8c5dda02b7e14a17e
>> >>
>> >> Signed-off-by: Graeme Gregory
>> >> +#ifdef CONFIG_ACPI
>> >> +static const struct acpi_device_id virtio_mmio_acpi_match[] = {
>> >> + { "LNRO0005", },
>> >> + { }
>> >> +};
>> >
>> > Hmm - we have reserved QEMU in ASWG explicitly for this purpose.
>> >
>> > Pater - do you think it's a good idea to change this before QEMU 2.4
>> > is released?
>>
>> Shannon's call, I guess. I don't know enough about ACPI to say.
>> I thought these ACPI IDs were already fixed because they were
>> what the kernel was looking for...
> Apparently not :)
Mmm. I'm not terribly happy about stuff being in QEMU before the
ACPI spec for it has been finalised. We should not be picking
stuff randomly on the fly...
If we want to fix the ACPI IDs QEMU is using for 2.4 then we
really need to do that now (ie within the next day or two).
-- PMM
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] virtio_mmio: add ACPI probing
On 28 July 2015 at 21:28, G Gregory wrote: > On 28 July 2015 at 21:12, Peter Maydell wrote: >> Mmm. I'm not terribly happy about stuff being in QEMU before the >> ACPI spec for it has been finalised. We should not be picking >> stuff randomly on the fly... >> >> If we want to fix the ACPI IDs QEMU is using for 2.4 then we >> really need to do that now (ie within the next day or two). >> > It is upto the owner of the QEMU prefix to allocate numbers. This is > not an issue for ACPI spec at all. I mean "the specification for how this device should be advertised in an ACPI table". I don't care whether that's an official ACPI consortium thing or something less official. The table is constructed by QEMU and read by the kernel (and possibly also by UEFI?), so everybody needs to agree on what the string is... thanks -- PMM ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 12:33 PM, Jan Kiszka wrote: > On 2015-07-28 21:24, Andy Lutomirski wrote: >> On Tue, Jul 28, 2015 at 12:06 PM, Jan Kiszka wrote: >>> On 2015-07-28 20:22, Andy Lutomirski wrote: On Tue, Jul 28, 2015 at 10:17 AM, Jan Kiszka wrote: > On 2015-07-28 19:10, Andy Lutomirski wrote: >> The trouble is that this is really a property of the bus and not of >> the device. If you build a virtio device that physically plugs into a >> PCIe slot, the device has no concept of an IOMMU in the first place. > > If one would build a real virtio device today, it would be broken > because every IOMMU would start to translate its requests. Already from > that POV, we really need to introduce a feature flag "I will be > IOMMU-translated" so that a potential physical implementation can carry > it unconditionally. > Except that, with my patches, it would work correctly. ISTM the thing >>> >>> I haven't looked at your patches yet - they make the virtio PCI driver >>> in Linux IOMMU-compatible? Perfect - except for a compatibility check, >>> right? >> >> Yes. (virtio_pci_legacy, anyway. Presumably virtio_pci_modern is >> easy to adapt, too.) >> >>> that's broken right now is QEMU and the virtio_pci driver. My patches fix the driver. Last year that would have been the end of the story except for PPC. Now we have to deal with QEMU. >> Similarly, if you take an L0-provided IOMMU-supporting device and pass >> it through to L2 using current QEMU on L1 (with Q35 emulation and >> iommu enabled), then, from L2's perspective, the device is 1:1 no >> matter what the device thinks. >> >> IOW, I think the original design was wrong and now we have to deal >> with it. I think the best solution would be to teach QEMU to fix its >> ACPI tables so that 1:1 virtio devices are actually exposed as 1:1. > > Only the current drivers are broken. And we can easily tell them apart > from newer ones via feature flags. Sorry, don't get the problem. I still don't see how feature flags solve the problem. Suppose we added a feature flag meaning "respects IOMMU". Bad case 1: Build a malicious device that advertises non-IOMMU-respecting virtio. Plug it in behind an IOMMU. Host starts leaking physical addresses to the device (and the device doesn't work, of course). Maybe that's only barely a security problem, but still... >>> >>> I don't see right now how critical such a hypothetical case could be. >>> But the OS / its drivers could still decide to refuse talking to such a >>> device. >>> >> >> How does OS know it's such a device as opposed to a QEMU-supplied thing? > > It can restrict itself to virtio devices exposing the feature if it > feels uncomfortable that it might be talking to some evil piece of > silicon (instead of the hypervisor, which has to be trusted anyway). > >> Bad case 2: Some hypothetical well-behaved new QEMU provides a virtio device that *does* respect the IOMMU and sets the feature flag. They emulate Q35 with an IOMMU. They boot Linux 4.1. Data corruption in the guest. >>> >>> No. In that case, the feature negotiation of "virtio-with-iommu-support" >>> would have failed for older drivers, and the device would have never >>> been used by the guest. >> >> So are you suggesting that newer virtio devices always provide this >> feature flag and, if supplied by QEMU with iommu=on, simply refuse to >> operate of the driver doesn't support that flag? > > Exactly. > >> >> That could work as long as QEMU with the current (broken?) iommu=on >> never exposes such a device. > > QEMU would have to be adjusted first so that all its virtio-pci device > models take IOMMUs into account - if they exist or not. Only then it > could expose the feature and expect the guest to acknowledge it. > > For compat reasons, QEMU should still be able to expose virtio devices > without the flag set - but then without any IOMMU emulation enabled as > well. That would prevent the current setup we are using today, but it's > trivial to update the guest kernel to a newer virtio driver which would > restore our scenario again. Seems reasonable. >> >> If we apply something similar enough to my patches, then even old >> hypervisors (e.g. Amazon's hardware virt systems) will support Xen >> with virtio devices passed in just fine. > > Then it seems we can make everyone happy - perfect. :) Yay. FWIW, I have no intention to touch the QEMU code for this. I'm willing to do the vring bit and the virtio-pci bit as long as it's well specified. --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
Let me try to summarize a proposal: Add a feature flag that indicates IOMMU support. New kernels acknowledge that flag on any device that advertises it. New kernels always respect the IOMMU (except on PowerPC). New kernels optionally refuse to talk to devices that don't have that feature flag if the device appears to be behind an IOMMU. (This presumably includes any device whatsoever on an x86 platform with an IOMMU, including Xen's fake IOMMU.) New QEMU always respects the IOMMU, if any, except on PPC. New QEMU always advertises this feature flag. If iommu=on, QEMU's virtio devices refuse to work unless the driver acknowledges the flag. On PPC, new QEMU will not respect the IOMMU and will not set the flag. New kernels will not talk to devices that set the flag. If someone wants to fix that, then they get to figure out how. This results in: New kernels work fine with old QEMU unless iommu=on. New kernels work with new devices (QEMU and physical devices that set the flag) under all circumstances, except on PPC where physical devices are and remain broken. Xen works work new QEMU and cleanly refuses to interoperate with old QEMU. (This is worse than with just my patches, but it's better than the status quo in which the Xen guest corrupts itself and possibly corrupts the Xen hypervisor.) New kernels with old QEMU with iommu=on optionally refuses to interoperate. Old kernels are oblivious. They work exactly the same as they do today except that they fail cleanly with new QEMU with iommu=on. Old kernels continue to fail with physical virtio devices if they're behind an iommu. Old physical virtio devices that don't advertise the flag fail cleanly if the host uses an iommu. The driver could optionally whitelist such devices. PPC works as well as it currently does. I'm unsure about the arm64 situation. Did I get this right? --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, 2015-07-28 at 15:43 -0700, Andy Lutomirski wrote: > Let me try to summarize a proposal: > > Add a feature flag that indicates IOMMU support. > > New kernels acknowledge that flag on any device that advertises it. > > New kernels always respect the IOMMU (except on PowerPC). Why ? I disagree, the flag should be honored when set in any architecture. PowerPC is no different than any other platform in that regard. > New kernels > optionally refuse to talk to devices that don't have that feature flag > if the device appears to be behind an IOMMU. (This presumably > includes any device whatsoever on an x86 platform with an IOMMU, > including Xen's fake IOMMU.) > > New QEMU always respects the IOMMU, if any, except on PPC. This is just a matter of what is the default of the flag, ie we should have a machine flag that indicates what the default is for new virtio devices, otherwise, it should be specified per device as an attribute of the device instance. I would argue that we should default to "bypass IOMMU" on *all* architecture due to the performance impact, and to essentially default to the same behaviour as today. With things like DDW even powerpc might be able to mostly alleviate the performance impact so we might to change in the long term, but I tend to prefer more incremental approaches. > New QEMU > always advertises this feature flag. If iommu=on, QEMU's virtio > devices refuse to work unless the driver acknowledges the flag. This should be configurable. > On PPC, new QEMU will not respect the IOMMU and will not set the flag. > New kernels will not talk to devices that set the flag. If someone > wants to fix that, then they get to figure out how. I disagree with the kernel bit and I disagree with special casing PPC in any shape or form in the code. The only difference should be a default value for the iommu mode of virtio in qemu set per machine. You can then feel free to change that default (in a separate patch for bisectability) on x86 for the sake of Xen. Ben. > This results in: > > New kernels work fine with old QEMU unless iommu=on. > > New kernels work with new devices (QEMU and physical devices that set > the flag) under all circumstances, except on PPC where physical > devices are and remain broken. > > Xen works work new QEMU and cleanly refuses to interoperate with old > QEMU. (This is worse than with just my patches, but it's better than > the status quo in which the Xen guest corrupts itself and possibly > corrupts the Xen hypervisor.) > > New kernels with old QEMU with iommu=on optionally refuses to interoperate. > > Old kernels are oblivious. They work exactly the same as they do > today except that they fail cleanly with new QEMU with iommu=on. Old > kernels continue to fail with physical virtio devices if they're > behind an iommu. > > Old physical virtio devices that don't advertise the flag fail cleanly > if the host uses an iommu. The driver could optionally whitelist such > devices. > > PPC works as well as it currently does. > > I'm unsure about the arm64 situation. > > > Did I get this right? > > --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 4:21 PM, Benjamin Herrenschmidt wrote: > On Tue, 2015-07-28 at 15:43 -0700, Andy Lutomirski wrote: >> Let me try to summarize a proposal: >> >> Add a feature flag that indicates IOMMU support. >> >> New kernels acknowledge that flag on any device that advertises it. >> >> New kernels always respect the IOMMU (except on PowerPC). > > Why ? I disagree, the flag should be honored when set in any > architecture. PowerPC is no different than any other platform in that > regard. Perhaps I should have said instead "someone more familiar with PPC than I am should figure out what PPC should do". For the non-PPC case, there is only one instance that I know of in which ignoring the IOMMU is beneficial, and that case is the experimental Q35 thing. If new kernels ignore the IOMMU for devices that don't set the flag and there are physical devices that already exist and don't set the flag, then those devices won't work reliably on most modern non-virtual platforms, PPC included. > >> New kernels >> optionally refuse to talk to devices that don't have that feature flag >> if the device appears to be behind an IOMMU. (This presumably >> includes any device whatsoever on an x86 platform with an IOMMU, >> including Xen's fake IOMMU.) >> >> New QEMU always respects the IOMMU, if any, except on PPC. > > This is just a matter of what is the default of the flag, ie we > should have a machine flag that indicates what the default is for > new virtio devices, otherwise, it should be specified per device > as an attribute of the device instance. On x86, I think that even super-peformance-critical virtio devices should always honor the iommu, but that the iommu in question should be a 1:1 iommu. I *think* that x86 supports that. IOW x86 would always set the feature flag. > > I would argue that we should default to "bypass IOMMU" on *all* > architecture due to the performance impact, and to essentially > default to the same behaviour as today. With things like DDW even > powerpc might be able to mostly alleviate the performance impact > so we might to change in the long term, but I tend to prefer > more incremental approaches. As above, there's a difference between "bypass IOMMU" and "there is no IOMMU". x86 and, I think, most other platforms are capable of the latter. I'm not sure PPC is. I think that, in an ideal world, there would be no feature flag and all virtio devices would always respect the IOMMU. Unfortunately we have existing practice in the form of PPC and Q35 iommu=on that conflict with that. > >> New QEMU >> always advertises this feature flag. If iommu=on, QEMU's virtio >> devices refuse to work unless the driver acknowledges the flag. > > This should be configurable. Would any non-PPC user ever configure it differently? I suppose if you want to support old kernels on new QEMU, you'd flip the switch. > >> On PPC, new QEMU will not respect the IOMMU and will not set the flag. >> New kernels will not talk to devices that set the flag. If someone >> wants to fix that, then they get to figure out how. > > I disagree with the kernel bit and I disagree with special casing PPC in > any shape or form in the code. The only difference should be a default > value for the iommu mode of virtio in qemu set per machine. > > You can then feel free to change that default (in a separate patch for > bisectability) on x86 for the sake of Xen. I think we should flip the default everywhere to "respects IOMMU". That's the setting that will work in all cases on new guest + new host, and it's the setting that's safest. vfio will probably always malfunction if given a device that looks like it's behind an IOMMU but doesn't respect it. For people who need the last bit of performance, they should use bus-level controls where available (they should be available everywhere except PPC and maybe arm64) and, ideally, someone would teach PPC how to exclude devices from the IOMMU cleanly if possible. If that can't be done, then there can be an option to bypass the IOMMU the way it's currently done and no one except PPC would do it. PPC really is different from everything except x86 Q35 iommu=on, and the latter is experimental. AFAIK in all other cases, the IOMMU is respected by virtio, but there is no non-1:1 IOMMU. --Andy ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, 2015-07-28 at 16:33 -0700, Andy Lutomirski wrote: > On Tue, Jul 28, 2015 at 4:21 PM, Benjamin Herrenschmidt > wrote: > > On Tue, 2015-07-28 at 15:43 -0700, Andy Lutomirski wrote: > >> Let me try to summarize a proposal: > >> > >> Add a feature flag that indicates IOMMU support. > >> > >> New kernels acknowledge that flag on any device that advertises it. > >> > >> New kernels always respect the IOMMU (except on PowerPC). > > > > Why ? I disagree, the flag should be honored when set in any > > architecture. PowerPC is no different than any other platform in that > > regard. > > Perhaps I should have said instead "someone more familiar with PPC > than I am should figure out what PPC should do". For the non-PPC > case, there is only one instance that I know of in which ignoring the > IOMMU is beneficial, and that case is the experimental Q35 thing. "ppc" is many fairly different platforms, some with iommu, some without, some benefiting from bypass, some less etc... I think ARM will soon be in a similar basket. > If new kernels ignore the IOMMU for devices that don't set the flag > and there are physical devices that already exist and don't set the > flag, then those devices won't work reliably on most modern > non-virtual platforms, PPC included. Are there many virtio physical devices out there ? We are talking about a virtio flag right ? Or have you been considering something else ? > >> New kernels > >> optionally refuse to talk to devices that don't have that feature flag > >> if the device appears to be behind an IOMMU. (This presumably > >> includes any device whatsoever on an x86 platform with an IOMMU, > >> including Xen's fake IOMMU.) > >> > >> New QEMU always respects the IOMMU, if any, except on PPC. > > > > This is just a matter of what is the default of the flag, ie we > > should have a machine flag that indicates what the default is for > > new virtio devices, otherwise, it should be specified per device > > as an attribute of the device instance. > > On x86, I think that even super-peformance-critical virtio devices > should always honor the iommu, but that the iommu in question should > be a 1:1 iommu. I *think* that x86 supports that. IOW x86 would > always set the feature flag. Ok. > > I would argue that we should default to "bypass IOMMU" on *all* > > architecture due to the performance impact, and to essentially > > default to the same behaviour as today. With things like DDW even > > powerpc might be able to mostly alleviate the performance impact > > so we might to change in the long term, but I tend to prefer > > more incremental approaches. > > As above, there's a difference between "bypass IOMMU" and "there is no > IOMMU". x86 and, I think, most other platforms are capable of the > latter. I'm not sure PPC is. Depends on the platform. "pseries" isn't since it's already a paravirtualized plaform, but there are other ppc platforms out there which behave differently. That's why I think: - The kernel should just honor what qemu says, ie, whether the qemu device honors or bypasses the iommu. - Qemu default behaviour should be set via a machine attribute which can be overriden both globally (the machine one) or per-device. > I think that, in an ideal world, there would be no feature flag and > all virtio devices would always respect the IOMMU. Unfortunately we > have existing practice in the form of PPC and Q35 iommu=on that > conflict with that. And possibly more as in this is how the qemu virtio devices are written today, they do not use the proper DMA accessors, they always bypass, whatever the platform is (so sparc would be in the same boat for example). > >> New QEMU > >> always advertises this feature flag. If iommu=on, QEMU's virtio > >> devices refuse to work unless the driver acknowledges the flag. > > > > This should be configurable. > > Would any non-PPC user ever configure it differently? I suppose if > you want to support old kernels on new QEMU, you'd flip the switch. Possibly, have we looked at what ia64, sparc, arm, ... do ? At least sparc has iommus as well. Let's try to not make it an architecture issue. As I said above, we have a kernel that just reacts appropriately based on what qemu says it's doing, and what qemu does is a per-machine flag to set the default. > >> On PPC, new QEMU will not respect the IOMMU and will not set the flag. > >> New kernels will not talk to devices that set the flag. If someone > >> wants to fix that, then they get to figure out how. > > > > I disagree with the kernel bit and I disagree with special casing PPC in > > any shape or form in the code. The only difference should be a default > > value for the iommu mode of virtio in qemu set per machine. > > > > You can then feel free to change that default (in a separate patch for > > bisectability) on x86 for the sake of Xen. > > I think we should flip the default everywhere to "respects IOMMU". On new machine types, we shouldn't change the behaviour
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, Jul 28, 2015 at 5:36 PM, Benjamin Herrenschmidt wrote: > On Tue, 2015-07-28 at 16:33 -0700, Andy Lutomirski wrote: >> On Tue, Jul 28, 2015 at 4:21 PM, Benjamin Herrenschmidt >> wrote: >> > On Tue, 2015-07-28 at 15:43 -0700, Andy Lutomirski wrote: >> >> Let me try to summarize a proposal: >> >> >> >> Add a feature flag that indicates IOMMU support. >> >> >> >> New kernels acknowledge that flag on any device that advertises it. >> >> >> >> New kernels always respect the IOMMU (except on PowerPC). >> > >> > Why ? I disagree, the flag should be honored when set in any >> > architecture. PowerPC is no different than any other platform in that >> > regard. >> >> Perhaps I should have said instead "someone more familiar with PPC >> than I am should figure out what PPC should do". For the non-PPC >> case, there is only one instance that I know of in which ignoring the >> IOMMU is beneficial, and that case is the experimental Q35 thing. > > "ppc" is many fairly different platforms, some with iommu, some without, > some benefiting from bypass, some less etc... I think ARM will soon be > in a similar basket. > >> If new kernels ignore the IOMMU for devices that don't set the flag >> and there are physical devices that already exist and don't set the >> flag, then those devices won't work reliably on most modern >> non-virtual platforms, PPC included. > > Are there many virtio physical devices out there ? We are talking about > a virtio flag right ? Or have you been considering something else ? Yes, virtio flag. I dislike having a virtio flag at all, but so far no one has come up with any better ideas. If there was a reliable, cross-platform mechanism for per-device PCI bus properties, I'd be all for using that instead. > >> >> New kernels >> >> optionally refuse to talk to devices that don't have that feature flag >> >> if the device appears to be behind an IOMMU. (This presumably >> >> includes any device whatsoever on an x86 platform with an IOMMU, >> >> including Xen's fake IOMMU.) >> >> >> >> New QEMU always respects the IOMMU, if any, except on PPC. >> > >> > This is just a matter of what is the default of the flag, ie we >> > should have a machine flag that indicates what the default is for >> > new virtio devices, otherwise, it should be specified per device >> > as an attribute of the device instance. >> >> On x86, I think that even super-peformance-critical virtio devices >> should always honor the iommu, but that the iommu in question should >> be a 1:1 iommu. I *think* that x86 supports that. IOW x86 would >> always set the feature flag. > > Ok. > >> > I would argue that we should default to "bypass IOMMU" on *all* >> > architecture due to the performance impact, and to essentially >> > default to the same behaviour as today. With things like DDW even >> > powerpc might be able to mostly alleviate the performance impact >> > so we might to change in the long term, but I tend to prefer >> > more incremental approaches. >> >> As above, there's a difference between "bypass IOMMU" and "there is no >> IOMMU". x86 and, I think, most other platforms are capable of the >> latter. I'm not sure PPC is. > > Depends on the platform. "pseries" isn't since it's already a > paravirtualized plaform, but there are other ppc platforms out there > which behave differently. That's why I think: > > - The kernel should just honor what qemu says, ie, whether the qemu > device honors or bypasses the iommu. Except for vfio, which maybe just needs a special case: vfio checks if the device claims to be virtio and doesn't set the flag, in which case vfio just refuses to bind the device. > > - Qemu default behaviour should be set via a machine attribute which > can be overriden both globally (the machine one) or per-device. > >> I think that, in an ideal world, there would be no feature flag and >> all virtio devices would always respect the IOMMU. Unfortunately we >> have existing practice in the form of PPC and Q35 iommu=on that >> conflict with that. > > And possibly more as in this is how the qemu virtio devices are written > today, they do not use the proper DMA accessors, they always bypass, > whatever the platform is (so sparc would be in the same boat for > example). Except that AFAIK Q35 is the only QEMU platform that supports a nontrivial IOMMU in the first place. Are there pseries hosts that have a working IOMMU? Maybe I've just misunderstood. > >> >> New QEMU >> >> always advertises this feature flag. If iommu=on, QEMU's virtio >> >> devices refuse to work unless the driver acknowledges the flag. >> > >> > This should be configurable. >> >> Would any non-PPC user ever configure it differently? I suppose if >> you want to support old kernels on new QEMU, you'd flip the switch. > > Possibly, have we looked at what ia64, sparc, arm, ... do ? At least > sparc has iommus as well. I think (I hope!) that ia64 is irrelevant, and last I checked ARM didn't have a QEMU-emulated IOMMU. Maybe things
Re: [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API
On Tue, 2015-07-28 at 17:47 -0700, Andy Lutomirski wrote: > Yes, virtio flag. I dislike having a virtio flag at all, but so far > no one has come up with any better ideas. If there was a reliable, > cross-platform mechanism for per-device PCI bus properties, I'd be all > for using that instead. There isn't that I know of, so I think it's the best approach we have. .../... > > - The kernel should just honor what qemu says, ie, whether the qemu > > device honors or bypasses the iommu. > > Except for vfio, which maybe just needs a special case: vfio checks if > the device claims to be virtio and doesn't set the flag, in which case > vfio just refuses to bind the device. Right but passing virtio through isn't the highest priority on the radar, but yes, indeed, it should identify them and reject them. > > - Qemu default behaviour should be set via a machine attribute which > > can be overriden both globally (the machine one) or per-device. > > > >> I think that, in an ideal world, there would be no feature flag and > >> all virtio devices would always respect the IOMMU. Unfortunately we > >> have existing practice in the form of PPC and Q35 iommu=on that > >> conflict with that. > > > > And possibly more as in this is how the qemu virtio devices are written > > today, they do not use the proper DMA accessors, they always bypass, > > whatever the platform is (so sparc would be in the same boat for > > example). > > Except that AFAIK Q35 is the only QEMU platform that supports a > nontrivial IOMMU in the first place. Are there pseries hosts that > have a working IOMMU? Maybe I've just misunderstood. You may well be correct, I remember that we actually created the iommu infrastructure to a large extent in qemu for ppc/pseries, then it got extended when q35 came in. > >> >> New QEMU > >> >> always advertises this feature flag. If iommu=on, QEMU's virtio > >> >> devices refuse to work unless the driver acknowledges the flag. > >> > > >> > This should be configurable. > >> > >> Would any non-PPC user ever configure it differently? I suppose if > >> you want to support old kernels on new QEMU, you'd flip the switch. > > > > Possibly, have we looked at what ia64, sparc, arm, ... do ? At least > > sparc has iommus as well. > > I think (I hope!) that ia64 is irrelevant, and last I checked ARM > didn't have a QEMU-emulated IOMMU. Maybe things have changed. Not yet... .../... > > > > On new machine types, we shouldn't change the behaviour of an existing > > machine type, and we should keep the default to 0 on ppc/pseries because > > of backward compatibility issue. But that should be the only place that > > is "ppc specific", ie, a default value in a machine def structure. > > Fair enough, except I still think we should change the default to be > "respect IOMMU" on machine types that don't have an IOMMU in the first > place. Ok, but do it in a separate patch because it *is* a behaviour change to some extent. > That way Xen works with old machine types, and I don't think > we lose anything. > > > > >> That's the setting that will work in all cases on new guest + new > >> host, and it's the setting that's safest. vfio will probably always > >> malfunction if given a device that looks like it's behind an IOMMU but > >> doesn't respect it. For people who need the last bit of performance, > >> they should use bus-level controls where available (they should be > >> available everywhere except PPC and maybe arm64) and, ideally, someone > >> would teach PPC how to exclude devices from the IOMMU cleanly if > >> possible. If that can't be done, then there can be an option to > >> bypass the IOMMU the way it's currently done and no one except PPC > >> would do it. > >> > >> PPC really is different from everything except x86 Q35 iommu=on, and > >> the latter is experimental. AFAIK in all other cases, the IOMMU is > >> respected by virtio, but there is no non-1:1 IOMMU. > > > > What about sparc ? I though it was pretty similar to PPC in that > > regard... > > No clue, honestly. I could be wrong about the set of existing QEMU > machine types. Ok. Cheers, Ben. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
