Re: VT-d support for device assignment
On Saturday, September 13, 2008 5:46 pm Avi Kivity wrote: Amit Shah wrote: Sorry for the resends; this one fixes two compile errors introduced by me and a warning. Applied both, thanks. You can add my s-o-b to the IOMMU patch if you want Avi, it's fine with me if you push those particular PCI bits (along with the recent build fix). Thanks, Jesse -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: Sorry for the resends; this one fixes two compile errors introduced by me and a warning. Applied both, thanks. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 10:10:52AM +0300, Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on I don't like the vtd=on flag. I think its not good to have seperate parameters for Intel and AMD IOMMUs. So I think a parameter called iommu=on is the better choice. Joerg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. Since the last send, I've updated the 2nd patch to expose KVM_CAP_IOMMU and check for the existence of an IOMMU before using one. I've also preserved the 'From' field for both the patches so that the original authors are credited for the patches. The command line currently should be invoked with the parameter: -pcidevice dev=00:13.0,dma=iommu Why not default to iommu if one is available? btw, -usbdevice prefixes host devices with host:, no? maybe adopt the convention for -pcidevice? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Tuesday 26 Aug 2008 14:40:57 Avi Kivity wrote: Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. Since the last send, I've updated the 2nd patch to expose KVM_CAP_IOMMU and check for the existence of an IOMMU before using one. I've also preserved the 'From' field for both the patches so that the original authors are credited for the patches. The command line currently should be invoked with the parameter: -pcidevice dev=00:13.0,dma=iommu Why not default to iommu if one is available? I've set that as default now. What should the parameter be to disable just the iommu? dma=noiommu looks ugly. Or do we not want to disable an iommu at all (even for debugging or testing something else, like pvdma)? btw, -usbdevice prefixes host devices with host:, no? maybe adopt the convention for -pcidevice? Sounds ok. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: -pcidevice dev=00:13.0,dma=iommu Why not default to iommu if one is available? I've set that as default now. What should the parameter be to disable just the iommu? dma=noiommu looks ugly. dma=none ? Or do we not want to disable an iommu at all (even for debugging or testing something else, like pvdma)? Good question. Maybe it's safer to always enable it, and to let developers hack it out. We must be careful of people's feet. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: So long as there is no potential for performance or security impact, having pvdma turned on automatically is better. We could still have dma=noparavirt to disable it. So we fail the hypercalls once the guest tries to contact us? Yes. Also don't expose the feature bit to the guest. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Muli Ben-Yehuda wrote: On Sat, Aug 23, 2008 at 12:33:46PM +0300, Avi Kivity wrote: So long as there is no potential for performance or security impact, having pvdma turned on automatically is better. We could still have dma=noparavirt to disable it. pvdma has advantages and disadvantages. It has a significant performance impact (compared to direct map) That needs to be fixed. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Friday 22 Aug 2008 23:51:15 Avi Kivity wrote: Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on You mean, iommu=on. I did mean vtd=on, since we'll also have AMD's iommu implementation here. So something like: -pcidevice dev=00:13.0,vtd=on,pvdma=on or -pcidevice dev=00:13.0,amd-iommu=on,pvdma=on or do you mean we should autodetect which IOMMU we have and then select the appropriate one instead of bothering the user with it? Hmm, that seems a better UI and also such startup scripts can be ported across architectures. Or rather dma=iommu dma=none (1:1 mapping, or dma-less devices, or I'm Feeling Lucky) dma=cooperative (paravirt) This looks much better! Once we have KVM_CAP_VTD, KVM_CAP_AMD_IOMMU and KVM_CAP_PVDMA, dma=iommu means use either of VTD or AMD, whichever one is available dma=none means I'm feeling lucky PVDMA will automatically get used if the guest has PVDMA support compiled in. Enabling/disabling pvdma would be a guest option rather than a host option, I think (host only exposes CAP_PVDMA). Is this ok? Amit -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: * On Friday 22 Aug 2008 23:51:15 Avi Kivity wrote: Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on You mean, iommu=on. I did mean vtd=on, since we'll also have AMD's iommu implementation here. So something like: -pcidevice dev=00:13.0,vtd=on,pvdma=on or -pcidevice dev=00:13.0,amd-iommu=on,pvdma=on or do you mean we should autodetect which IOMMU we have and then select the appropriate one instead of bothering the user with it? Hmm, that seems a better UI and also such startup scripts can be ported across architectures. Yes of course. Note there's no need for kvm to autodetect the iommu either; I won't let the amd iommu in without a proper abstraction via an iommu api. Or rather dma=iommu dma=none (1:1 mapping, or dma-less devices, or I'm Feeling Lucky) dma=cooperative (paravirt) This looks much better! Once we have KVM_CAP_VTD, KVM_CAP_AMD_IOMMU and KVM_CAP_PVDMA, Why KVM_CAP_VTD and KVM_CAP_AMD_IOMMU? Do they actually have differences? if so, the capabilities should report the differences as features, not as vendor identifiers. dma=iommu means use either of VTD or AMD, whichever one is available dma=none means I'm feeling lucky PVDMA will automatically get used if the guest has PVDMA support compiled in. Enabling/disabling pvdma would be a guest option rather than a host option, I think (host only exposes CAP_PVDMA). Is this ok? So long as there is no potential for performance or security impact, having pvdma turned on automatically is better. We could still have dma=noparavirt to disable it. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Saturday 23 Aug 2008 15:03:46 Avi Kivity wrote: Amit Shah wrote: * On Friday 22 Aug 2008 23:51:15 Avi Kivity wrote: Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on You mean, iommu=on. I did mean vtd=on, since we'll also have AMD's iommu implementation here. So something like: -pcidevice dev=00:13.0,vtd=on,pvdma=on or -pcidevice dev=00:13.0,amd-iommu=on,pvdma=on or do you mean we should autodetect which IOMMU we have and then select the appropriate one instead of bothering the user with it? Hmm, that seems a better UI and also such startup scripts can be ported across architectures. Yes of course. Note there's no need for kvm to autodetect the iommu either; I won't let the amd iommu in without a proper abstraction via an iommu api. Yes; we're going to have an API once the Intel IOMMU support goes in. Or rather dma=iommu dma=none (1:1 mapping, or dma-less devices, or I'm Feeling Lucky) dma=cooperative (paravirt) This looks much better! Once we have KVM_CAP_VTD, KVM_CAP_AMD_IOMMU and KVM_CAP_PVDMA, Why KVM_CAP_VTD and KVM_CAP_AMD_IOMMU? Do they actually have differences? if so, the capabilities should report the differences as features, not as vendor identifiers. Agreed. dma=iommu means use either of VTD or AMD, whichever one is available dma=none means I'm feeling lucky PVDMA will automatically get used if the guest has PVDMA support compiled in. Enabling/disabling pvdma would be a guest option rather than a host option, I think (host only exposes CAP_PVDMA). Is this ok? So long as there is no potential for performance or security impact, having pvdma turned on automatically is better. We could still have dma=noparavirt to disable it. So we fail the hypercalls once the guest tries to contact us? Amit -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 10:10:52AM +0300, Amit Shah wrote: The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on Could you please send your changes as a seperate, follow-on patch? The original patch is complicated enough, and that way the authorship information does not get muddled further, and it's much easier to see what you changed. Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Saturday 23 Aug 2008 15:27:47 Muli Ben-Yehuda wrote: On Fri, Aug 22, 2008 at 10:10:52AM +0300, Amit Shah wrote: The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on Could you please send your changes as a seperate, follow-on patch? The original patch is complicated enough, and that way the authorship information does not get muddled further, and it's much easier to see what you changed. I already sent my patch in a separate mail. The authorship info and the commit log stays the same; just contains my signoff. A new patch based on our current discussion will have to be spun anyway. Amit -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Sat, Aug 23, 2008 at 03:55:25PM +0530, Amit Shah wrote: The authorship info and the commit log stays the same; just contains my signoff. Actually, unless you add an explicit 'From:' header, the email From header is used by git as the author of the patch. Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Saturday 23 Aug 2008 16:10:54 Muli Ben-Yehuda wrote: On Sat, Aug 23, 2008 at 03:55:25PM +0530, Amit Shah wrote: The authorship info and the commit log stays the same; just contains my signoff. Actually, unless you add an explicit 'From:' header, the email From header is used by git as the author of the patch. If you notice, Patch 1/2, VT-d one has the From: line set to Allen's name (as Ben had sent). The patch 2/2 was modified by me, so it gets my 'From'. Amit -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Sat, Aug 23, 2008 at 04:41:02PM +0530, Amit Shah wrote: * On Saturday 23 Aug 2008 16:10:54 Muli Ben-Yehuda wrote: On Sat, Aug 23, 2008 at 03:55:25PM +0530, Amit Shah wrote: The authorship info and the commit log stays the same; just contains my signoff. Actually, unless you add an explicit 'From:' header, the email From header is used by git as the author of the patch. If you notice, Patch 1/2, VT-d one has the From: line set to Allen's name (as Ben had sent). The patch 2/2 was modified by me, so it gets my 'From'. Hence my comment about muddled authorship information and why it was better to keep your changes as a separate patch. Anyway, moving this thread to more constructive avenues: I think you said you are planning to implement and submit Avi's dma=xxx suggestion, and also had an outstanding userspace patch? Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Saturday 23 Aug 2008 17:41:32 Muli Ben-Yehuda wrote: On Sat, Aug 23, 2008 at 04:41:02PM +0530, Amit Shah wrote: * On Saturday 23 Aug 2008 16:10:54 Muli Ben-Yehuda wrote: On Sat, Aug 23, 2008 at 03:55:25PM +0530, Amit Shah wrote: The authorship info and the commit log stays the same; just contains my signoff. Actually, unless you add an explicit 'From:' header, the email From header is used by git as the author of the patch. If you notice, Patch 1/2, VT-d one has the From: line set to Allen's name (as Ben had sent). The patch 2/2 was modified by me, so it gets my 'From'. Hence my comment about muddled authorship information and why it was better to keep your changes as a separate patch. I don't know what you mean; isn't that the way patches flow? This is how collaboration is shown. Anyway, moving this thread to more constructive avenues: I think you said you are planning to implement and submit Avi's dma=xxx suggestion, and also had an outstanding userspace patch? Yes; both are in the baking oven. Amit. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, 22 Aug 2008, Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on Amit, Is it possible to log any physical PCI accesses from the guest? This could be handy for reverse-engineering device drivers, among other things. Thanks, -Byron -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
* On Friday 22 Aug 2008 19:24:13 Byron Stanoszek wrote: On Fri, 22 Aug 2008, Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on Amit, Is it possible to log any physical PCI accesses from the guest? This could be handy for reverse-engineering device drivers, among other things. Hello Byron, All the PCI accesses happen from the userspace components in qemu; it's certainly possible to log activity. Amit -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 09:54:13AM -0400, Byron Stanoszek wrote: Is it possible to log any physical PCI accesses from the guest? This could be handy for reverse-engineering device drivers, among other things. With PCI passthrough It is possible in theory to log mmio's and pio's executed by the guest, as well as interrupts delivered to the guest, although it will have a performance impact. It is not possible to log DMAs in a simple manner, but you could hook up a PCI analyzer to get the missing bits. Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
Amit Shah wrote: The following two patches contain VT-d support for device assignment for KVM guests. The first patch contains the changes that are required to the generic VT-d code. The second patch contains the changes to KVM. I've updated the 2nd patch to use VT-d only when requested by a parameter on the command line, making it easier to support iommu with pvdma and multiple iommu types. The command line currently should be invoked as: -pcidevice dev=00:13.0,vtd=on You mean, iommu=on. Or rather dma=iommu dma=none (1:1 mapping, or dma-less devices, or I'm Feeling Lucky) dma=cooperative (paravirt) -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 09:12:32PM +0300, Avi Kivity wrote: Muli Ben-Yehuda wrote: With PCI passthrough It is possible in theory to log mmio's and pio's executed by the guest, as well as interrupts delivered to the guest, although it will have a performance impact. It is not possible to log DMAs in a simple manner, but you could hook up a PCI analyzer to get the missing bits. Well, with a pci analyzer you can log anything going in and out of the device; you don't need a virtual machine monitor. Yep, that's true. I was thinking along the lines of doing the bulk of reverse engineering without a PCI analyzer (if you understand the device well enough, looking at the DMA data may not be all that important) and then using a PCI analyzer just for any remaining bits. By the way, with effort we should be able to capture some DMAs too, either by taking before and after snapshots of VM memory (for DMA write to memory) and/or by triggering DMA faults (by not mapping the VM's memory in an isolation-capable IOMMU and enabling the IOMMU). Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 09:21:15PM +0300, Avi Kivity wrote: You mean, iommu=on. Agreed. This should also be the default if not dma=xxx is specfied and a capable IOMMU is found. Or rather dma=iommu dma=none (1:1 mapping, or dma-less devices, or I'm Feeling Lucky) dma=cooperative (paravirt) What does the 'cooperative' mean? I assume you are thinking of PVDMA-without-IOMMU, but in that case the name is misleading, since you can do cooperative (PVDMA) just fine with an IOMMU. Perhaps dma=itrusttheguest, since with PVDMA0-without-IOMMU the guest is trusted to not screw up the system? Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 09:02:33PM +0530, Amit Shah wrote: All the PCI accesses happen from the userspace components in qemu; it's certainly possible to log activity. Actually, with direct-mmio we bypass QEMU and get a nice boost in performance, and it should be possible to bypass QEMU for (most) PIO's too. Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: VT-d support for device assignment
On Fri, Aug 22, 2008 at 09:37:16PM +0300, Avi Kivity wrote: Actually, with direct-mmio we bypass QEMU and get a nice boost in performance, and it should be possible to bypass QEMU for (most) PIO's too. Since there is no pio remapping, there's no way to ensure there's a free pio range where the guest maps its ports. True, and we also need to trap pio's for things like configuration space accesses that the host needs to know about, such as BAR re-mappings. Hence my (parenthetical) comment with regards to most. It's not at all clear that the gain will be worth the complexity to decide which pio's to trap and which to let through directly. Cheers, Muli -- Workshop on I/O Virtualization (WIOV '08) Co-located with OSDI '08, Dec 2008, San Diego, CA http://www.usenix.org/wiov08 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
