Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for?non-privileged processes
* Tom Lyon (p...@lyon-about.com) wrote: > On Friday 09 April 2010 02:58:19 am Avi Kivity wrote: > > - access to all config space, but BARs must be translated so userspace > > cannot attack the host > Please elaborate. All of PCI config? All of PCIe config? Seems like a huge > mess. All of config space, but not raw access to all bits. So the MSI/MSI-X capability writes need to be intermediated. There's bits in the header too. And it's not just PCI, it's extended config space as well, drivers may care about finding their whizzybang PCIe capability and doing someting with it (and worse...they are allowed to put device specific registers there, and worse yet...they do!). thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
* Avi Kivity (a...@redhat.com) wrote: > On 04/02/2010 08:05 PM, Greg KH wrote: > - access to all config space, but BARs must be translated so > userspace cannot attack the host Specifically, intermediated access to config space. For example, need to know about MSI/MSI-X updates in config space. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/09/2010 07:34 PM, Tom Lyon wrote: - access to all config space, but BARs must be translated so userspace cannot attack the host Please elaborate. All of PCI config? All of PCIe config? Seems like a huge mess. Yes. Anything a guest's device driver may want to access. The 'check' items are already done, not fully tested; probably available next week. Can we leave the others for future patches? Please? Hey, I was expecting we'd have to do all of this. The requirements list was to get the uio maintainers confirmation that this is going in an acceptable direction. We can definitely proceed incrementally. And I definitely need help with the PCI config stuff. Sure. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for?non-privileged processes
On Fri, Apr 09, 2010 at 09:34:16AM -0700, Tom Lyon wrote: > The 'check' items are already done, not fully tested; probably available next > week. > Can we leave the others for future patches? Please? And I definitely need > help with > the PCI config stuff. Yeah, go in small steps forward. Just post again what you have next week. We can add more functionality step by step. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Friday 09 April 2010 02:58:19 am Avi Kivity wrote: > On 04/02/2010 08:05 PM, Greg KH wrote: > > > >> Currently kvm does device assignment with its own code, I'd like to unify > >> it with uio, not split it off. > >> > >> Separate notifications for msi-x interrupts are just as useful for uio as > >> they are for kvm. > >> > > I agree, there should not be a difference here for KVM vs. the "normal" > > version. > > > > Just so you know what you got into, here are the kvm requirements: > > - msi interrupts delivered via eventfd (these allow us to inject > interrupts from uio to a guest without going through userspace) Check. > - nonlinear iommu mapping (i.e. map discontiguous ranges of the device > address space into ranges of the virtual address space) Check. > - dynamic iommu mapping (support guest memory hotplug) Check. > - unprivileged operation once an admin has assigned a device (my > preferred implementation is to have all operations go through an fd, > which can be passed via SCM_RIGHTS from a privileged application that > opens the file) Check. > - access to all config space, but BARs must be translated so userspace > cannot attack the host Please elaborate. All of PCI config? All of PCIe config? Seems like a huge mess. > - some mechanism which allows us to affine device interrupts with their > target vcpus (eventually, this is vague) Do-able. > - anything mst might add mst? > - a pony Rainbow or glitter? The 'check' items are already done, not fully tested; probably available next week. Can we leave the others for future patches? Please? And I definitely need help with the PCI config stuff. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/02/2010 08:05 PM, Greg KH wrote: Currently kvm does device assignment with its own code, I'd like to unify it with uio, not split it off. Separate notifications for msi-x interrupts are just as useful for uio as they are for kvm. I agree, there should not be a difference here for KVM vs. the "normal" version. Just so you know what you got into, here are the kvm requirements: - msi interrupts delivered via eventfd (these allow us to inject interrupts from uio to a guest without going through userspace) - nonlinear iommu mapping (i.e. map discontiguous ranges of the device address space into ranges of the virtual address space) - dynamic iommu mapping (support guest memory hotplug) - unprivileged operation once an admin has assigned a device (my preferred implementation is to have all operations go through an fd, which can be passed via SCM_RIGHTS from a privileged application that opens the file) - access to all config space, but BARs must be translated so userspace cannot attack the host - some mechanism which allows us to affine device interrupts with their target vcpus (eventually, this is vague) - anything mst might add - a pony -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Fri, Apr 02, 2010 at 09:43:35AM +0300, Avi Kivity wrote: > On 04/01/2010 10:24 PM, Tom Lyon wrote: >> >>> But there are multiple msi-x interrupts, how do you know which one >>> triggered? >>> >> You don't. This would suck for KVM, I guess, but we'd need major rework of >> the >> generic UIO stuff to have a separate event channel for each MSI-X. >> > > Doesn't it suck for non-kvm in the same way? Multiple vectors are there > for a reason. For example, if you have a multiqueue NIC, you'd have to > process all queues instead of just the one that triggered. > >> For my purposes, collapsing all the MSI-Xs into one MSI-look-alike is fine, >> because I'd be using MSI anyways if I could. The weird Intel 82599 VF only >> supports MSI-X. >> >> So one big question is - do we expand the whole UIO framework for KVM >> requirements, or do we split off either KVM or non-VM into a separate driver? >> Hans or Greg - care to opine? >> > > Currently kvm does device assignment with its own code, I'd like to unify > it with uio, not split it off. > > Separate notifications for msi-x interrupts are just as useful for uio as > they are for kvm. I agree, there should not be a difference here for KVM vs. the "normal" version. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/02/2010 12:27 AM, Tom Lyon wrote: kvm really wants the event counter to be an eventfd, that allows hooking it directly to kvm (which can inject an interrupt on an eventfd_signal), can you adapt your patch to do this? I looked further into eventfds - they seem the perfect solution for the MSI/MSI-X interrupts. Will include in V2. They are indeed. Thanks. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/01/2010 10:24 PM, Tom Lyon wrote: But there are multiple msi-x interrupts, how do you know which one triggered? You don't. This would suck for KVM, I guess, but we'd need major rework of the generic UIO stuff to have a separate event channel for each MSI-X. Doesn't it suck for non-kvm in the same way? Multiple vectors are there for a reason. For example, if you have a multiqueue NIC, you'd have to process all queues instead of just the one that triggered. For my purposes, collapsing all the MSI-Xs into one MSI-look-alike is fine, because I'd be using MSI anyways if I could. The weird Intel 82599 VF only supports MSI-X. So one big question is - do we expand the whole UIO framework for KVM requirements, or do we split off either KVM or non-VM into a separate driver? Hans or Greg - care to opine? Currently kvm does device assignment with its own code, I'd like to unify it with uio, not split it off. Separate notifications for msi-x interrupts are just as useful for uio as they are for kvm. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 08:54:14 am Avi Kivity wrote: > On 04/01/2010 06:39 PM, Tom Lyon wrote: > >>> - support for MSI and MSI-X interrupts (the intel 82599 VFs support > >>> only MSI-X) > >> > >> How does a userspace program receive those interrupts? > > > > Same as other UIO drivers - by read()ing an event counter. > > IIRC the usual event counter is /dev/uioX, what's your event counter now? > > kvm really wants the event counter to be an eventfd, that allows hooking > it directly to kvm (which can inject an interrupt on an eventfd_signal), > can you adapt your patch to do this? I looked further into eventfds - they seem the perfect solution for the MSI/MSI-X interrupts. Will include in V2. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thu, Apr 01, 2010 at 12:24:45PM -0700, Tom Lyon wrote: > For my purposes, collapsing all the MSI-Xs into one MSI-look-alike is fine, > because I'd be using MSI anyways if I could. The weird Intel 82599 VF only > supports MSI-X. For KVM this is not fine. The device should look in the guest as it looks in the host. Some devices might only support MSI-X and thus the drivers for it only search for MSI-X and get confused when they only find MSI. > So one big question is - do we expand the whole UIO framework for KVM > requirements, or do we split off either KVM or non-VM into a separate driver? > Hans or Greg - care to opine? We should definitly work towards a single implementation. The KVM device passthrough requirements are not very different from that of userspace device access. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thu, Apr 01, 2010 at 12:18:27PM -0700, Tom Lyon wrote: > On Thursday 01 April 2010 09:07:47 am Joerg Roedel wrote: > > For the KVM use-case we need to be able to specify the io virtual > > address for a given process virtual address. This is not possible with > > the dma-api interface. So if we want to have uio-dma without an hardware > > iommu we need two distinct interfaces for userspace to cover all > > use-cases. I don't think its worth it to have two interfaces. > > I started to add that capability but then realized that the IOMMU API also > doesn't allow it. The map function allows a range of physically contiguous > pages, not virtual. The IOMMU-API allows that. You have to convert the user-virtual addresses into physical addresses first. The current KVM code already does this and uses the IOMMU-API later. You can have a look at the gfn_to_pfn() function for a way to implement this. > My preferred approach would be to add a DMA_ATTR that would request > allocation of DMA at a specific device/iommu address. No, that would be feature duplication between both APIs. Not to mention the implementation hell this additional dma-api feature would cause for the iommu driver developers. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 09:10:57 am Avi Kivity wrote: > On 04/01/2010 07:06 PM, Tom Lyon wrote: > > On Thursday 01 April 2010 08:54:14 am Avi Kivity wrote: > >> On 04/01/2010 06:39 PM, Tom Lyon wrote: > > - support for MSI and MSI-X interrupts (the intel 82599 VFs support > > only MSI-X) > > How does a userspace program receive those interrupts? > >>> > >>> Same as other UIO drivers - by read()ing an event counter. > >> > >> IIRC the usual event counter is /dev/uioX, what's your event counter > >> now? > > > > Exact same mechanism. > > But there are multiple msi-x interrupts, how do you know which one > triggered? You don't. This would suck for KVM, I guess, but we'd need major rework of the generic UIO stuff to have a separate event channel for each MSI-X. For my purposes, collapsing all the MSI-Xs into one MSI-look-alike is fine, because I'd be using MSI anyways if I could. The weird Intel 82599 VF only supports MSI-X. So one big question is - do we expand the whole UIO framework for KVM requirements, or do we split off either KVM or non-VM into a separate driver? Hans or Greg - care to opine? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 09:07:47 am Joerg Roedel wrote: > On Thu, Apr 01, 2010 at 08:40:34AM -0700, Tom Lyon wrote: > > On Thursday 01 April 2010 05:52:18 am Joerg Roedel wrote: > > > > The point of this patch is to beef up the uio_pci_generic driver so > > > > that a non-privileged user process can run a user level driver for > > > > most PCIe devices. This can only be safe if there is an IOMMU in the > > > > system with per-device domains. Privileged users (CAP_SYS_RAWIO) are > > > > allowed if there is no IOMMU. > > > > > > If you rely on an IOMMU you can use the IOMMU-API instead of the > > > DMA-API for dma mappings. This change makes this driver suitable for > > > KVM use too. If the interface is designed clever enough we can even use > > > it for IOMMU emulation for pass-through devices. > > > > The use with privileged processes and no IOMMUs is still quite useful, so > > I'd rather stick with the DMA interface. > > For the KVM use-case we need to be able to specify the io virtual > address for a given process virtual address. This is not possible with > the dma-api interface. So if we want to have uio-dma without an hardware > iommu we need two distinct interfaces for userspace to cover all > use-cases. I don't think its worth it to have two interfaces. > > Joerg I started to add that capability but then realized that the IOMMU API also doesn't allow it. The map function allows a range of physically contiguous pages, not virtual. My preferred approach would be to add a DMA_ATTR that would request allocation of DMA at a specific device/iommu address. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > uio_pci_generic has previously been discussed on the KVM list, but this patch > has nothing to do with KVM, so it is also going to LKML. Just a side note: It would be nice if you Cc:'ed the UIO maintainers (Greg and me) on patches like that. UIO is an actively maintained framework in the kernel. Thanks, Hans -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thu, Apr 01, 2010 at 05:25:04PM +0300, Michael S. Tsirkin wrote: > On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > > uio_pci_generic has previously been discussed on the KVM list, but this > > patch > > has nothing to do with KVM, so it is also going to LKML. > > > > The point of this patch is to beef up the uio_pci_generic driver so that a > > non-privileged user process can run a user level driver for most PCIe > > devices. This can only be safe if there is an IOMMU in the system with > > per-device domains. > > Why? Per-guest domain should be safe enough. Hardware IOMMUs don't have something like a per-guest domain ;-) Anyway, if we want to emulate an IOMMU in the guest and make this working for pass-through devices too we need more than one domain per guest. Essentially we may need one domain per device. > > Privileged users (CAP_SYS_RAWIO) are allowed if there is > > no IOMMU. > > qemu does not support it, I doubt this last option is worth having. Agreed. > For this reason, I think we should address the problem somwwhat > differently: > - Create a character device to represent the iommu > - This device will handle memory locking etc > - Allow binding this device to iommu > - Allow other operations only after iommu is bound Yes, something like this is needed. But I think we can implement this in the generic uio-pci-driver. A seperate interface which basically passes the iommu-api functions to userspace doesn't make sense because it would also be device-centric like the uio-pci-driver. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > uio_pci_generic has previously been discussed on the KVM list, but this patch > has nothing to do with KVM, so it is also going to LKML. > > The point of this patch is to beef up the uio_pci_generic driver so that a > non-privileged user process can run a user level driver for most PCIe > devices. This can only be safe if there is an IOMMU in the system with > per-device domains. Why? Per-guest domain should be safe enough. > Privileged users (CAP_SYS_RAWIO) are allowed if there is > no IOMMU. qemu does not support it, I doubt this last option is worth having. > Specifically, I seek to allow low-latency user level network drivers (non > tcp/ip) which directly access SR-IOV style virtual network adapters, for use > with packages such as OpenMPI. > > Key areas of change: > - ioctl extensions to allow registration and dma mapping of memory regions, > with lock accounting > - support for mmu notifier driven de-mapping > - support for MSI and MSI-X interrupts (the intel 82599 VFs support only > MSI-X) > - allowing interrupt enabling and device register mapping all > through /dev/uio* so that permissions may be granted just by chmod > on /dev/uio* For non-priveledged users, we need a way to enforce that device is bound to an iommu. Further, locking really needs to be scoped with iommu domain existance and with iommu mappings: as long as a page is mapped in iommu, it must be locked. This patch does not seem to enforce that. Also note that what we really want is a single iommu domain per guest, not per device. For this reason, I think we should address the problem somwwhat differently: - Create a character device to represent the iommu - This device will handle memory locking etc - Allow binding this device to iommu - Allow other operations only after iommu is bound Thanks! -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/01/2010 07:06 PM, Tom Lyon wrote: On Thursday 01 April 2010 08:54:14 am Avi Kivity wrote: On 04/01/2010 06:39 PM, Tom Lyon wrote: - support for MSI and MSI-X interrupts (the intel 82599 VFs support only MSI-X) How does a userspace program receive those interrupts? Same as other UIO drivers - by read()ing an event counter. IIRC the usual event counter is /dev/uioX, what's your event counter now? Exact same mechanism. But there are multiple msi-x interrupts, how do you know which one triggered? kvm really wants the event counter to be an eventfd, that allows hooking it directly to kvm (which can inject an interrupt on an eventfd_signal), can you adapt your patch to do this? My patch does not currently go anywhere near the read/fd logic of /dev/uioX. I think a separate patch would be appropriate. Sure. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/01/2010 03:08 AM, Tom Lyon wrote: uio_pci_generic has previously been discussed on the KVM list, but this patch has nothing to do with KVM, so it is also going to LKML. (needs to go to lkml even if it was for kvm) The point of this patch is to beef up the uio_pci_generic driver so that a non-privileged user process can run a user level driver for most PCIe devices. This can only be safe if there is an IOMMU in the system with per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if there is no IOMMU. Specifically, I seek to allow low-latency user level network drivers (non tcp/ip) which directly access SR-IOV style virtual network adapters, for use with packages such as OpenMPI. Key areas of change: - ioctl extensions to allow registration and dma mapping of memory regions, with lock accounting - support for mmu notifier driven de-mapping Note that current iommus/devices don't support restart-on-fault dma, so userspace drivers will have to lock memory so that it is not swapped out. I don't think this prevents page migration, though. - support for MSI and MSI-X interrupts (the intel 82599 VFs support only MSI-X) How does a userspace program receive those interrupts? - allowing interrupt enabling and device register mapping all through /dev/uio* so that permissions may be granted just by chmod on /dev/uio* That was always broken with the sysfs interface. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On 04/01/2010 06:39 PM, Tom Lyon wrote: - support for MSI and MSI-X interrupts (the intel 82599 VFs support only MSI-X) How does a userspace program receive those interrupts? Same as other UIO drivers - by read()ing an event counter. IIRC the usual event counter is /dev/uioX, what's your event counter now? kvm really wants the event counter to be an eventfd, that allows hooking it directly to kvm (which can inject an interrupt on an eventfd_signal), can you adapt your patch to do this? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 08:54:14 am Avi Kivity wrote: > On 04/01/2010 06:39 PM, Tom Lyon wrote: > >>> - support for MSI and MSI-X interrupts (the intel 82599 VFs support > >>> only MSI-X) > >> > >> How does a userspace program receive those interrupts? > > > > Same as other UIO drivers - by read()ing an event counter. > > IIRC the usual event counter is /dev/uioX, what's your event counter now? Exact same mechanism. > > kvm really wants the event counter to be an eventfd, that allows hooking > it directly to kvm (which can inject an interrupt on an eventfd_signal), > can you adapt your patch to do this? My patch does not currently go anywhere near the read/fd logic of /dev/uioX. I think a separate patch would be appropriate. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thu, Apr 01, 2010 at 08:40:34AM -0700, Tom Lyon wrote: > On Thursday 01 April 2010 05:52:18 am Joerg Roedel wrote: > > > The point of this patch is to beef up the uio_pci_generic driver so that > > > a non-privileged user process can run a user level driver for most PCIe > > > devices. This can only be safe if there is an IOMMU in the system with > > > per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if > > > there is no IOMMU. > > > > If you rely on an IOMMU you can use the IOMMU-API instead of the DMA-API > > for dma mappings. This change makes this driver suitable for KVM use > > too. If the interface is designed clever enough we can even use it for > > IOMMU emulation for pass-through devices. > The use with privileged processes and no IOMMUs is still quite useful, so I'd > rather stick with the DMA interface. For the KVM use-case we need to be able to specify the io virtual address for a given process virtual address. This is not possible with the dma-api interface. So if we want to have uio-dma without an hardware iommu we need two distinct interfaces for userspace to cover all use-cases. I don't think its worth it to have two interfaces. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 07:25:04 am Michael S. Tsirkin wrote: > On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > > uio_pci_generic has previously been discussed on the KVM list, but this > > patch has nothing to do with KVM, so it is also going to LKML. > > > > The point of this patch is to beef up the uio_pci_generic driver so that > > a non-privileged user process can run a user level driver for most PCIe > > devices. This can only be safe if there is an IOMMU in the system with > > per-device domains. > > Why? Per-guest domain should be safe enough. I'm not sure what 'per-guest' means in an ordinary process context. > > > Privileged users (CAP_SYS_RAWIO) are allowed if there is > > no IOMMU. > > qemu does not support it, I doubt this last option is worth having. This is extremely useful in non IOMMU systems - again, we're talking ordinary processes, nothing to do with VMs. As long as the program can be trusted, e.g., in embedded apps. > > > Specifically, I seek to allow low-latency user level network drivers (non > > tcp/ip) which directly access SR-IOV style virtual network adapters, for > > use with packages such as OpenMPI. > > > > Key areas of change: > > - ioctl extensions to allow registration and dma mapping of memory > > regions, with lock accounting > > - support for mmu notifier driven de-mapping > > - support for MSI and MSI-X interrupts (the intel 82599 VFs support only > > MSI-X) > > - allowing interrupt enabling and device register mapping all > > through /dev/uio* so that permissions may be granted just by chmod > > on /dev/uio* > > For non-priveledged users, we need a way to enforce that > device is bound to an iommu. Right now I just use iommu_found - assuming that if we have one, it is in use. Something better would be nice. > Further, locking really needs to be scoped with iommu domain existance > and with iommu mappings: as long as a page is mapped in iommu, > it must be locked. This patch does not seem to enforce that. Sure it does. The DMA API - get_user_pages and dma_map_sg lock pages into the MMU and the IOMMU. The MMU notifier unlocks if the user forgets to do it explicitly. > Also note that what we really want is a single iommu domain per guest, > not per device. For my networking applications, I will need the ability to talk to multiple devices on potentially separate IOMMUs. What would per-guest mean then? > > For this reason, I think we should address the problem somwwhat > differently: > - Create a character device to represent the iommu > - This device will handle memory locking etc > - Allow binding this device to iommu > - Allow other operations only after iommu is bound There are still per-device issues with locking - in particular the size of the device's DMA address space. The DMA API already handles this - why not use it? It would be nice to have a way to test whether a device is truly covered by an IOMMU, but today it appears that if an IOMMU exists, then it covers all devices (at least as far as I can see for x86). -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 05:52:18 am Joerg Roedel wrote: > On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > > uio_pci_generic has previously been discussed on the KVM list, but this > > patch has nothing to do with KVM, so it is also going to LKML. > > But since you send it to the KVM list it should be suitable for KVM too, > no? I know not. > > > The point of this patch is to beef up the uio_pci_generic driver so that > > a non-privileged user process can run a user level driver for most PCIe > > devices. This can only be safe if there is an IOMMU in the system with > > per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if > > there is no IOMMU. > > If you rely on an IOMMU you can use the IOMMU-API instead of the DMA-API > for dma mappings. This change makes this driver suitable for KVM use > too. If the interface is designed clever enough we can even use it for > IOMMU emulation for pass-through devices. The use with privileged processes and no IOMMUs is still quite useful, so I'd rather stick with the DMA interface. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Thursday 01 April 2010 02:09:09 am Avi Kivity wrote: > On 04/01/2010 03:08 AM, Tom Lyon wrote: > > uio_pci_generic has previously been discussed on the KVM list, but this > > patch has nothing to do with KVM, so it is also going to LKML. > > (needs to go to lkml even if it was for kvm) > > > The point of this patch is to beef up the uio_pci_generic driver so that > > a non-privileged user process can run a user level driver for most PCIe > > devices. This can only be safe if there is an IOMMU in the system with > > per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if > > there is no IOMMU. > > > > Specifically, I seek to allow low-latency user level network drivers (non > > tcp/ip) which directly access SR-IOV style virtual network adapters, for > > use with packages such as OpenMPI. > > > > Key areas of change: > > - ioctl extensions to allow registration and dma mapping of memory > > regions, with lock accounting > > - support for mmu notifier driven de-mapping > > Note that current iommus/devices don't support restart-on-fault dma, so > userspace drivers will have to lock memory so that it is not swapped > out. I don't think this prevents page migration, though. The driver provides a way to lock memory for DMA; the mmu notifier support is to catch things when the user accidentally frees locked pages. > > - support for MSI and MSI-X interrupts (the intel 82599 VFs support only > > MSI-X) > > How does a userspace program receive those interrupts? Same as other UIO drivers - by read()ing an event counter. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
On Wed, Mar 31, 2010 at 05:08:38PM -0700, Tom Lyon wrote: > uio_pci_generic has previously been discussed on the KVM list, but this patch > has nothing to do with KVM, so it is also going to LKML. But since you send it to the KVM list it should be suitable for KVM too, no? > The point of this patch is to beef up the uio_pci_generic driver so that a > non-privileged user process can run a user level driver for most PCIe > devices. This can only be safe if there is an IOMMU in the system with > per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if there is > no IOMMU. If you rely on an IOMMU you can use the IOMMU-API instead of the DMA-API for dma mappings. This change makes this driver suitable for KVM use too. If the interface is designed clever enough we can even use it for IOMMU emulation for pass-through devices. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/1] uio_pci_generic: extensions to allow access for non-privileged processes
uio_pci_generic has previously been discussed on the KVM list, but this patch has nothing to do with KVM, so it is also going to LKML. The point of this patch is to beef up the uio_pci_generic driver so that a non-privileged user process can run a user level driver for most PCIe devices. This can only be safe if there is an IOMMU in the system with per-device domains. Privileged users (CAP_SYS_RAWIO) are allowed if there is no IOMMU. Specifically, I seek to allow low-latency user level network drivers (non tcp/ip) which directly access SR-IOV style virtual network adapters, for use with packages such as OpenMPI. Key areas of change: - ioctl extensions to allow registration and dma mapping of memory regions, with lock accounting - support for mmu notifier driven de-mapping - support for MSI and MSI-X interrupts (the intel 82599 VFs support only MSI-X) - allowing interrupt enabling and device register mapping all through /dev/uio* so that permissions may be granted just by chmod on /dev/uio* -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
