RE: Device specific pass through in host systems - discuss user interface
> > The present value shows the existing type of default domain. > > If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he > attempts to write the new value. > > Kernel performs checks to make sure that the driver in unbinded and it's > > safe > to change the default domain type. > > After successfully changing the default_domain type internally, kernel > > reflects > the new value in the file. > > Ay errors in the process will be reported in dmesg. > > I prefer this way. Writing to the file should fail with -EBUSY when it is not > safe to > change the default domain-type. Writing should only succeed when no device in > the group is assigned to a device driver. Thanks a lot! Joerg for the reply. I have just sent out an RFC of this patch set to the IOMMU mailing list. Could you please take a look at it and let me know your feedback? Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
On Tue, Jun 11, 2019 at 05:27:15PM +, Prakhya, Sai Praneeth wrote: > 1. Since we already have "type" file, which is "read-only", we could make it > R/W. > > The present value shows the existing type of default domain. > If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he > attempts to write the new value. > Kernel performs checks to make sure that the driver in unbinded and it's safe > to change the default domain type. > After successfully changing the default_domain type internally, kernel > reflects the new value in the file. > Ay errors in the process will be reported in dmesg. I prefer this way. Writing to the file should fail with -EBUSY when it is not safe to change the default domain-type. Writing should only succeed when no device in the group is assigned to a device driver. Regards, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: Device specific pass through in host systems - discuss user interface
> > > > Sure! Makes sense.. per-group default domain type sounds good. > > > > I am planning to implement an RFC (supporting only runtime case for > > now) which works as below > > > > 1. User unbinds the driver by writing to sysfs 2. User puts a group in > > pass through mode by writing "1" to > > /sys/kernel/iommu_groups//pt > > might be better to read current value of default domain for that group.. > /sys/kernel/iommu_groups//default_domain Presently, we already have a file that gives out "type" of default_domain and the file is /sys/kernel/iommu_groups//type > reading the above value shows current setting. > provide a differnet file next_def_domain, and you can echo "pt" or > "dma_domain" > to switch to pass-through, or normal dma isolation mode. We have couple of options here: 1. Since we already have "type" file, which is "read-only", we could make it R/W. The present value shows the existing type of default domain. If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he attempts to write the new value. Kernel performs checks to make sure that the driver in unbinded and it's safe to change the default domain type. After successfully changing the default_domain type internally, kernel reflects the new value in the file. Ay errors in the process will be reported in dmesg. 2. As you have suggested, we could have a *new* file named "next_def_domain_type", which takes string as an input. Please let me know if there is any preference among these approaches and why :) Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
On Mon, Jun 10, 2019 at 09:38:11PM -0700, Sai Praneeth Prakhya wrote: > Hi All, > > + Sohil and Rob Clark (as there are dropped from CC'list) > > > > > Most iommu vendor drivers have switched from per-device to per-group > > > > domain (a.k.a. default domain). So per-group pass-through mode makes > > more sense? > > > > > > > > By the way, can we extend this to "per-group default domain type", > > > > instead of only "per-group pass-through mode"? Currently we have > > > > system level default domain type, if we have finer granularity of > > > > default domain type, both iommu drivers and end users will benefit from > > > > it. > > > > > > Sure! Makes sense.. per-group default domain type sounds good. > > I am planning to implement an RFC (supporting only runtime case for now) > which works as below > > 1. User unbinds the driver by writing to sysfs > 2. User puts a group in pass through mode by writing "1" to > /sys/kernel/iommu_groups//pt might be better to read current value of default domain for that group.. /sys/kernel/iommu_groups//default_domain reading the above value shows current setting. provide a differnet file next_def_domain, and you can echo "pt" or "dma_domain" to switch to pass-through, or normal dma isolation mode. For devices that automatically set to pass through today like graphics, or isoch audio you can show "pt" as default_domain. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: Device specific pass through in host systems - discuss user interface
Hi All, + Sohil and Rob Clark (as there are dropped from CC'list) > > > Most iommu vendor drivers have switched from per-device to per-group > > > domain (a.k.a. default domain). So per-group pass-through mode makes > more sense? > > > > > > By the way, can we extend this to "per-group default domain type", > > > instead of only "per-group pass-through mode"? Currently we have > > > system level default domain type, if we have finer granularity of > > > default domain type, both iommu drivers and end users will benefit from > > > it. > > > > Sure! Makes sense.. per-group default domain type sounds good. I am planning to implement an RFC (supporting only runtime case for now) which works as below 1. User unbinds the driver by writing to sysfs 2. User puts a group in pass through mode by writing "1" to /sys/kernel/iommu_groups//pt 3. User re-binds the driver by writing to sysfs As suggested by Lu, Baolu will look into implementing this by using "per-group default domain type" If anyone has suggestions/comments/concerns, please reply. Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
Hi Sai On Sun, Jun 09, 2019 at 10:41:10PM -0700, Sai Praneeth Prakhya wrote: > > > I am working on an IOMMU driver feature that allows a user to specify > > > if the DMA from a device should be translated by IOMMU or not. > > > Presently, we support only all devices or none mode i.e. if user > > > specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through > > > kernel command line, all the devices would be in pass through mode and > > > we don't have per device granularity, but, we were requested by a > > > customer to selectively put devices in pass through mode and not all. > > > > Most iommu vendor drivers have switched from per-device to per-group domain > > (a.k.a. default domain). So per-group pass-through mode makes more sense? > > > > By the way, can we extend this to "per-group default domain type", instead > > of > > only "per-group pass-through mode"? Currently we have system level default > > domain type, if we have finer granularity of default domain type, both iommu > > drivers and end users will benefit from it. > > Sure! Makes sense.. per-group default domain type sounds good. > > > > I am looking for a consensus on **how the kernel command line argument > > > should look like and path for sysfs entry**. Also, please note that if > > > a device is put in pass through mode it won't be available for the > > > guest and that's ok. > > > > Just out of curiosity, what's the limitation for a device using pass- > > through DMA > > domain to be assignable. > > Sorry! I don't know about assignable devices. Probably, Ashok or Jacob could > answer this question We don't switch the domain for assigned devices. Only the "type" of the default domain is changed from dma-protected to passthrough type. When assigning devices to user-space, there is no change in this proposal. > > Regards, > Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: Device specific pass through in host systems - discuss user interface
> > I am working on an IOMMU driver feature that allows a user to specify > > if the DMA from a device should be translated by IOMMU or not. > > Presently, we support only all devices or none mode i.e. if user > > specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through > > kernel command line, all the devices would be in pass through mode and > > we don't have per device granularity, but, we were requested by a > > customer to selectively put devices in pass through mode and not all. > > Most iommu vendor drivers have switched from per-device to per-group domain > (a.k.a. default domain). So per-group pass-through mode makes more sense? > > By the way, can we extend this to "per-group default domain type", instead of > only "per-group pass-through mode"? Currently we have system level default > domain type, if we have finer granularity of default domain type, both iommu > drivers and end users will benefit from it. Sure! Makes sense.. per-group default domain type sounds good. > > I am looking for a consensus on **how the kernel command line argument > > should look like and path for sysfs entry**. Also, please note that if > > a device is put in pass through mode it won't be available for the > > guest and that's ok. > > Just out of curiosity, what's the limitation for a device using pass- through > DMA > domain to be assignable. Sorry! I don't know about assignable devices. Probably, Ashok or Jacob could answer this question Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
Hi Sai, On 6/7/19 10:24 AM, Prakhya, Sai Praneeth wrote: Hi All, I am working on an IOMMU driver feature that allows a user to specify if the DMA from a device should be translated by IOMMU or not. Presently, we support only all devices or none mode i.e. if user specifies “iommu=pt” [X86] or “iommu.passthrough” [ARM64] through kernel command line, all the devices would be in pass through mode and we don’t have per device granularity, but, we were requested by a customer to selectively put devices in pass through mode and not all. Most iommu vendor drivers have switched from per-device to per-group domain (a.k.a. default domain). So per-group pass-through mode makes more sense? By the way, can we extend this to "per-group default domain type", instead of only "per-group pass-through mode"? Currently we have system level default domain type, if we have finer granularity of default domain type, both iommu drivers and end users will benefit from it. Since, this feature could be generic across architectures, we thought it would be better if the user interface is discussed in the community first. We are envisioning this to be used both during boot time and runtime and hence having a kernel command line argument along with a sysfs entry are needed. So, please pour in your suggestions on how the user interface should look like to make it architecture agnostic. 1.Have a kernel command line argument that takes a list of BDF’s as an input and puts them in pass through mode a.Accepting BDF as an input has a downside – BDF is dynamic and could change if BIOS/OS enumerates a new device in next reboot b.Accepting pair as an input has a downside – What to do when there are multiple such devices and user would like to put only some of them in PT mode 2.Have a sysfs file which takes 1 or 0 as an input to enable/disable pass through mode. Some places that seem to be reasonable are a./sys/class/iommu/dmar0/devices/ b./sys/kernel/iommu_groups//devices I am looking for a consensus on **how the kernel command line argument should look like and path for sysfs entry**. Also, please note that if a device is put in pass through mode it won’t be available for the guest and that’s ok. Just out of curiosity, what's the limitation for a device using pass- through DMA domain to be assignable. Best regards, Baolu Regards, Sai PS: Idea credits: Ashok Raj ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
On Sat, 2019-06-08 at 09:27 +0200, h...@lst.de wrote: > Just curious, what exactly is the use case? Explaining how someone > would wan to use this should drive the way we design an interface for it. Makes sense. Some example use cases: 1. Assume an SR-IOV device and if the admin decides to use some VF's natively (i.e. they are available only to host) and he wants them to be high performing and also believes that his OS is secure enough (so decides to by pass IOMMU). Presently, we don't support this use case because "iommu=pt" kernel command line argument is an all or none feature i.e. either all BDF's are translated through IOMMU or none. So, we would like to propose a per-BDF on/off feature. We would also want it to be run-time (i.e. sysfs based interface) rather than just boot-time interface (kernel command line argument). Also, want to be clear that it's not just SR-IOV devices that we are looking at but could be any unrelated PCIe devices i.e. a SATA device, USB, NIC or GFx. Wherein the admin wants to selectively put NIC and GFx through IOMMU and does not want transactions by SATA and USB to be translated by IOMMU. Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
Just curious, what exactly is the use case? Explaining how someone would wan to use this should drive the way we design an interface for it. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
> It's interesting to see this from a fresh angle which isn't clouded by > other SoC GPU details - thanks for the proposal! A couple more thoughts > jump out immediately... > Thanks a lot! for jumping in and providing your valuable insights :) Sorry! for taking time to reply on this. Did some quick testing on unbinding/binding stuff. > > Since, this feature could be generic across architectures, we thought it > > would be better if the user interface is discussed in the community first. > > We are envisioning this to be used both during boot time and runtime and > > hence having a kernel command line argument along with a sysfs entry are > > needed. So, please pour in your suggestions on how the user interface > > should look like to make it architecture agnostic. > > > > > > 1. Have a kernel command line argument that takes a list of BDF's as > > an input and puts them in pass through mode > > > > a. Accepting BDF as an input has a downside - BDF is dynamic and > > could change if BIOS/OS enumerates a new device in next reboot > > > > b. Accepting pair as an input has a downside - > > What to do when there are multiple such devices and user would like to put > > only some of them in PT mode > > > > c. Not all devices are PCI in the first place. Agreed, maybe we could limit this feature only to PCIe devices :( > > > 2. Have a sysfs file which takes 1 or 0 as an input to enable/disable > > pass through mode. Some places that seem to be reasonable are > > > > a. /sys/class/iommu/dmar0/devices/ > > > > b. /sys/kernel/iommu_groups//devices > > Note that this this works out a bit tricky to actually use, since > changing the meaning of DMA addresses under the device's feet would be > disastrous. Yes, that makes sense. > It can only safely take effect by unbinding and rebinding > the driver and/or resetting the device (as a side note, this starts to > overlap with the IOMMU-drivers-as-modules concept, and whatever solution > to this we end up with may be helpful in making that work too). In most > cases that's probably viable, but not every driver supports unbinding, > and either way what if the device in question is the one hosting the > root filesystem... :/ We would like to propose something like this 1. User will first unbind the device driver by echo "BDF" > /sys/bus/pci/drivers//unbind 2. Set the Pass Through bit (this is to say the intent of the user that upon the next rebind, please make this device as pass through) echo 1 > /sys/class/iommu/dmar0/devices//pt 3. Re-bind the driver again echo "BDF" > /sys/bus/pci/drivers//bind So, if the driver doesn't support unbind then the user cannot put the device in pass through mode, in which case he has to use kernel command line argument. I did unbind on my device that has root file system and after that I was unable to run any command (like cat or echo), which made sense. So, it's upto the user to decide which devices can go through unbind and hence can be put in pass through mode. Since this feature requires sudo privileges, I think, we can assume admin does the right thing. Please let me know what you think about this and also please do suggest if you have any other better ways to deal with this. Regards, Sai ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: Device specific pass through in host systems - discuss user interface
On 07/06/2019 03:24, Prakhya, Sai Praneeth wrote: Hi All, I am working on an IOMMU driver feature that allows a user to specify if the DMA from a device should be translated by IOMMU or not. Presently, we support only all devices or none mode i.e. if user specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through kernel command line, all the devices would be in pass through mode and we don't have per device granularity, but, we were requested by a customer to selectively put devices in pass through mode and not all. It's interesting to see this from a fresh angle which isn't clouded by other SoC GPU details - thanks for the proposal! A couple more thoughts jump out immediately... Since, this feature could be generic across architectures, we thought it would be better if the user interface is discussed in the community first. We are envisioning this to be used both during boot time and runtime and hence having a kernel command line argument along with a sysfs entry are needed. So, please pour in your suggestions on how the user interface should look like to make it architecture agnostic. 1. Have a kernel command line argument that takes a list of BDF's as an input and puts them in pass through mode a. Accepting BDF as an input has a downside - BDF is dynamic and could change if BIOS/OS enumerates a new device in next reboot b. Accepting pair as an input has a downside - What to do when there are multiple such devices and user would like to put only some of them in PT mode c. Not all devices are PCI in the first place. 2. Have a sysfs file which takes 1 or 0 as an input to enable/disable pass through mode. Some places that seem to be reasonable are a. /sys/class/iommu/dmar0/devices/ b. /sys/kernel/iommu_groups//devices Note that this this works out a bit tricky to actually use, since changing the meaning of DMA addresses under the device's feet would be disastrous. It can only safely take effect by unbinding and rebinding the driver and/or resetting the device (as a side note, this starts to overlap with the IOMMU-drivers-as-modules concept, and whatever solution to this we end up with may be helpful in making that work too). In most cases that's probably viable, but not every driver supports unbinding, and either way what if the device in question is the one hosting the root filesystem... :/ Robin. I am looking for a consensus on *how the kernel command line argument should look like and path for sysfs entry*. Also, please note that if a device is put in pass through mode it won't be available for the guest and that's ok. Regards, Sai PS: Idea credits: Ashok Raj ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Device specific pass through in host systems - discuss user interface
Hi All, I am working on an IOMMU driver feature that allows a user to specify if the DMA from a device should be translated by IOMMU or not. Presently, we support only all devices or none mode i.e. if user specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through kernel command line, all the devices would be in pass through mode and we don't have per device granularity, but, we were requested by a customer to selectively put devices in pass through mode and not all. Since, this feature could be generic across architectures, we thought it would be better if the user interface is discussed in the community first. We are envisioning this to be used both during boot time and runtime and hence having a kernel command line argument along with a sysfs entry are needed. So, please pour in your suggestions on how the user interface should look like to make it architecture agnostic. 1. Have a kernel command line argument that takes a list of BDF's as an input and puts them in pass through mode a. Accepting BDF as an input has a downside - BDF is dynamic and could change if BIOS/OS enumerates a new device in next reboot b. Accepting pair as an input has a downside - What to do when there are multiple such devices and user would like to put only some of them in PT mode 2. Have a sysfs file which takes 1 or 0 as an input to enable/disable pass through mode. Some places that seem to be reasonable are a. /sys/class/iommu/dmar0/devices/ b. /sys/kernel/iommu_groups//devices I am looking for a consensus on *how the kernel command line argument should look like and path for sysfs entry*. Also, please note that if a device is put in pass through mode it won't be available for the guest and that's ok. Regards, Sai PS: Idea credits: Ashok Raj ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
