RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings
-Original Message- From: Sethi Varun-B16395 Sent: Tuesday, June 17, 2014 5:27 AM To: Yoder Stuart-B08248; Will Deacon Cc: Thierry Reding; Mark Rutland; devicet...@vger.kernel.org; linux- samsung-...@vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux- te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- ker...@lists.infradead.org Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings -Original Message- From: Yoder Stuart-B08248 Sent: Tuesday, June 17, 2014 12:24 AM To: Will Deacon Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicet...@vger.kernel.org; linux-samsung-soc@vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux- arm- ker...@lists.infradead.org Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings -Original Message- From: Will Deacon [mailto:will.dea...@arm.com] Sent: Monday, June 16, 2014 12:04 PM To: Yoder Stuart-B08248 Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicet...@vger.kernel.org; linux-samsung-soc@vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- ker...@lists.infradead.org Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings Hi Stuart, On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote: Do you have use-cases where you really need to change these mappings dynamically? Yes. In the case of a PCI bus-- you may not know in advance how many PCI devices there are until you probe the bus. We have another FSL proprietary bus we call the fsl-mc bus that is similar. For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping. Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling multiple TBUs around. However, this means that the stream ID namespace is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many more masters than that. So we look at stream IDs as really corresponding to an 'isolation context' and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables. Ok, this is more compelling. So, perhaps by default some/most SMMU masters may have a default stream ID of 0x0 that is used by the host...and that could be represented statically in the device tree. But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices are added to it. All the devices in a domain will share the same stream ID. So whatever we do, let's please have an architecture flexible enough to allow for this. What is the software interface to the logic that assigns the StreamIDs? Is it part of the SMMU, or a separate device (or set of devices)? For us at the hardware level there are a few different ways that the streamIDs can be set. It is not part of the SMMU. In the cases where there is simply 1 device to 1 streamID (e.g. USB controller) there is an SoC register where you just program the stream ID. In the case of PCI, our PCI controller has a RequesterID-to-streamID table that you set via some PCI controller registers. The way we generally thought it would work was something like this: -u-boot/bootloader makes any static streamID allocation if needed, sets a default streamID (e.g. 0x0) in device and expresses that in the device tree -device tree would express relationship between devices (including bus controllers) and the SMMU through mmu-masters property -u-boot would express the range of unused (or used) streamIDs via a new device tree property so the kernel SMMU driver knows what streamIDs are free -in the SMMU driver a different vendor specific 'add_device' callback could
RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings
-Original Message- From: Sethi Varun-B16395 Sent: Tuesday, June 17, 2014 6:22 AM To: Will Deacon Cc: Mark Rutland; devicet...@vger.kernel.org; linux-samsung- s...@vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux- ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar Gala; linux-te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- ker...@lists.infradead.org Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings -Original Message- From: iommu-boun...@lists.linux-foundation.org [mailto:iommu- boun...@lists.linux-foundation.org] On Behalf Of Will Deacon Sent: Tuesday, June 17, 2014 4:13 PM To: Sethi Varun-B16395 Cc: Mark Rutland; devicet...@vger.kernel.org; linux-samsung- s...@vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux- ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar Gala; linux-te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux- arm- ker...@lists.infradead.org Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote: The way we generally thought it would work was something like this: -u-boot/bootloader makes any static streamID allocation if needed, sets a default streamID (e.g. 0x0) in device and expresses that in the device tree -device tree would express relationship between devices (including bus controllers) and the SMMU through mmu-masters property -u-boot would express the range of unused (or used) streamIDs via a new device tree property so the kernel SMMU driver knows what streamIDs are free -in the SMMU driver a different vendor specific 'add_device' callback could be used to handle our special cases where we need to set/change the stream ID for devices added to a domain Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID. I think for simple masters (i.e. those that have all their StreamIDs under control of one driver), then setting something during attach (or add?) based on the DT could work pretty well. The other case is when we have masters behind a bridge (such as a PCI RC). In this case, it might actually be better to ask the bridge for the IDs and let it sort out the allocation itself. That would also move the RequesterID - StreamID mapping out of the SMMU code. What do you think? The PCI bus iommu group creation code would be part of the SMMU driver (it is handled by other IOMMU drivers as well). My understanding is that there would be one is to one correspondence between the requestor ID and the iommu group. May be, we can have an API provided by the PCI bridge (architecture specific) for setting the stream ID. I think Will is suggesting something along those lines-- I think it's a question of where the streamID allocation happens. You could either do something like the following in the SMMU driver when attaching a PCI device: id = alloc_stream_id(...); pci_set_streamid(pci-dev, id); or id = pci_get_streamid(pci-dev); ...i.e the PCI RC could allocate (from some TBD allocator) and set the stream ID itself. Not sure how big a deal it is to extend PCI RC interfaces for something like that. Stuart -- To unsubscribe from this list: send the line unsubscribe linux-samsung-soc in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings
-Original Message- From: Will Deacon [mailto:will.dea...@arm.com] Sent: Monday, June 16, 2014 10:28 AM To: Sethi Varun-B16395 Cc: Thierry Reding; Mark Rutland; devicet...@vger.kernel.org; linux- samsung-...@vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux- te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- ker...@lists.infradead.org; Yoder Stuart-B08248 Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings Hi Varun, On Thu, Jun 05, 2014 at 08:10:19PM +0100, Varun Sethi wrote: The set of StreamIDs that can be generated by a master is fixed in the hardware. The SMMU can then be programmed to map these incoming IDs onto a context ID (or a set of context IDs), which are the IDs used internally by the SMMU to find the page tables etc. The StreamID - ContextID mapping is dynamic and controlled by software. The Master - StreamIDs mapping is fixed in the hardware. The Master - StreamIDs mapping may not always be fixed in the hardware. At, least in our case we plan to program these via software. PCI devices is one place where this mapping would have to be dynamic (based on the topology i.e. if the devices are connected to a bridge etc). Also, we have a hot plug device architecture where the stream ID is software programmable. Other than that, based on the isolation requirements (iommu domain) software programmability offers greater flexibility. For the sake of consistency, I'd really like to treat the master - streamIDs mapping as fixed. If that means setting it once during boot in your case, then so be it (ideally in the firmware). That way, we just describe the fixed mapping to the kernel and the driver doesn't have to worry about changing the mapping, especially given that that's likely to be highly error-prone once the SMMU is in us. Do you have use-cases where you really need to change these mappings dynamically? Yes. In the case of a PCI bus-- you may not know in advance how many PCI devices there are until you probe the bus. We have another FSL proprietary bus we call the fsl-mc bus that is similar. Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling multiple TBUs around. However, this means that the stream ID namespace is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many more masters than that. So we look at stream IDs as really corresponding to an 'isolation context' and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables. So, perhaps by default some/most SMMU masters may have a default stream ID of 0x0 that is used by the host...and that could be represented statically in the device tree. But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices are added to it. All the devices in a domain will share the same stream ID. So whatever we do, let's please have an architecture flexible enough to allow for this. Thanks, Stuart -- To unsubscribe from this list: send the line unsubscribe linux-samsung-soc in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings
-Original Message- From: Will Deacon [mailto:will.dea...@arm.com] Sent: Monday, June 16, 2014 12:04 PM To: Yoder Stuart-B08248 Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicet...@vger.kernel.org; linux-samsung-soc@vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- ker...@vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-te...@vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- ker...@lists.infradead.org Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings Hi Stuart, On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote: Do you have use-cases where you really need to change these mappings dynamically? Yes. In the case of a PCI bus-- you may not know in advance how many PCI devices there are until you probe the bus. We have another FSL proprietary bus we call the fsl-mc bus that is similar. For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping. Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling multiple TBUs around. However, this means that the stream ID namespace is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many more masters than that. So we look at stream IDs as really corresponding to an 'isolation context' and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables. Ok, this is more compelling. So, perhaps by default some/most SMMU masters may have a default stream ID of 0x0 that is used by the host...and that could be represented statically in the device tree. But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices are added to it. All the devices in a domain will share the same stream ID. So whatever we do, let's please have an architecture flexible enough to allow for this. What is the software interface to the logic that assigns the StreamIDs? Is it part of the SMMU, or a separate device (or set of devices)? For us at the hardware level there are a few different ways that the streamIDs can be set. It is not part of the SMMU. In the cases where there is simply 1 device to 1 streamID (e.g. USB controller) there is an SoC register where you just program the stream ID. In the case of PCI, our PCI controller has a RequesterID-to-streamID table that you set via some PCI controller registers. The way we generally thought it would work was something like this: -u-boot/bootloader makes any static streamID allocation if needed, sets a default streamID (e.g. 0x0) in device and expresses that in the device tree -device tree would express relationship between devices (including bus controllers) and the SMMU through mmu-masters property -u-boot would express the range of unused (or used) streamIDs via a new device tree property so the kernel SMMU driver knows what streamIDs are free -in the SMMU driver a different vendor specific 'add_device' callback could be used to handle our special cases where we need to set/change the stream ID for devices added to a domain Thanks, Stuart -- To unsubscribe from this list: send the line unsubscribe linux-samsung-soc in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
