Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
Hi Will, Michael, On 12/07/2017 19:54, Will Deacon wrote: > Hi Eric, > > On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote: >> This series adds a new tlbi-on-map option to the smmuv3 driver. >> When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for >> LPAE tables and the smmuv3 driver sends TLB invalidations on map. >> >> This mode is useful when running the driver on a guest as it allows >> the virtualizer to trap any change to the translation structures. >> This is similar to the Intel vtd caching mode (CM). >> >> This is especially needed for VFIO integration integration where >> guest mappings must be applied to the physical IOMMU. >> >> At the moment the option only is available for DT probing. > > I'm really not a fan of this approach. If a virtual IOMMU implementation is > advertising itself as an SMMUv3, then it should adhere to the SMMUv3 > architecture and not require non-standard behaviour from the driver. If > we're going to allow that, then we're better off going the extra mile and > using a PV approach. Given that the the SMMU3 architecture does *not* > require TLBI on map, then I don't think we should be quirking our behaviour > in this way. The fact that you only have this implemented for DT is the > canary in the coal mine imo. I did not proceed with ACPI integration as I focused on the QEMU integration POC instead. I intended to propose a new model id just like 12275bf iommu/arm-smmu-v3, acpi: Add temporary Cavium SMMU-V3 IORT model number definitions? Is it really costly to maintain such a quirk (which by the way does the same thing as Intel caching mode, unfortunately not in an HW specified manner) compared to other quirks? Having an smmu emulated device integrated with vfio can always be useful for debugging purpose and also to compare perfs/functionality against virtio-iommu solution. We said both solutions could co-exist, serving different purposes. Now if you close the door to vfio integration, emulated smmu just serves the purpose of guests using isolated virtio devices which limits quite a lot its usage, somehow questioning the efforts invested to develop/upstream it. Besides, before any further decision, I need to complete the DPDK use case enablement as I get a storm of page tlbi's although hugepages are used. And this hasn't worked yet ... Thanks Eric > > Will >
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
On Thu, Jul 13, 2017 at 08:17:49PM +0100, Jean-Philippe Brucker wrote: > But it is now mostly obsolete. Lots of things had to change while writing > a prototype and following public discussions. At the moment my focus is on > tidying up the base, but I will send another RFC afterwards. Thanks! Pls remember to copy virtio-dev at oasis (you need to subscribe to post) as that's a requirement for anything involving host/guest APIs. -- MST
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
On 13/07/17 18:44, Michael S. Tsirkin wrote: > On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote: >> On 12/07/17 23:07, Michael S. Tsirkin wrote: >> [...] >>> I think using hardware support for nesting is the right final >>> solution. It will take some time though. Given this, what should >>> we do meanwhile? >>> >>> Assuming that's the final destination, a simple quirk like this >>> seems to be preferable to virtio-iommu which we'll never be >>> able to offload to hardware. >> >> That's not entirely true. virtio-iommu will have an extension for hardware >> nesting support. It was presented in my initial proposal, and I've made >> significant progress since then. >> >> Thanks, >> Jean > > I don't recall seeing this. > > Hardware specific extensions to virtio would be interesting, the > difficulty is in finding the balance between enabling minor quirks and > each vendor going their own way. Yes, in order to avoid too many quirks and vendor-specific formats, we'd like the guest to only manage page tables, which have relatively stable format, while the host kernel manages context tables (PASID tables) and other structures, that are more volatile across implementations. Unavoidably, a few architecture-specific details need to be described in the API, but it seems manageable. The host tells the guest which page table format is supported by hardware, and the guest sends back a page directory pointer along with a few configuration parameters. In the guest, page table operations may be abstracted in a module and used by multiple IOMMU drivers (what ARM does in drivers/iommu/io-pgtable-arm.c for the various vendor drivers). This abstraction is quite helpful to find out which information needs to be exchanged between virtio-iommu device and driver. For ARM-based IOMMUs it amounts to 6 configuration registers and 4 quirk bits. The other problem is forwarding TLB invalidations to the host kernel. There is an ongoing discussion [1] about the best way to do it in VFIO, and it seems like the format can stay mostly generic, with a few optimization hints depending on the hardware IOMMU. > Is this the proposal you refer to? > https://www.spinics.net/lists/kvm/msg147990.html Yes, I'm referring to "II. Page table sharing" in https://www.spinics.net/lists/kvm/msg147993.html But it is now mostly obsolete. Lots of things had to change while writing a prototype and following public discussions. At the moment my focus is on tidying up the base, but I will send another RFC afterwards. Thanks, Jean [1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg01117.html
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote: > On 12/07/17 23:07, Michael S. Tsirkin wrote: > [...] > > I think using hardware support for nesting is the right final > > solution. It will take some time though. Given this, what should > > we do meanwhile? > > > > Assuming that's the final destination, a simple quirk like this > > seems to be preferable to virtio-iommu which we'll never be > > able to offload to hardware. > > That's not entirely true. virtio-iommu will have an extension for hardware > nesting support. It was presented in my initial proposal, and I've made > significant progress since then. > > Thanks, > Jean I don't recall seeing this. Hardware specific extensions to virtio would be interesting, the difficulty is in finding the balance between enabling minor quirks and each vendor going their own way. Is this the proposal you refer to? https://www.spinics.net/lists/kvm/msg147990.html I couldn't find any mention of nesting, it seems to say that map requests are relayed through host. -- MST
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
On 12/07/17 23:07, Michael S. Tsirkin wrote: [...] > I think using hardware support for nesting is the right final > solution. It will take some time though. Given this, what should > we do meanwhile? > > Assuming that's the final destination, a simple quirk like this > seems to be preferable to virtio-iommu which we'll never be > able to offload to hardware. That's not entirely true. virtio-iommu will have an extension for hardware nesting support. It was presented in my initial proposal, and I've made significant progress since then. Thanks, Jean
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
On Wed, Jul 12, 2017 at 06:54:56PM +0100, Will Deacon wrote: > Hi Eric, > > On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote: > > This series adds a new tlbi-on-map option to the smmuv3 driver. > > When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for > > LPAE tables and the smmuv3 driver sends TLB invalidations on map. > > > > This mode is useful when running the driver on a guest as it allows > > the virtualizer to trap any change to the translation structures. > > This is similar to the Intel vtd caching mode (CM). > > > > This is especially needed for VFIO integration integration where > > guest mappings must be applied to the physical IOMMU. > > > > At the moment the option only is available for DT probing. > > I'm really not a fan of this approach. If a virtual IOMMU implementation is > advertising itself as an SMMUv3, then it should adhere to the SMMUv3 > architecture and not require non-standard behaviour from the driver. You can also just enable e.g. building existing intel VTD code on ARM, that emulation is already there. > If > we're going to allow that, then we're better off going the extra mile and > using a PV approach. Given that the the SMMU3 architecture does *not* > require TLBI on map, then I don't think we should be quirking our behaviour > in this way. I think using hardware support for nesting is the right final solution. It will take some time though. Given this, what should we do meanwhile? Assuming that's the final destination, a simple quirk like this seems to be preferable to virtio-iommu which we'll never be able to offload to hardware. >The fact that you only have this implemented for DT is the > canary in the coal mine imo. > > Will As to the last point, I agree absolutely. Need to support ACPI as well or nothing. It would be nicer IMHO to get it from some kind of device register, assuming ARM can promise to reserve some bits for hypervisors to play with. -- MST
Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option
Hi Eric, On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote: > This series adds a new tlbi-on-map option to the smmuv3 driver. > When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for > LPAE tables and the smmuv3 driver sends TLB invalidations on map. > > This mode is useful when running the driver on a guest as it allows > the virtualizer to trap any change to the translation structures. > This is similar to the Intel vtd caching mode (CM). > > This is especially needed for VFIO integration integration where > guest mappings must be applied to the physical IOMMU. > > At the moment the option only is available for DT probing. I'm really not a fan of this approach. If a virtual IOMMU implementation is advertising itself as an SMMUv3, then it should adhere to the SMMUv3 architecture and not require non-standard behaviour from the driver. If we're going to allow that, then we're better off going the extra mile and using a PV approach. Given that the the SMMU3 architecture does *not* require TLBI on map, then I don't think we should be quirking our behaviour in this way. The fact that you only have this implemented for DT is the canary in the coal mine imo. Will
[RFC 0/2] arm-smmu-v3 tlbi-on-map option
This series adds a new tlbi-on-map option to the smmuv3 driver. When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for LPAE tables and the smmuv3 driver sends TLB invalidations on map. This mode is useful when running the driver on a guest as it allows the virtualizer to trap any change to the translation structures. This is similar to the Intel vtd caching mode (CM). This is especially needed for VFIO integration integration where guest mappings must be applied to the physical IOMMU. At the moment the option only is available for DT probing. Best Regards Eric Eric Auger (2): iommu/io-pgtable-arm: flush TLBs when IO_PGTABLE_QUIRK_TLBI_ON_MAP arm-smmu-v3: Add tlbi_on_map option Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt | 4 drivers/iommu/arm-smmu-v3.c | 5 + drivers/iommu/io-pgtable-arm.c | 13 +++-- 3 files changed, 20 insertions(+), 2 deletions(-) -- 2.5.5
