RE: [PATCH v4] Documentation: x86: rework IOMMU documentation
[Public] > -Original Message- > From: Robin Murphy > Sent: Friday, April 22, 2022 3:41 PM > To: Deucher, Alexander ; linux- > d...@vger.kernel.org; linux-ker...@vger.kernel.org; cor...@lwn.net; > h...@zytor.com; x...@kernel.org; dave.han...@linux.intel.com; > b...@alien8.de; mi...@redhat.com; t...@linutronix.de; j...@8bytes.org; > Suthikulpanit, Suravee ; w...@kernel.org; > iommu@lists.linux-foundation.org; Hegde, Vasant > Subject: Re: [PATCH v4] Documentation: x86: rework IOMMU documentation > > On 2022-04-22 18:54, Alex Deucher wrote: > [...] > > +Intel Specific Notes > > + > > + > > +Graphics Problems? > > +^^ > > + > > +If you encounter issues with graphics devices, you can try adding > > +option intel_iommu=igfx_off to turn off the integrated graphics engine. > > +If this fixes anything, please ensure you file a bug reporting the problem. > > + > > +Some exceptions to IOVA > > +^^^ > > + > > +Interrupt ranges are not address translated, (0xfee0 - 0xfeef). > > +The same is true for peer to peer transactions. Hence we reserve the > > +address from PCI MMIO ranges so they are not allocated for IOVA > addresses. > > Note that this should be true for both drivers. > > > + > > +AMD Specific Notes > > +-- > > + > > +Graphics Problems? > > +^^ > > + > > +If you encounter issues with integrated graphics devices, you can try > > +adding option iommu=pt to the kernel command line use a 1:1 mapping > > +for the IOMMU. If this fixes anything, please ensure you file a bug > reporting the problem. > > And indeed this is a generic option. I reckon we could simply merge these two > sections together, with the first paragraph being something like: > > If you encounter issues with integrated graphics devices, you can try adding > the option "iommu.passthrough=1", or the equivalent "iommu=pt", to the > kernel command line to use a 1:1 mapping for the IOMMU in general. On > Intel you can also try "intel_iommu=igfx_off" to turn off translation > specifically > for the integrated graphics engine only. If this fixes anything, please > ensure > you file a bug reporting the problem. > > > + > > +Fault reporting > > +--- > > +When errors are reported, the IOMMU signals via an interrupt. The > > +fault reason and device that caused it is printed on the console. > > + > > + > > +Kernel Log Samples > > +-- > > + > > +Intel Boot Messages > > +^^^ > > + > > +Something like this gets printed indicating presence of DMAR tables > > +in ACPI. > > + > > +:: > > + > > + ACPI: DMAR (v001 A M I OEMDMAR 0x0001 MSFT > 0x0097) @ > > +0x7f5b5ef0 > > + > > +When DMAR is being processed and initialized by ACPI, prints DMAR > > +locations and any RMRR's processed > > + > > +:: > > + > > + ACPI DMAR:Host address width 36 > > + ACPI DMAR:DRHD (flags: 0x)base: 0xfed9 > > + ACPI DMAR:DRHD (flags: 0x)base: 0xfed91000 > > + ACPI DMAR:DRHD (flags: 0x0001)base: 0xfed93000 > > + ACPI DMAR:RMRR base: 0x000ed000 end: > 0x000e > > + ACPI DMAR:RMRR base: 0x7f60 end: > 0x7fff > > + > > +When DMAR is enabled for use, you will notice > > + > > +:: > > + > > + PCI-DMA: Using DMAR IOMMU > > + > > +Intel Fault reporting > > +^ > > + > > +:: > > + > > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000 > > + DMAR:[fault reason 05] PTE Write access is not set > > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000 > > + DMAR:[fault reason 05] PTE Write access is not set > > + > > +AMD Boot Messages > > +^ > > + > > +Something like this gets printed indicating presence of the IOMMU. > > + > > +:: > > + > > + iommu: Default domain type: Translated > > + iommu: DMA domain TLB invalidation policy: lazy mode > > Similarly, that's common IOMMU API reporting which will be seen on all > architectures (let alone IOMMU drivers). Maybe some of the messages from > print_iommu_info() might be better AMD-specific examples? > All good points. I've integrated these suggestions and will send out a new version. Thanks! Alex > Cheers, > Robin. > > > + > > +AMD Fault reporting > > +^^^ > > + > > +:: > > + > > + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 > address=0xc02000 flags=0x] > > + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 > domain=0x0007 > > +address=0xc02000 flags=0x] ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v4] Documentation: x86: rework IOMMU documentation
On 2022-04-22 18:54, Alex Deucher wrote: [...] +Intel Specific Notes + + +Graphics Problems? +^^ + +If you encounter issues with graphics devices, you can try adding +option intel_iommu=igfx_off to turn off the integrated graphics engine. +If this fixes anything, please ensure you file a bug reporting the problem. + +Some exceptions to IOVA +^^^ + +Interrupt ranges are not address translated, (0xfee0 - 0xfeef). +The same is true for peer to peer transactions. Hence we reserve the +address from PCI MMIO ranges so they are not allocated for IOVA addresses. Note that this should be true for both drivers. + +AMD Specific Notes +-- + +Graphics Problems? +^^ + +If you encounter issues with integrated graphics devices, you can try adding +option iommu=pt to the kernel command line use a 1:1 mapping for the IOMMU. If +this fixes anything, please ensure you file a bug reporting the problem. And indeed this is a generic option. I reckon we could simply merge these two sections together, with the first paragraph being something like: If you encounter issues with integrated graphics devices, you can try adding the option "iommu.passthrough=1", or the equivalent "iommu=pt", to the kernel command line to use a 1:1 mapping for the IOMMU in general. On Intel you can also try "intel_iommu=igfx_off" to turn off translation specifically for the integrated graphics engine only. If this fixes anything, please ensure you file a bug reporting the problem. + +Fault reporting +--- +When errors are reported, the IOMMU signals via an interrupt. The fault +reason and device that caused it is printed on the console. + + +Kernel Log Samples +-- + +Intel Boot Messages +^^^ + +Something like this gets printed indicating presence of DMAR tables +in ACPI. + +:: + + ACPI: DMAR (v001 A M I OEMDMAR 0x0001 MSFT 0x0097) @ 0x7f5b5ef0 + +When DMAR is being processed and initialized by ACPI, prints DMAR locations +and any RMRR's processed + +:: + + ACPI DMAR:Host address width 36 + ACPI DMAR:DRHD (flags: 0x)base: 0xfed9 + ACPI DMAR:DRHD (flags: 0x)base: 0xfed91000 + ACPI DMAR:DRHD (flags: 0x0001)base: 0xfed93000 + ACPI DMAR:RMRR base: 0x000ed000 end: 0x000e + ACPI DMAR:RMRR base: 0x7f60 end: 0x7fff + +When DMAR is enabled for use, you will notice + +:: + + PCI-DMA: Using DMAR IOMMU + +Intel Fault reporting +^ + +:: + + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000 + DMAR:[fault reason 05] PTE Write access is not set + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000 + DMAR:[fault reason 05] PTE Write access is not set + +AMD Boot Messages +^ + +Something like this gets printed indicating presence of the IOMMU. + +:: + + iommu: Default domain type: Translated + iommu: DMA domain TLB invalidation policy: lazy mode Similarly, that's common IOMMU API reporting which will be seen on all architectures (let alone IOMMU drivers). Maybe some of the messages from print_iommu_info() might be better AMD-specific examples? Cheers, Robin. + +AMD Fault reporting +^^^ + +:: + + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xc02000 flags=0x] + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xc02000 flags=0x] ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH v4] Documentation: x86: rework IOMMU documentation
[Public] > -Original Message- > From: Deucher, Alexander > Sent: Friday, April 22, 2022 1:54 PM > To: linux-...@vger.kernel.org; linux-ker...@vger.kernel.org; > cor...@lwn.net; h...@zytor.com; x...@kernel.org; > dave.han...@linux.intel.com; b...@alien8.de; mi...@redhat.com; > t...@linutronix.de; j...@8bytes.org; Suthikulpanit, Suravee > ; w...@kernel.org; iommu@lists.linux- > foundation.org; robin.mur...@arm.com; Hegde, Vasant > > Cc: Deucher, Alexander > Subject: [PATCH v4] Documentation: x86: rework IOMMU documentation > > Add preliminary documentation for AMD IOMMU and combine with the > existing Intel IOMMU documentation and clean up and modernize some of the > existing documentation to align with the current state of the kernel. > > Signed-off-by: Alex Deucher > --- > > V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g., > a device) and document proper DMA API. Also correct the fact that > the AMD IOMMU is not limited to managing PCI devices. > v3: Fix spelling and rework text as suggested by Vasant > v4: Combine Intel and AMD documents into a single document as suggested > by Dave Hansen > > Documentation/x86/index.rst | 2 +- > Documentation/x86/intel-iommu.rst | 115 -- > Documentation/x86/iommu.rst | 153 > ++ > 3 files changed, 154 insertions(+), 116 deletions(-) delete mode 100644 > Documentation/x86/intel-iommu.rst create mode 100644 > Documentation/x86/iommu.rst > > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst > index f498f1d36cd3..6f8409fe0674 100644 > --- a/Documentation/x86/index.rst > +++ b/Documentation/x86/index.rst > @@ -21,7 +21,7 @@ x86-specific Documentation > tlb > mtrr > pat > - intel-iommu > + iommu > intel_txt > amd-memory-encryption > pti > diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel- > iommu.rst > deleted file mode 100644 > index 099f13d51d5f.. > --- a/Documentation/x86/intel-iommu.rst > +++ /dev/null > @@ -1,115 +0,0 @@ > -=== > -Linux IOMMU Support > -=== > - > -The architecture spec can be obtained from the below location. > - > -http://www.intel.com/content/dam/www/public/us/en/documents/product- > specifications/vt-directed-io-spec.pdf > - > -This guide gives a quick cheat sheet for some basic understanding. > - > -Some Keywords > - > -- DMAR - DMA remapping > -- DRHD - DMA Remapping Hardware Unit Definition > -- RMRR - Reserved memory Region Reporting Structure > -- ZLR - Zero length reads from PCI devices > -- IOVA - IO Virtual address. > - > -Basic stuff > > - > -ACPI enumerates and lists the different DMA engines in the platform, and - > device scope relationships between PCI devices and which DMA engine > controls -them. > - > -What is RMRR? > -- > - > -There are some devices the BIOS controls, for e.g USB devices to perform > -PS2 emulation. The regions of memory used for these devices are marked - > reserved in the e820 map. When we turn on DMA translation, DMA to those - > regions will fail. Hence BIOS uses RMRR to specify these regions along with - > devices that need to access these regions. OS is expected to setup -unity > mappings for these regions for these devices to access these regions. > - > -How is IOVA generated? > --- > - > -Well behaved drivers call pci_map_*() calls before sending command to > device -that needs to perform DMA. Once DMA is completed and mapping is > no longer -required, device performs a pci_unmap_*() calls to unmap the > region. > - > -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE - > device has its own domain (hence protection). Devices under p2p bridges - > share the virtual address with all devices under the p2p bridge due to - > transaction id aliasing for p2p bridges. > - > -IOVA generation is pretty generic. We used the same technique as vmalloc() - > but these are not global address spaces, but separate for each domain. > -Different DMA engines may support different number of domains. > - > -We also allocate guard pages with each mapping, so we can attempt to catch - > any overflow that might happen. > - > - > -Graphics Problems? > --- > -If you encounter issues with graphics devices, you can try adding -option > intel_iommu=igfx_off to turn off the integrated graphics engine. > -If this fixes anything, please ensure you file a bug reporting the problem. > - > -Some exceptions to IOVA > > -Interrupt ranges a
[PATCH v4] Documentation: x86: rework IOMMU documentation
Add preliminary documentation for AMD IOMMU and combine with the existing Intel IOMMU documentation and clean up and modernize some of the existing documentation to align with the current state of the kernel. Signed-off-by: Alex Deucher --- V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g., a device) and document proper DMA API. Also correct the fact that the AMD IOMMU is not limited to managing PCI devices. v3: Fix spelling and rework text as suggested by Vasant v4: Combine Intel and AMD documents into a single document as suggested by Dave Hansen Documentation/x86/index.rst | 2 +- Documentation/x86/intel-iommu.rst | 115 -- Documentation/x86/iommu.rst | 153 ++ 3 files changed, 154 insertions(+), 116 deletions(-) delete mode 100644 Documentation/x86/intel-iommu.rst create mode 100644 Documentation/x86/iommu.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index f498f1d36cd3..6f8409fe0674 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -21,7 +21,7 @@ x86-specific Documentation tlb mtrr pat - intel-iommu + iommu intel_txt amd-memory-encryption pti diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst deleted file mode 100644 index 099f13d51d5f.. --- a/Documentation/x86/intel-iommu.rst +++ /dev/null @@ -1,115 +0,0 @@ -=== -Linux IOMMU Support -=== - -The architecture spec can be obtained from the below location. - -http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf - -This guide gives a quick cheat sheet for some basic understanding. - -Some Keywords - -- DMAR - DMA remapping -- DRHD - DMA Remapping Hardware Unit Definition -- RMRR - Reserved memory Region Reporting Structure -- ZLR - Zero length reads from PCI devices -- IOVA - IO Virtual address. - -Basic stuff - -ACPI enumerates and lists the different DMA engines in the platform, and -device scope relationships between PCI devices and which DMA engine controls -them. - -What is RMRR? -- - -There are some devices the BIOS controls, for e.g USB devices to perform -PS2 emulation. The regions of memory used for these devices are marked -reserved in the e820 map. When we turn on DMA translation, DMA to those -regions will fail. Hence BIOS uses RMRR to specify these regions along with -devices that need to access these regions. OS is expected to setup -unity mappings for these regions for these devices to access these regions. - -How is IOVA generated? --- - -Well behaved drivers call pci_map_*() calls before sending command to device -that needs to perform DMA. Once DMA is completed and mapping is no longer -required, device performs a pci_unmap_*() calls to unmap the region. - -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE -device has its own domain (hence protection). Devices under p2p bridges -share the virtual address with all devices under the p2p bridge due to -transaction id aliasing for p2p bridges. - -IOVA generation is pretty generic. We used the same technique as vmalloc() -but these are not global address spaces, but separate for each domain. -Different DMA engines may support different number of domains. - -We also allocate guard pages with each mapping, so we can attempt to catch -any overflow that might happen. - - -Graphics Problems? --- -If you encounter issues with graphics devices, you can try adding -option intel_iommu=igfx_off to turn off the integrated graphics engine. -If this fixes anything, please ensure you file a bug reporting the problem. - -Some exceptions to IOVA -Interrupt ranges are not address translated, (0xfee0 - 0xfeef). -The same is true for peer to peer transactions. Hence we reserve the -address from PCI MMIO ranges so they are not allocated for IOVA addresses. - - -Fault reporting -When errors are reported, the DMA engine signals via an interrupt. The fault -reason and device that caused it with fault reason is printed on console. - -See below for sample. - - -Boot Message Sample - -Something like this gets printed indicating presence of DMAR tables -in ACPI. - -ACPI: DMAR (v001 A M I OEMDMAR 0x0001 MSFT 0x0097) @ 0x7f5b5ef0 - -When DMAR is being processed and initialized by ACPI, prints DMAR locations -and any RMRR's processed:: - - ACPI DMAR:Host address width 36 - ACPI DMAR:DRHD (flags: 0x)base: 0xfed9 - ACPI DMAR:DRHD (flags: 0x)base: 0xfed91000 - ACPI DMAR:DRHD (flags: 0x0001)base: 0xfed93000 - ACPI DMAR:RMRR base: 0x000ed000 end: 0x000e - ACPI DMAR:RMRR base: 0x7f60 end: 0x7fff - -When DMAR is
