IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
> > > > On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but > > > > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from > > > > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has > > > > booted successful so far in 2 tries, one with rc6 and one with rc7. > > > > > > More testing shows it can NMI on acpi_idle too, not just intel_idle. > > > > > > > Screenshot with some backtrace: > > > > http://kodu.ut.ee/~mroos/intel-idle-NMI.png > > > > > > http://kodu.ut.ee/~mroos/acpi-idle-NMI.png > > > > There were almost no changes in the ACPI idle driver between 4.6 and > > 4.7-rc, so the reason is somewhere else. > > > > Have you tried any earlier 4.7-rc? > > I tried selfcompiled 4.6.0 now with the same conf that 4.7-rc's have and > after multiple tries I got the same NMI out of 4.6 too, from kernfs > lookup (that seems to just a random victim). > > So this is not a 4.7 regression and probably not idle-releated. > > Seems my kernel config is causing NMI-s for some reason. I was suprised > to see the kernel recommend to see HP ILO logs on HP but there was > nothing in the logs. Bisecting kernel configs shows that it's DMAR+IOMMU. When it is activated, there is high probability of NMI-s in random places. So it is no longer relevant to linux-pm and intel-idle, dropped CC-s and added iommu related mails. Full dmesg from successful boot with IOMMU enabled is below. > # > # DMA Devices > # > CONFIG_DMA_ENGINE=y > CONFIG_DMA_VIRTUAL_CHANNELS=y > CONFIG_DMA_ACPI=y > CONFIG_INTEL_IDMA64=m > CONFIG_INTEL_IOATDMA=m > # CONFIG_QCOM_HIDMA_MGMT is not set > # CONFIG_QCOM_HIDMA is not set > # CONFIG_DW_DMAC is not set > # CONFIG_DW_DMAC_PCI is not set > CONFIG_HSU_DMA=y [...] > # > # Generic IOMMU Pagetable Support > # > CONFIG_IOMMU_IOVA=y > # CONFIG_AMD_IOMMU is not set > CONFIG_DMAR_TABLE=y > CONFIG_INTEL_IOMMU=y > CONFIG_INTEL_IOMMU_SVM=y > CONFIG_INTEL_IOMMU_DEFAULT_ON=y CONFIG_INTEL_IOMMU_DEFAULT_ON=y is the setting that switches the instability on and off. > CONFIG_INTEL_IOMMU_FLOPPY_WA=y > CONFIG_IRQ_REMAP=y [0.00] Linux version 4.6.0 (mroos@dl360g6) (gcc version 5.4.0 20160609 (Debian 5.4.0-6) ) #9 SMP Wed Jul 13 01:00:26 EEST 2016 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.6.0 root=/dev/sda1 ro [0.00] x86/fpu: Legacy x87 FPU detected. [0.00] x86/fpu: Using 'eager' FPU context switches. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009f3ff] usable [0.00] BIOS-e820: [mem 0x0009f400-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0xdf61efff] usable [0.00] BIOS-e820: [mem 0xdf61f000-0xdf62bfff] ACPI data [0.00] BIOS-e820: [mem 0xdf62c000-0xdf62cfff] usable [0.00] BIOS-e820: [mem 0xdf62d000-0xe3ff] reserved [0.00] BIOS-e820: [mem 0xfec0-0xfee0] reserved [0.00] BIOS-e820: [mem 0xff80-0x] reserved [0.00] BIOS-e820: [mem 0x0001-0x00061fffefff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 2.7 present. [0.00] DMI: HP ProLiant DL360 G6, BIOS P64 01/22/2015 [0.00] e820: update [mem 0x-0x0fff] usable ==> reserved [0.00] e820: remove [mem 0x000a-0x000f] usable [0.00] e820: last_pfn = 0x61 max_arch_pfn = 0x4 [0.00] MTRR default type: write-back [0.00] MTRR fixed ranges enabled: [0.00] 0-9 write-back [0.00] A-B uncachable [0.00] C-F write-protect [0.00] MTRR variable ranges enabled: [0.00] 0 base 00E000 mask FFE000 uncachable [0.00] 1 disabled [0.00] 2 disabled [0.00] 3 disabled [0.00] 4 disabled [0.00] 5 disabled [0.00] 6 disabled [0.00] 7 disabled [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [0.00] e820: last_pfn = 0xdf62d max_arch_pfn = 0x4 [0.00] found SMP MP-table at [mem 0x000f4f80-0x000f4f8f] mapped at [880f4f80] [0.00] Base memory trampoline at [88099000] 99000 size 24576 [0.00] BRK [0x01d35000, 0x01d35fff] PGTABLE [0.00] BRK [0x01d36000, 0x01d36fff] PGTABLE [0.00] BRK [0x01d37000, 0x01d37fff] PGTABLE [0.00] BRK [0x01d38000, 0x01d38fff] PGTABLE [0.00] BRK [0x01d39000, 0x01d39fff] PGTABLE [0.00] BRK [0x01d3a000, 0x01d3afff] PGTABLE [0.00] RAMDISK: [mem 0x21572000-0x2cab0fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000F4F00 24 (v02 HP) [0.00] ACPI: XSDT 0xDF620140 B4 (v01 HP
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, Jul 13, 2016 at 10:17:59AM +0300, Meelis Roos wrote: > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > activated, there is high probability of NMI-s in random places. Hmm, strange. But nothing could really surprise when you have an HP BIOS. Can you probably use the faulty config and bisect this down to a specific commit? In v4.7-rc1 some changes to the iova-allocation code got merged, but I have no idea how those could cause NMIs. Thanks, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
> > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > activated, there is high probability of NMI-s in random places. > > Hmm, strange. But nothing could really surprise when you have an HP > BIOS. BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but the release notes only describe updated CPU microcode for security reasons. > Can you probably use the faulty config and bisect this down to a > specific commit? In v4.7-rc1 some changes to the iova-allocation code > got merged, but I have no idea how those could cause NMIs. Will try but I do not know a working base yet - this was broken in both 4.6 and 4.7-rc. -- Meelis Roos (mr...@linux.ee) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH v6 00/46] dma-mapping: Use unsigned long for dma_attrs
Hi, The fifth version of this patchset was merged by Andrew Morton few days ago. It was rebased on v4.7-rc5 so it missed some ongoing changes. This is just rebase on next-20160713. For easier testing the patchset is available here: repo: https://github.com/krzk/linux branch: for-next/dma-attrs-const-v6 Changes since v5 1. New patches: 1/46: [media] mtk-vcodec: Remove unused dma_attrs 44/46: remoteproc: qcom: Use unsigned long for dma_attrs 2. 19/46: rebased on next, some more changes inside 3. Added accumulated acks: Marek Szyprowski, Richard Kuo, Konrad Rzeszutek Wilk, Daniel Vetter and Joerg Roedel. Changes since v4 1. Collect some acks. Still need more. 2. Minor fixes pointed by Robin Murphy. 3. Applied changes from Bart Van Assche's comment. 4. More tests and builds (using https://www.kernel.org/pub/tools/crosstool/). Changes since v3 1. Collect some acks. 2. Drop wrong patch 1/45 ("powerpc: dma-mapping: Don't hard-code the value of DMA_ATTR_WEAK_ORDERING"). 3. Minor fix pointed out by Michael Ellerman. Changes since v2 1. Follow Christoph Hellwig's comments (don't use BIT add documentation, remove dma_get_attr). Rationale = The dma-mapping core and the implementations do not change the DMA attributes passed by pointer. Thus the pointer can point to const data. However the attributes do not have to be a bitfield. Instead unsigned long will do fine: 1. This is just simpler. Both in terms of reading the code and setting attributes. Instead of initializing local attributes on the stack and passing pointer to it to dma_set_attr(), just set the bits. 2. It brings safeness and checking for const correctness because the attributes are passed by value. Best regards, Krzysztof Krzysztof Kozlowski (46): [media] mtk-vcodec: Remove unused dma_attrs dma-mapping: Use unsigned long for dma_attrs alpha: dma-mapping: Use unsigned long for dma_attrs arc: dma-mapping: Use unsigned long for dma_attrs ARM: dma-mapping: Use unsigned long for dma_attrs arm64: dma-mapping: Use unsigned long for dma_attrs avr32: dma-mapping: Use unsigned long for dma_attrs blackfin: dma-mapping: Use unsigned long for dma_attrs c6x: dma-mapping: Use unsigned long for dma_attrs cris: dma-mapping: Use unsigned long for dma_attrs frv: dma-mapping: Use unsigned long for dma_attrs drm/exynos: dma-mapping: Use unsigned long for dma_attrs drm/mediatek: dma-mapping: Use unsigned long for dma_attrs drm/msm: dma-mapping: Use unsigned long for dma_attrs drm/nouveau: dma-mapping: Use unsigned long for dma_attrs drm/rockship: dma-mapping: Use unsigned long for dma_attrs infiniband: dma-mapping: Use unsigned long for dma_attrs iommu: dma-mapping: Use unsigned long for dma_attrs [media] dma-mapping: Use unsigned long for dma_attrs xen: dma-mapping: Use unsigned long for dma_attrs swiotlb: dma-mapping: Use unsigned long for dma_attrs powerpc: dma-mapping: Use unsigned long for dma_attrs video: dma-mapping: Use unsigned long for dma_attrs x86: dma-mapping: Use unsigned long for dma_attrs iommu: intel: dma-mapping: Use unsigned long for dma_attrs h8300: dma-mapping: Use unsigned long for dma_attrs hexagon: dma-mapping: Use unsigned long for dma_attrs ia64: dma-mapping: Use unsigned long for dma_attrs m68k: dma-mapping: Use unsigned long for dma_attrs metag: dma-mapping: Use unsigned long for dma_attrs microblaze: dma-mapping: Use unsigned long for dma_attrs mips: dma-mapping: Use unsigned long for dma_attrs mn10300: dma-mapping: Use unsigned long for dma_attrs nios2: dma-mapping: Use unsigned long for dma_attrs openrisc: dma-mapping: Use unsigned long for dma_attrs parisc: dma-mapping: Use unsigned long for dma_attrs misc: mic: dma-mapping: Use unsigned long for dma_attrs s390: dma-mapping: Use unsigned long for dma_attrs sh: dma-mapping: Use unsigned long for dma_attrs sparc: dma-mapping: Use unsigned long for dma_attrs tile: dma-mapping: Use unsigned long for dma_attrs unicore32: dma-mapping: Use unsigned long for dma_attrs xtensa: dma-mapping: Use unsigned long for dma_attrs remoteproc: qcom: Use unsigned long for dma_attrs dma-mapping: Remove dma_get_attr dma-mapping: Document the DMA attributes next to the declaration Documentation/DMA-API.txt | 33 +++--- Documentation/DMA-attributes.txt | 2 +- arch/alpha/include/asm/dma-mapping.h | 2 - arch/alpha/kernel/pci-noop.c | 2 +- arch/alpha/kernel/pci_iommu.c | 12 +- arch/arc/mm/dma.c | 12 +- arch/arm/common/dmabounce.c| 4 +- arch/arm/include/asm/dma-mapping.h | 13 +-- arch/arm/include/asm/xen/page-coherent.h | 16 +-- arch/arm/mm/dma-mapping.c
[PATCH v6 18/46] iommu: dma-mapping: Use unsigned long for dma_attrs
Split out subsystem specific changes for easier reviews. This will be squashed with main commit. Signed-off-by: Krzysztof Kozlowski [for iommu] Acked-by: Joerg Roedel --- drivers/iommu/amd_iommu.c | 12 ++-- drivers/iommu/dma-iommu.c | 6 +++--- include/linux/dma-iommu.h | 6 +++--- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 92e65c8c..b225e82a5228 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2717,7 +2717,7 @@ static void __unmap_single(struct dma_ops_domain *dma_dom, static dma_addr_t map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, - struct dma_attrs *attrs) + unsigned long attrs) { phys_addr_t paddr = page_to_phys(page) + offset; struct protection_domain *domain; @@ -2739,7 +2739,7 @@ static dma_addr_t map_page(struct device *dev, struct page *page, * The exported unmap_single function for dma_ops. */ static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, struct dma_attrs *attrs) + enum dma_data_direction dir, unsigned long attrs) { struct protection_domain *domain; @@ -2756,7 +2756,7 @@ static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, */ static int map_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction dir, - struct dma_attrs *attrs) + unsigned long attrs) { struct protection_domain *domain; int i; @@ -2804,7 +2804,7 @@ unmap: */ static void unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction dir, -struct dma_attrs *attrs) +unsigned long attrs) { struct protection_domain *domain; struct scatterlist *s; @@ -2826,7 +2826,7 @@ static void unmap_sg(struct device *dev, struct scatterlist *sglist, */ static void *alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_addr, gfp_t flag, - struct dma_attrs *attrs) + unsigned long attrs) { u64 dma_mask = dev->coherent_dma_mask; struct protection_domain *domain; @@ -2880,7 +2880,7 @@ out_free: */ static void free_coherent(struct device *dev, size_t size, void *virt_addr, dma_addr_t dma_addr, - struct dma_attrs *attrs) + unsigned long attrs) { struct protection_domain *domain; struct page *page; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ea5a9ebf0f78..6c1bda504fb1 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -286,7 +286,7 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size, *or NULL on failure. */ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - struct dma_attrs *attrs, int prot, dma_addr_t *handle, + unsigned long attrs, int prot, dma_addr_t *handle, void (*flush_page)(struct device *, const void *, phys_addr_t)) { struct iommu_domain *domain = iommu_get_domain_for_dev(dev); @@ -400,7 +400,7 @@ dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, } void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size, - enum dma_data_direction dir, struct dma_attrs *attrs) + enum dma_data_direction dir, unsigned long attrs) { __iommu_dma_unmap(iommu_get_domain_for_dev(dev), handle); } @@ -560,7 +560,7 @@ out_restore_sg: } void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir, struct dma_attrs *attrs) + enum dma_data_direction dir, unsigned long attrs) { /* * The scatterlist segments are mapped into a single diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h index 8443bbb5c071..81c5c8d167ad 100644 --- a/include/linux/dma-iommu.h +++ b/include/linux/dma-iommu.h @@ -39,7 +39,7 @@ int dma_direction_to_prot(enum dma_data_direction dir, bool coherent); * the arch code to take care of attributes and cache maintenance */ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - struct dma_attrs *attrs, int prot, dma_addr_t *handle, + unsigned long attrs, int prot, dma_addr_t *handle, void (*flush_page)(struct device *, const void *, phys_addr_t)); void iommu_dma_free(struct device *dev, struct page **pages, size_t size, dma_addr_t *handle); @@ -56,9 +56,9 @@ in
[PATCH v6 25/46] iommu: intel: dma-mapping: Use unsigned long for dma_attrs
Split out subsystem specific changes for easier reviews. This will be squashed with main commit. Signed-off-by: Krzysztof Kozlowski [for iommu] Acked-by: Joerg Roedel --- drivers/iommu/intel-iommu.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 69ea66dcc534..3431684ad848 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -3552,7 +3552,7 @@ error: static dma_addr_t intel_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, -struct dma_attrs *attrs) +unsigned long attrs) { return __intel_map_single(dev, page_to_phys(page) + offset, size, dir, *dev->dma_mask); @@ -3711,14 +3711,14 @@ static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size) static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr, size_t size, enum dma_data_direction dir, -struct dma_attrs *attrs) +unsigned long attrs) { intel_unmap(dev, dev_addr, size); } static void *intel_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flags, - struct dma_attrs *attrs) + unsigned long attrs) { struct page *page = NULL; int order; @@ -3764,7 +3764,7 @@ static void *intel_alloc_coherent(struct device *dev, size_t size, } static void intel_free_coherent(struct device *dev, size_t size, void *vaddr, - dma_addr_t dma_handle, struct dma_attrs *attrs) + dma_addr_t dma_handle, unsigned long attrs) { int order; struct page *page = virt_to_page(vaddr); @@ -3779,7 +3779,7 @@ static void intel_free_coherent(struct device *dev, size_t size, void *vaddr, static void intel_unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction dir, - struct dma_attrs *attrs) + unsigned long attrs) { dma_addr_t startaddr = sg_dma_address(sglist) & PAGE_MASK; unsigned long nrpages = 0; @@ -3808,7 +3808,7 @@ static int intel_nontranslate_map_sg(struct device *hddev, } static int intel_map_sg(struct device *dev, struct scatterlist *sglist, int nelems, - enum dma_data_direction dir, struct dma_attrs *attrs) + enum dma_data_direction dir, unsigned long attrs) { int i; struct dmar_domain *domain; -- 1.9.1 ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH v6 24/46] x86: dma-mapping: Use unsigned long for dma_attrs
Split out subsystem specific changes for easier reviews. This will be squashed with main commit. Signed-off-by: Krzysztof Kozlowski --- arch/x86/include/asm/dma-mapping.h | 5 ++--- arch/x86/include/asm/swiotlb.h | 4 ++-- arch/x86/include/asm/xen/page-coherent.h | 9 - arch/x86/kernel/amd_gart_64.c| 20 ++-- arch/x86/kernel/pci-calgary_64.c | 14 +++--- arch/x86/kernel/pci-dma.c| 4 ++-- arch/x86/kernel/pci-nommu.c | 4 ++-- arch/x86/kernel/pci-swiotlb.c| 4 ++-- arch/x86/pci/sta2x11-fixup.c | 2 +- arch/x86/pci/vmd.c | 16 10 files changed, 40 insertions(+), 42 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 3a27b93e6261..44461626830e 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -9,7 +9,6 @@ #include #include #include -#include #include #include #include @@ -48,11 +47,11 @@ extern int dma_supported(struct device *hwdev, u64 mask); extern void *dma_generic_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_addr, gfp_t flag, - struct dma_attrs *attrs); + unsigned long attrs); extern void dma_generic_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_addr, - struct dma_attrs *attrs); + unsigned long attrs); #ifdef CONFIG_X86_DMA_REMAP /* Platform code defines bridge-specific code */ extern bool dma_capable(struct device *dev, dma_addr_t addr, size_t size); diff --git a/arch/x86/include/asm/swiotlb.h b/arch/x86/include/asm/swiotlb.h index ab05d73e2bb7..d2f69b9ff732 100644 --- a/arch/x86/include/asm/swiotlb.h +++ b/arch/x86/include/asm/swiotlb.h @@ -31,9 +31,9 @@ static inline void dma_mark_clean(void *addr, size_t size) {} extern void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, dma_addr_t *dma_handle, gfp_t flags, - struct dma_attrs *attrs); + unsigned long attrs); extern void x86_swiotlb_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_addr, - struct dma_attrs *attrs); + unsigned long attrs); #endif /* _ASM_X86_SWIOTLB_H */ diff --git a/arch/x86/include/asm/xen/page-coherent.h b/arch/x86/include/asm/xen/page-coherent.h index acd844c017d3..f02f025ff988 100644 --- a/arch/x86/include/asm/xen/page-coherent.h +++ b/arch/x86/include/asm/xen/page-coherent.h @@ -2,12 +2,11 @@ #define _ASM_X86_XEN_PAGE_COHERENT_H #include -#include #include static inline void *xen_alloc_coherent_pages(struct device *hwdev, size_t size, dma_addr_t *dma_handle, gfp_t flags, - struct dma_attrs *attrs) + unsigned long attrs) { void *vstart = (void*)__get_free_pages(flags, get_order(size)); *dma_handle = virt_to_phys(vstart); @@ -16,18 +15,18 @@ static inline void *xen_alloc_coherent_pages(struct device *hwdev, size_t size, static inline void xen_free_coherent_pages(struct device *hwdev, size_t size, void *cpu_addr, dma_addr_t dma_handle, - struct dma_attrs *attrs) + unsigned long attrs) { free_pages((unsigned long) cpu_addr, get_order(size)); } static inline void xen_dma_map_page(struct device *hwdev, struct page *page, dma_addr_t dev_addr, unsigned long offset, size_t size, -enum dma_data_direction dir, struct dma_attrs *attrs) { } +enum dma_data_direction dir, unsigned long attrs) { } static inline void xen_dma_unmap_page(struct device *hwdev, dma_addr_t handle, size_t size, enum dma_data_direction dir, - struct dma_attrs *attrs) { } + unsigned long attrs) { } static inline void xen_dma_sync_single_for_cpu(struct device *hwdev, dma_addr_t handle, size_t size, enum dma_data_direction dir) { } diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 8e3842fc8bea..4aff288e37a4 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -242,7 +242,7 @@ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem, static dma_addr_t gart_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, - struct dma_attrs *attrs) + unsigned long attrs) {
[PATCH v6 45/46] dma-mapping: Remove dma_get_attr
After switching DMA attributes to unsigned long it is easier to just compare the bits. Signed-off-by: Krzysztof Kozlowski [for avr32] Acked-by: Hans-Christian Noren Egtvedt [for arc] Acked-by: Vineet Gupta [for arm64 and dma-iommu] Acked-by: Robin Murphy --- Documentation/DMA-API.txt | 4 +-- arch/arc/mm/dma.c | 4 +-- arch/arm/mm/dma-mapping.c | 36 -- arch/arm/xen/mm.c | 4 +-- arch/arm64/mm/dma-mapping.c| 10 +++ arch/avr32/mm/dma-coherent.c | 4 +-- arch/ia64/sn/pci/pci_dma.c | 10 ++- arch/metag/kernel/dma.c| 2 +- arch/mips/mm/dma-default.c | 6 ++--- arch/openrisc/kernel/dma.c | 4 +-- arch/parisc/kernel/pci-dma.c | 2 +- arch/powerpc/platforms/cell/iommu.c| 12 - drivers/gpu/drm/rockchip/rockchip_drm_gem.c| 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/media/v4l2-core/videobuf2-dma-contig.c | 2 +- include/linux/dma-mapping.h| 10 --- 16 files changed, 47 insertions(+), 67 deletions(-) diff --git a/Documentation/DMA-API.txt b/Documentation/DMA-API.txt index 24f9688bb98a..1d26eeb6b5f6 100644 --- a/Documentation/DMA-API.txt +++ b/Documentation/DMA-API.txt @@ -422,9 +422,7 @@ void whizco_dma_map_sg_attrs(struct device *dev, dma_addr_t dma_addr, unsigned long attrs) { - int foo = dma_get_attr(DMA_ATTR_FOO, attrs); - - if (foo) + if (attrs & DMA_ATTR_FOO) /* twizzle the frobnozzle */ diff --git a/arch/arc/mm/dma.c b/arch/arc/mm/dma.c index 3d1f467d1792..74bbe68dce9d 100644 --- a/arch/arc/mm/dma.c +++ b/arch/arc/mm/dma.c @@ -46,7 +46,7 @@ static void *arc_dma_alloc(struct device *dev, size_t size, * (vs. always going to memory - thus are faster) */ if ((is_isa_arcv2() && ioc_exists) || - dma_get_attr(DMA_ATTR_NON_CONSISTENT, attrs)) + (attrs & DMA_ATTR_NON_CONSISTENT)) need_coh = 0; /* @@ -95,7 +95,7 @@ static void arc_dma_free(struct device *dev, size_t size, void *vaddr, struct page *page = virt_to_page(dma_handle); int is_non_coh = 1; - is_non_coh = dma_get_attr(DMA_ATTR_NON_CONSISTENT, attrs) || + is_non_coh = (attrs & DMA_ATTR_NON_CONSISTENT) || (is_isa_arcv2() && ioc_exists); if (PageHighMem(page) || !is_non_coh) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index ebb3fde99043..43e03b5293d0 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -126,7 +126,7 @@ static dma_addr_t arm_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) { - if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) + if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) __dma_page_cpu_to_dev(page, offset, size, dir); return pfn_to_dma(dev, page_to_pfn(page)) + offset; } @@ -155,7 +155,7 @@ static dma_addr_t arm_coherent_dma_map_page(struct device *dev, struct page *pag static void arm_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, unsigned long attrs) { - if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) + if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) __dma_page_dev_to_cpu(pfn_to_page(dma_to_pfn(dev, handle)), handle & ~PAGE_MASK, size, dir); } @@ -622,9 +622,9 @@ static void __free_from_contiguous(struct device *dev, struct page *page, static inline pgprot_t __get_dma_pgprot(unsigned long attrs, pgprot_t prot) { - prot = dma_get_attr(DMA_ATTR_WRITE_COMBINE, attrs) ? - pgprot_writecombine(prot) : - pgprot_dmacoherent(prot); + prot = (attrs & DMA_ATTR_WRITE_COMBINE) ? + pgprot_writecombine(prot) : + pgprot_dmacoherent(prot); return prot; } @@ -744,7 +744,7 @@ static void *__dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, .gfp = gfp, .prot = prot, .caller = caller, - .want_vaddr = !dma_get_attr(DMA_ATTR_NO_KERNEL_MAPPING, attrs), + .want_vaddr = ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) == 0), }; #ifdef CONFIG_DMA_API_DEBUG @@ -887,7 +887,7 @@ static void __arm_dma_free(struct device *dev, size_t size, void *cpu_addr, .size = PAGE_ALIGN(size), .cpu_addr = cpu_addr, .page = page, - .want_vaddr = !dma_get_attr(DMA_ATTR_NO_KERNEL_MAPPING, attrs), +
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote: > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > activated, there is high probability of NMI-s in random places. > > > > Hmm, strange. But nothing could really surprise when you have an HP > > BIOS. > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > the release notes only describe updated CPU microcode for security > reasons. It is probably something HP is selling as a "feature" and not a BIOS bug. > > Can you probably use the faulty config and bisect this down to a > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > got merged, but I have no idea how those could cause NMIs. > > Will try but I do not know a working base yet - this was broken in both > 4.6 and 4.7-rc. Oh, in that case it is not related to the recent iova changes. Does the box have any hardware error log which you can access and send to us (right after some NMIs happened)? Thanks, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
> > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > > activated, there is high probability of NMI-s in random places. > > > > > > Hmm, strange. But nothing could really surprise when you have an HP > > > BIOS. > > > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > > the release notes only describe updated CPU microcode for security > > reasons. > > It is probably something HP is selling as a "feature" and not a BIOS > bug. ROM setup settings that might be of interest: Advanced memory protection: advanced ecc support No-Execute memory protection: enabled Intel virtualization technology: enabled Intel hyperthreading options: enabled Processor core disable: all cored enabled Intel turbo boost technology: enabled Intel VT-d: enabled HP power profile: custom HP power regulator: hp dynamic power savings mode (not OS control) Intel qpi link power management: enabled Minimum processor idle power core state: C6 Minimum processor idle power package state: C6 Dynamic power saving mode response: fast Collaborative power control: enabled MPS table: full table apic NMI debug button: enabled PCI bus padding options: enabled HW prefetcher: enabled Adjacent sector prefetch: enabled Node interleaving: disabled > > > Can you probably use the faulty config and bisect this down to a > > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > > got merged, but I have no idea how those could cause NMIs. > > > > Will try but I do not know a working base yet - this was broken in both > > 4.6 and 4.7-rc. > > Oh, in that case it is not related to the recent iova changes. Does the > box have any hardware error log which you can access and send to us > (right after some NMIs happened)? Nothing in ILO log or integrated management log (IML). -- Meelis Roos (mr...@linux.ee) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
> > > Can you probably use the faulty config and bisect this down to a > > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > > got merged, but I have no idea how those could cause NMIs. > > > > Will try but I do not know a working base yet - this was broken in both > > 4.6 and 4.7-rc. > > Oh, in that case it is not related to the recent iova changes. Does the > box have any hardware error log which you can access and send to us > (right after some NMIs happened)? Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with BIOS settings (disabling NUMA). It is the first time I see at least some info in NMI decode. -- Meelis Roos (mr...@linux.ee) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, Jul 13, 2016 at 12:16:46PM +0300, Meelis Roos wrote: > ROM setup settings that might be of interest: > > Advanced memory protection: advanced ecc support > No-Execute memory protection: enabled > Intel virtualization technology: enabled > Intel hyperthreading options: enabled > Processor core disable: all cored enabled > Intel turbo boost technology: enabled > Intel VT-d: enabled > HP power profile: custom > HP power regulator: hp dynamic power savings mode (not OS control) > Intel qpi link power management: enabled > Minimum processor idle power core state: C6 > Minimum processor idle power package state: C6 > Dynamic power saving mode response: fast > Collaborative power control: enabled > MPS table: full table apic > NMI debug button: enabled > PCI bus padding options: enabled > HW prefetcher: enabled > Adjacent sector prefetch: enabled > Node interleaving: disabled AFAICT these settings look normal. > Nothing in ILO log or integrated management log (IML). Then I think we need a bisect to make progress here. Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 00/20] iommu/amd: Use generic IOVA allocator
On 2016年07月12日 18:55, Joerg Roedel wrote: Hey Vincent, On Tue, Jul 12, 2016 at 05:03:08PM +0800, Wan Zongshun wrote: Currently, those patches can not work at my eCarrizo board. When I merged your patches, boot failed, and no any info print to me. I set iommu=pt, it also does not work; set iommu=soft, boot ok. When I removed those patches, kernel boot ok. This eCarrizo board uart doesnot work, so I can not get useful information from kernel by uart console, I also set 'text' in boot option, but still cannot print any log. Thanks for your testing. The issue turned out to be an older bug, which just got uncovered by these patches. I updated the branch at git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux.git amd-iommu-iova This branch boots now on my Kaveri and Carrizo system. Can you please give it a test too? Joerg, I have tested the patches, it works now. Thanks, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, Jul 13, 2016 at 12:40:39PM +0300, Meelis Roos wrote: > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > BIOS settings (disabling NUMA). It is the first time I see at least some > info in NMI decode. This looks interesting. Can you please post output of 'lspci -vvv' and 'lspci -t'? Thanks, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 00/20] iommu/amd: Use generic IOVA allocator
On Wed, Jul 13, 2016 at 05:44:24PM +0800, Wan Zongshun wrote: > > > On 2016年07月12日 18:55, Joerg Roedel wrote: > >This branch boots now on my Kaveri and Carrizo system. Can you please > >give it a test too? > > Joerg, I have tested the patches, it works now. Great, thanks a lot for your testing. Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] iommu/vt-d: Remove unnecassary qi clflushes
On Fri, Jun 24, 2016 at 06:13:14AM -0700, Nadav Amit wrote: > According to the manual: "Hardware access to ... invalidation queue ... > are always coherent." > > Remove unnecassary clflushes accordingly. > > Signed-off-by: Nadav Amit Applied, thanks. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote: > > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > > > BIOS settings (disabling NUMA). It is the first time I see at least some > > > info in NMI decode. > > > > This looks interesting. Can you please post output of 'lspci -vvv' and > > 'lspci -t'? > > Here. Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit PCI-devices behind it. One of these devices tries to read address 0xb000, which is blocked by the IOMMU and causes the fault seen in the screen-shot. The fault also causes a PCI-error which is then reported through the NMI, causing your kernel panic. So the 32bit PCI devices behind the bridge are: 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI]) 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01) Can you try to disable this 'Lights Out' processor? Maybe it is causing the issues. On the other side, the radeon driver for the ATI card is also know for causing faults from time to time. Can you capture the kernel messages right before a crash too? Thanks, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 16/20 v2] iommu/amd: Optimize map_sg and unmap_sg
On Tue, Jul 12, 2016 at 04:34:16PM +0100, Robin Murphy wrote: > The boundary masks for block devices are tricky to track down through so > many layers of indirection in the common frameworks, but there are a lot > of 64K ones there. After some more impromptu digging into the subject > I've finally satisfied my curiosity - it seems this restriction stems > from the ATA DMA PRD table format, so it could perhaps still be a real > concern for anyone using some crusty old PCI IDE card in their modern > system. The boundary-mask is a capability of the underlying PCI device, no? The ATA or whatever-stack above should have no influence on it. > > Indeed, I wasn't suggesting making more than one call, just that > alloc_iova_fast() is quite likely to have to fall back to alloc_iova() > here, so there may be some mileage in going directly to the latter, with > the benefit of then being able to rely on find_iova() later (since you > know for sure you allocated out of the tree rather than the caches). My > hunch is that dma_map_sg() tends to be called for bulk data transfer > (block devices, DRM, etc.) so is probably a less contended path compared > to the network layer hammering dma_map_single(). Using different functions for allocation would also require special handling in the queued-freeing code, as I have to track the allocation then to know wheter I free it with the _fast variant or not. > > + mask = dma_get_seg_boundary(dev); > > + boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : > > + 1UL << (BITS_PER_LONG - PAGE_SHIFT); > > (mask >> PAGE_SHIFT) + 1 ? Should make no difference unless some of the first PAGE_SHIFT bits of mask is 0 (which shouldn't happen). Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] iommu: simplify and fix ida handling
On Mon, Jul 04, 2016 at 09:01:18PM +0200, Heiner Kallweit wrote: > Am 04.07.2016 um 13:46 schrieb Joerg Roedel: > > On Wed, Jun 29, 2016 at 09:13:59PM +0200, Heiner Kallweit wrote: > >> Ida handling can be much simplified by using the ida_simple_.. functions. > >> > >> This change also fixes the bug that previously checking for errors > >> returned by ida_get_new() was incomplete. > >> ida_get_new() can return errors other than EAGAIN, e.g. ENOSPC. > >> This case wasn't handled. > >> > >> Signed-off-by: Heiner Kallweit > >> --- > >> drivers/iommu/iommu.c | 24 ++-- > >> 1 file changed, 6 insertions(+), 18 deletions(-) > > > > The patch does not apply. Please rebase it against the latest upstream > > kernel and resend. > > > This patch applies on top of patch "[PATCH 1/3] iommu: simplify init function" > I submitted on June 28th. Do you have an opinion on that patch? Applied both, thanks. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH] iommu/amd: Init unity mappings only for dma_ops domains
From: Joerg Roedel The default domain for a device might also be identity-mapped. In this case the kernel would crash when unity mappings are defined for the device. Fix that by making sure the domain is a dma_ops domain. Fixes: 0bb6e243d7fb ('iommu/amd: Support IOMMU_DOMAIN_DMA type allocation') Cc: sta...@vger.kernel.org # v4.2+ Signed-off-by: Joerg Roedel --- drivers/iommu/amd_iommu.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 92e..b938a4a 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -467,9 +467,11 @@ static void init_iommu_group(struct device *dev) if (!domain) goto out; - dma_domain = to_pdomain(domain)->priv; + if (to_pdomain(domain)->flags == PD_DMA_OPS_MASK) { + dma_domain = to_pdomain(domain)->priv; + init_unity_mappings_for_device(dev, dma_domain); + } - init_unity_mappings_for_device(dev, dma_domain); out: iommu_group_put(group); } -- 2.6.6 ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH] iommu/vt-d: return error code in domain_context_mapping_one()
In 'commit <55d940430ab9> ("iommu/vt-d: Get rid of domain->iommu_lock")', the error handling path is changed a little, which makes the function always return 0. This path fixes this. Signed-off-by: Wei Yang --- drivers/iommu/intel-iommu.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 323dac9..d416242 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -2076,7 +2076,7 @@ out_unlock: spin_unlock(&iommu->lock); spin_unlock_irqrestore(&device_domain_lock, flags); - return 0; + return ret; } struct domain_context_mapping_data { -- 1.7.9.5 ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, 13 Jul 2016, Joerg Roedel wrote: > On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote: > > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > > activated, there is high probability of NMI-s in random places. > > > > > > Hmm, strange. But nothing could really surprise when you have an HP > > > BIOS. > > > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > > the release notes only describe updated CPU microcode for security > > reasons. > > It is probably something HP is selling as a "feature" and not a BIOS > bug. Well, based on the date, it should be the microcode-level fix for this: https://www.blackhat.com/docs/us-15/materials/us-15-Domas-The-Memory-Sinkhole-Unleashing-An-x86-Design-Flaw-Allowing-Universal-Privilege-Escalation-wp.pdf -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
On Wed, 13 Jul 2016 12:18:59 +0200 Joerg Roedel wrote: > On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote: > > > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > > > > BIOS settings (disabling NUMA). It is the first time I see at least > > > > some > > > > info in NMI decode. > > > > > > This looks interesting. Can you please post output of 'lspci -vvv' and > > > 'lspci -t'? > > > > Here. > > Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > PCI-devices behind it. One of these devices tries to read address > 0xb000, which is blocked by the IOMMU and causes the fault seen in the > screen-shot. The fault also causes a PCI-error which is then reported > through the NMI, causing your kernel panic. > > So the 32bit PCI devices behind the bridge are: > > 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] > ES1000 (rev 02) (prog-if 00 [VGA controller]) > 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out > Controller (rev 03) > 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out > Processor (rev 03) > 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out > Standard Virtual USB Controller (prog-if 00 [UHCI]) > 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out > Standard KCS Interface (prog-if 01) > > Can you try to disable this 'Lights Out' processor? Maybe it is causing > the issues. On the other side, the radeon driver for the ATI card is > also know for causing faults from time to time. Can you capture the > kernel messages right before a crash too? IIRC, blacklisting the hpwdt module can defuse those NMIs and might help us see more of the actual DMAR faults. Blacklist in modprobe.d and rebuild initrd. Thanks, Alex PS - never assume BIOS release notes are actually complete ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s
On 7/13/2016 10:48 AM, Alex Williamson wrote: > On Wed, 13 Jul 2016 12:18:59 +0200 > Joerg Roedel wrote: > >> On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote: > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > BIOS settings (disabling NUMA). It is the first time I see at least some > info in NMI decode. This looks interesting. Can you please post output of 'lspci -vvv' and 'lspci -t'? >>> >>> Here. >> >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit >> PCI-devices behind it. One of these devices tries to read address >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the >> screen-shot. The fault also causes a PCI-error which is then reported >> through the NMI, causing your kernel panic. >> >> So the 32bit PCI devices behind the bridge are: >> >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] >> ES1000 (rev 02) (prog-if 00 [VGA controller]) >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out >> Controller (rev 03) >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out >> Processor (rev 03) >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out >> Standard Virtual USB Controller (prog-if 00 [UHCI]) >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out >> Standard KCS Interface (prog-if 01) >> >> Can you try to disable this 'Lights Out' processor? Maybe it is causing >> the issues. On the other side, the radeon driver for the ATI card is >> also know for causing faults from time to time. Can you capture the >> kernel messages right before a crash too? > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might > help us see more of the actual DMAR faults. Blacklist in modprobe.d > and rebuild initrd. Thanks, > > Alex > > PS - never assume BIOS release notes are actually complete I agree. I'd do the BIOS update and also make sure the iLO FW is current. -- ljk > ___ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH] iommu/dma-iommu: respect established iova region limits
In the current dma-iommu implementation, the upper limit used when allocating iovas is based on the calling device's dma_mask without considering the potentially more restrictive iova limits established in iommu_dma_init_domain. To ensure that iovas are allocated within the expected iova region, this patch adds logic in __alloc_iova to clip input dma_limit values that are out of bounds. Signed-off-by: Nate Watterson --- drivers/iommu/dma-iommu.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ea5a9eb..2066066 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -157,11 +157,14 @@ static struct iova *__alloc_iova(struct iova_domain *iovad, size_t size, unsigned long shift = iova_shift(iovad); unsigned long length = iova_align(iovad, size) >> shift; + /* Respect the upper limit established in iommu_dma_init_domain */ + dma_limit = min_t(dma_addr_t, dma_limit >> shift, iovad->dma_32bit_pfn); + /* * Enforce size-alignment to be safe - there could perhaps be an * attribute to control this per-device, or at least per-domain... */ - return alloc_iova(iovad, length, dma_limit >> shift, true); + return alloc_iova(iovad, length, dma_limit, true); } /* The IOVA allocator knows what we mapped, so just unmap whatever that was */ -- Qualcomm Datacenter Technologies, Inc. on behalf of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[iommu:x86/amd 19/23] drivers/iommu/amd_iommu.c:2523:2: error: 'dma_dom' undeclared
tree: https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git x86/amd head: a2c9b667b1c0e78d39f339a90fd6a2bd4a6ff8d8 commit: 0b1c8fa308906ad01928609ac04e486d34894052 [19/23] iommu/amd: Optimize map_sg and unmap_sg config: x86_64-randconfig-s4-07140059 (attached as .config) compiler: gcc-6 (Debian 6.1.1-1) 6.1.1 20160430 reproduce: git checkout 0b1c8fa308906ad01928609ac04e486d34894052 # save the attached .config to linux build tree make ARCH=x86_64 Note: the iommu/x86/amd HEAD a2c9b667b1c0e78d39f339a90fd6a2bd4a6ff8d8 builds fine. It only hurts bisectibility. All errors (new ones prefixed by >>): drivers/iommu/amd_iommu.c: In function 'unmap_sg': >> drivers/iommu/amd_iommu.c:2523:2: error: 'dma_dom' undeclared (first use in >> this function) dma_dom = domain->priv; ^~~ drivers/iommu/amd_iommu.c:2523:2: note: each undeclared identifier is reported only once for each function it appears in vim +/dma_dom +2523 drivers/iommu/amd_iommu.c 2517 2518 domain = get_domain(dev); 2519 if (IS_ERR(domain)) 2520 return; 2521 2522 startaddr = sg_dma_address(sglist) & PAGE_MASK; > 2523 dma_dom = domain->priv; 2524 npages= sg_num_pages(dev, sglist, nelems); 2525 2526 __unmap_single(domain->priv, startaddr, npages << PAGE_SHIFT, dir); --- 0-DAY kernel test infrastructureOpen Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation .config.gz Description: Binary data ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] iommu/dma-iommu: respect established iova region limits
Hi Nate, On 13/07/16 17:41, Nate Watterson wrote: > In the current dma-iommu implementation, the upper limit used when > allocating iovas is based on the calling device's dma_mask without > considering the potentially more restrictive iova limits established > in iommu_dma_init_domain. To ensure that iovas are allocated within > the expected iova region, this patch adds logic in __alloc_iova to > clip input dma_limit values that are out of bounds. This was actually fairly deliberate - due to the current state of of_dma_configure(), iommu_dma_init() is mostly only ever called with the default 32-bit mask of any newly-created device. We only have true visibility of what the device might want to address after its probe routine has run, where the driver may have set a larger mask, but conversely that same probe routine may call dma_alloc_coherent() straight away, so the IOVA domain has to be initialised with *something* beforehand. Fortunately dma_32bit_pfn is more of an "expected upper bound" than an actual limit, so there's no real problem with allocating above it (in the same way intel-iommu does under similar circumstances). As such, I'm not sure that a change effectively just hard-limiting IOVAs to 4GB is all that worthwhile - what we're really missing here are much more significant things like being able to tie the IOVA size to what's actually wired up at the IOMMU input, and having proper dma_set_mask() implementations in the first place (for arm/arm64 in general, not just dma-iommu). Robin. > Signed-off-by: Nate Watterson > --- > drivers/iommu/dma-iommu.c | 5 - > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index ea5a9eb..2066066 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -157,11 +157,14 @@ static struct iova *__alloc_iova(struct iova_domain > *iovad, size_t size, > unsigned long shift = iova_shift(iovad); > unsigned long length = iova_align(iovad, size) >> shift; > > + /* Respect the upper limit established in iommu_dma_init_domain */ > + dma_limit = min_t(dma_addr_t, dma_limit >> shift, iovad->dma_32bit_pfn); > + > /* >* Enforce size-alignment to be safe - there could perhaps be an >* attribute to control this per-device, or at least per-domain... >*/ > - return alloc_iova(iovad, length, dma_limit >> shift, true); > + return alloc_iova(iovad, length, dma_limit, true); > } > > /* The IOVA allocator knows what we mapped, so just unmap whatever that was > */ > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH] iommu/iova: validate iova_domain input to put_iova_domain
Passing a NULL or uninitialized iova_domain into put_iova_domain will currently crash the kernel when the unconfigured iova_domain data members are accessed. To prevent this from occurring, this patch adds a check to make sure that the domain is non-NULL and that the domain granule is non-zero. The granule can be used to check if the domain was properly initialized because calling init_iova_domain with a granule of zero would have already triggered a BUG statement crashing the kernel. Signed-off-by: Nate Watterson --- drivers/iommu/iova.c | 4 1 file changed, 4 insertions(+) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index e23001b..3511a1c 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -459,6 +459,10 @@ void put_iova_domain(struct iova_domain *iovad) struct rb_node *node; unsigned long flags; + /* Only teardown properly initialized domains */ + if (!iovad || !iovad->granule) + return; + free_iova_rcaches(iovad); spin_lock_irqsave(&iovad->iova_rbtree_lock, flags); node = rb_first(&iovad->rbroot); -- Qualcomm Datacenter Technologies, Inc. on behalf of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s
> >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > >> PCI-devices behind it. One of these devices tries to read address > >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the > >> screen-shot. The fault also causes a PCI-error which is then reported > >> through the NMI, causing your kernel panic. > >> > >> So the 32bit PCI devices behind the bridge are: > >> > >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] > >> ES1000 (rev 02) (prog-if 00 [VGA controller]) > >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights > >> Out Controller (rev 03) > >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights > >> Out Processor (rev 03) > >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out > >> Standard Virtual USB Controller (prog-if 00 [UHCI]) > >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out > >> Standard KCS Interface (prog-if 01) > >> > >> Can you try to disable this 'Lights Out' processor? Maybe it is causing > >> the issues. On the other side, the radeon driver for the ATI card is > >> also know for causing faults from time to time. Can you capture the > >> kernel messages right before a crash too? > > > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might > > help us see more of the actual DMAR faults. Blacklist in modprobe.d > > and rebuild initrd. Thanks, > > > > Alex > > > > PS - never assume BIOS release notes are actually complete > > I agree. I'd do the BIOS update and also make sure the iLO FW is current. OK, updates to the latest BIOS, CPU microcode revison is now 0x1b instead of 0x19. ILO2 fw is already the latest. The NMI-s still happen. Disabled hpwdt. Now most boots are fine, only one hung with radeon loading: http://kodu.ut.ee/~mroos/boot-radeon-1.png Vusal patterns also remind me that all the previous hangs were before radeon mode change - once radeon changes the mode, it already works. -- Meelis Roos (mr...@linux.ee) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: IOMMU+DMAR causing NMIs-s
On Thu, 14 Jul 2016 00:52:02 +0300 (EEST) Meelis Roos wrote: > > >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > > >> PCI-devices behind it. One of these devices tries to read address > > >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the > > >> screen-shot. The fault also causes a PCI-error which is then reported > > >> through the NMI, causing your kernel panic. > > >> > > >> So the 32bit PCI devices behind the bridge are: > > >> > > >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. > > >> [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) > > >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights > > >> Out Controller (rev 03) > > >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights > > >> Out Processor (rev 03) > > >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out > > >> Standard Virtual USB Controller (prog-if 00 [UHCI]) > > >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated > > >> Lights-Out Standard KCS Interface (prog-if 01) > > >> > > >> Can you try to disable this 'Lights Out' processor? Maybe it is causing > > >> the issues. On the other side, the radeon driver for the ATI card is > > >> also know for causing faults from time to time. Can you capture the > > >> kernel messages right before a crash too? > > > > > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might > > > help us see more of the actual DMAR faults. Blacklist in modprobe.d > > > and rebuild initrd. Thanks, > > > > > > Alex > > > > > > PS - never assume BIOS release notes are actually complete > > > > I agree. I'd do the BIOS update and also make sure the iLO FW is current. > > OK, updates to the latest BIOS, CPU microcode revison is now 0x1b > instead of 0x19. ILO2 fw is already the latest. > > The NMI-s still happen. > > Disabled hpwdt. Now most boots are fine, only one hung with radeon > loading: > > http://kodu.ut.ee/~mroos/boot-radeon-1.png > > Vusal patterns also remind me that all the previous hangs were before > radeon mode change - once radeon changes the mode, it already works. There might be multiple things going on here, I tend to associate the NMIs with the DMAR faults on HP systems and I expect that disabling hpwdt will remove that DMAR fault to NMI escalation, but it won't fix the fact that DMAR faults are occurring. I'm not sure how any of that is related to radeon though other than the fact that the radeon device is behind the bridge that generated the DMAR fault and since we can't tell which device behind the bridge generated that transaction, it's possible that it was the radeon card (though we're all more likely to throw stones at the iLO first). With hpwdt disabled, can you reboot a few times and look for DMAR faults in the dmesg to see if they're all consistent, ie. device 1e.0 doing a read from 0xb000? Is there any correlation to radeon hanging and one of those DMAR faults appearing in the log (you can make use of the iLO for a serial console so you can record more of the boot than what can be captured on the VGA console). Also, this should be an X58-based system, which reminds me of Joerg's recent commit a4c34ff1c029 that's intended to fix a hang on such systems. That was included starting in v4.7-rc4, so if this is a new issue, testing before and after that commit might be relevant. Thanks, Alex ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu