Re: Fwd: USB device uses dma on its MMIO region?
On Thu, Jan 22, 2015 at 3:41 PM, Alan Stern wrote: > On Thu, 22 Jan 2015, Elena Ufimtseva wrote: > >> On Thu, Jan 22, 2015 at 1:10 PM, Alan Stern >> wrote: >> > On Thu, 22 Jan 2015, Elena Ufimtseva wrote: >> > >> >> Hello >> >> >> >> While working on IOMMU related problem for Xen Hypervisor, >> >> it was discovered that on some machines IOMMU page faults are triggered >> >> on certain addresses and requests that cause this are DMA requests >> >> with source device >> >> identified as USB Host controllers. >> > >> >> Thank you for taking a look at this Alan. >> >> > You mean EHCI controllers? >> >> Right, EHCI controllers. >> >> > >> >> Further analysis showed that faulting addresses are not mapped with IOMMU >> >> with >> >> RW as these regions are not enumerated as RMRRs. >> >> Xen does make sure that RMRRs are mapped RW in IOMMU. >> >> These faulting addresses fall into reserved region within e820 map. >> > >> > Are you saying that the addresses aren't mapped at all, or that they >> > are mapped RO rather than RW? >> >> These are not mapped. They dont have EPTs only because in code there is >> a condition missing. RMRRs enumerated by ACPI RMRRs are mapped with RW >> access. >> I think this lead to a question being discussed on xen-devel on why >> there is a device >> that would initiate DMA request on reserved and not rmrr marked region. > > I don't know what "EPT" and "RMRR" mean. RMRR is ACPI Reserved memory Region Reporting Structure reported for iommu unit. EPT - extended page tables in Intel VT-x. > >> >> This page faults with addresses/devices are being logged under Xen: >> >> >> >> (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs >> >> (XEN) IOMMU Page fault! >> >> (XEN) IOMMU Page fault! >> >> (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow >> >> (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault >> >> (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1a.0] >> >> fault addr d5d46000, iommu reg = 82c000203000 >> > >> > Why do you think d6d46000 is in the MMIO region for the :00:1a.0 >> > device? According to the log: >> > >> >> [ 13.340574] ehci-pci :00:1a.0: irq 17, io mem 0xf7c38000 >> > >> > So the MMIO region starts at f7c38000. >> >> Right, I agree and I saw it. Its not MMIO, rather its assumed that if its not >> RAM in e820 map, that it may be MMIO. I probably should not use this term. >> In this case faulting addresses do not match any of MMIOs reported per device >> or RMRRs. >> At the same time the identified devices do initiate DMA reads for the >> addresses mentioned. >> I just wanted to see what it can be and understand the better way to solve >> it. > > If that address isn't mapped to RAM then there's no reason for the EHCI > controller to issue a DMA using the address. Understood. > > On the other hand, it's not easy to find all the DMA addresses that an > EHCI controller uses. There are a few coherent DMA pools, some static > coherent mappings, and some dynamic streaming mappings. > > One place to start looking is in the files under > /sys/kernel/debug/usb/ehci/:00:1a.0/. Thank you Alan, I will look into this. > > Alan Stern > -- Elena -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Fwd: USB device uses dma on its MMIO region?
On Thu, 22 Jan 2015, Elena Ufimtseva wrote: > On Thu, Jan 22, 2015 at 1:10 PM, Alan Stern wrote: > > On Thu, 22 Jan 2015, Elena Ufimtseva wrote: > > > >> Hello > >> > >> While working on IOMMU related problem for Xen Hypervisor, > >> it was discovered that on some machines IOMMU page faults are triggered > >> on certain addresses and requests that cause this are DMA requests > >> with source device > >> identified as USB Host controllers. > > > > Thank you for taking a look at this Alan. > > > You mean EHCI controllers? > > Right, EHCI controllers. > > > > >> Further analysis showed that faulting addresses are not mapped with IOMMU > >> with > >> RW as these regions are not enumerated as RMRRs. > >> Xen does make sure that RMRRs are mapped RW in IOMMU. > >> These faulting addresses fall into reserved region within e820 map. > > > > Are you saying that the addresses aren't mapped at all, or that they > > are mapped RO rather than RW? > > These are not mapped. They dont have EPTs only because in code there is > a condition missing. RMRRs enumerated by ACPI RMRRs are mapped with RW access. > I think this lead to a question being discussed on xen-devel on why > there is a device > that would initiate DMA request on reserved and not rmrr marked region. I don't know what "EPT" and "RMRR" mean. > >> This page faults with addresses/devices are being logged under Xen: > >> > >> (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs > >> (XEN) IOMMU Page fault! > >> (XEN) IOMMU Page fault! > >> (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow > >> (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault > >> (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1a.0] > >> fault addr d5d46000, iommu reg = 82c000203000 > > > > Why do you think d6d46000 is in the MMIO region for the :00:1a.0 > > device? According to the log: > > > >> [ 13.340574] ehci-pci :00:1a.0: irq 17, io mem 0xf7c38000 > > > > So the MMIO region starts at f7c38000. > > Right, I agree and I saw it. Its not MMIO, rather its assumed that if its not > RAM in e820 map, that it may be MMIO. I probably should not use this term. > In this case faulting addresses do not match any of MMIOs reported per device > or RMRRs. > At the same time the identified devices do initiate DMA reads for the > addresses mentioned. > I just wanted to see what it can be and understand the better way to solve it. If that address isn't mapped to RAM then there's no reason for the EHCI controller to issue a DMA using the address. On the other hand, it's not easy to find all the DMA addresses that an EHCI controller uses. There are a few coherent DMA pools, some static coherent mappings, and some dynamic streaming mappings. One place to start looking is in the files under /sys/kernel/debug/usb/ehci/:00:1a.0/. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Fwd: USB device uses dma on its MMIO region?
On Thu, Jan 22, 2015 at 1:10 PM, Alan Stern wrote: > On Thu, 22 Jan 2015, Elena Ufimtseva wrote: > >> Hello >> >> While working on IOMMU related problem for Xen Hypervisor, >> it was discovered that on some machines IOMMU page faults are triggered >> on certain addresses and requests that cause this are DMA requests >> with source device >> identified as USB Host controllers. > Thank you for taking a look at this Alan. > You mean EHCI controllers? Right, EHCI controllers. > >> Further analysis showed that faulting addresses are not mapped with IOMMU >> with >> RW as these regions are not enumerated as RMRRs. >> Xen does make sure that RMRRs are mapped RW in IOMMU. >> These faulting addresses fall into reserved region within e820 map. > > Are you saying that the addresses aren't mapped at all, or that they > are mapped RO rather than RW? These are not mapped. They dont have EPTs only because in code there is a condition missing. RMRRs enumerated by ACPI RMRRs are mapped with RW access. I think this lead to a question being discussed on xen-devel on why there is a device that would initiate DMA request on reserved and not rmrr marked region. > >> Need help to understand what the use is of this? >> If it is some kind of USB debug device functionality using DMA on its >> MMIOs? Something else? >> I would like to understand this better what is going on here. > > USB controllers only do DMA to addresses that are given to them by > their driver. These addresses are obtained using the regular DMA > allocation routines (for example, dma_pool_create() or > dma_alloc_coherent() or dma_map_single()). They don't do DMA to > addresses in the MMIO region. > >> xen-devel related conversations: >> http://www.gossamer-threads.com/lists/xen/devel/363257 >> and >> http://www.gossamer-threads.com/lists/xen/devel/363355 >> >> Please let me know if more details needed. >> Thank you. >> >> >> >> This page faults with addresses/devices are being logged under Xen: >> >> (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs >> (XEN) IOMMU Page fault! >> (XEN) IOMMU Page fault! >> (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow >> (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault >> (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1a.0] >> fault addr d5d46000, iommu reg = 82c000203000 > > Why do you think d6d46000 is in the MMIO region for the :00:1a.0 > device? According to the log: > >> [ 13.340574] ehci-pci :00:1a.0: irq 17, io mem 0xf7c38000 > > So the MMIO region starts at f7c38000. Right, I agree and I saw it. Its not MMIO, rather its assumed that if its not RAM in e820 map, that it may be MMIO. I probably should not use this term. In this case faulting addresses do not match any of MMIOs reported per device or RMRRs. At the same time the identified devices do initiate DMA reads for the addresses mentioned. I just wanted to see what it can be and understand the better way to solve it. Elena > > Alan Stern > -- Elena -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Fwd: USB device uses dma on its MMIO region?
On Thu, 22 Jan 2015, Elena Ufimtseva wrote: > Hello > > While working on IOMMU related problem for Xen Hypervisor, > it was discovered that on some machines IOMMU page faults are triggered > on certain addresses and requests that cause this are DMA requests > with source device > identified as USB Host controllers. You mean EHCI controllers? > Further analysis showed that faulting addresses are not mapped with IOMMU with > RW as these regions are not enumerated as RMRRs. > Xen does make sure that RMRRs are mapped RW in IOMMU. > These faulting addresses fall into reserved region within e820 map. Are you saying that the addresses aren't mapped at all, or that they are mapped RO rather than RW? > Need help to understand what the use is of this? > If it is some kind of USB debug device functionality using DMA on its > MMIOs? Something else? > I would like to understand this better what is going on here. USB controllers only do DMA to addresses that are given to them by their driver. These addresses are obtained using the regular DMA allocation routines (for example, dma_pool_create() or dma_alloc_coherent() or dma_map_single()). They don't do DMA to addresses in the MMIO region. > xen-devel related conversations: > http://www.gossamer-threads.com/lists/xen/devel/363257 > and > http://www.gossamer-threads.com/lists/xen/devel/363355 > > Please let me know if more details needed. > Thank you. > > > > This page faults with addresses/devices are being logged under Xen: > > (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs > (XEN) IOMMU Page fault! > (XEN) IOMMU Page fault! > (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow > (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault > (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1a.0] > fault addr d5d46000, iommu reg = 82c000203000 Why do you think d6d46000 is in the MMIO region for the :00:1a.0 device? According to the log: > [ 13.340574] ehci-pci :00:1a.0: irq 17, io mem 0xf7c38000 So the MMIO region starts at f7c38000. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fwd: USB device uses dma on its MMIO region?
Hello While working on IOMMU related problem for Xen Hypervisor, it was discovered that on some machines IOMMU page faults are triggered on certain addresses and requests that cause this are DMA requests with source device identified as USB Host controllers. Further analysis showed that faulting addresses are not mapped with IOMMU with RW as these regions are not enumerated as RMRRs. Xen does make sure that RMRRs are mapped RW in IOMMU. These faulting addresses fall into reserved region within e820 map. Need help to understand what the use is of this? If it is some kind of USB debug device functionality using DMA on its MMIOs? Something else? I would like to understand this better what is going on here. xen-devel related conversations: http://www.gossamer-threads.com/lists/xen/devel/363257 and http://www.gossamer-threads.com/lists/xen/devel/363355 Please let me know if more details needed. Thank you. This page faults with addresses/devices are being logged under Xen: (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs (XEN) IOMMU Page fault! (XEN) IOMMU Page fault! (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1a.0] fault addr d5d46000, iommu reg = 82c000203000 (XEN) DMAR:[fault reason 06h] PTE Read access is not set (XEN) print_vtd_entries: iommu 83041ac0c9b0 dev :00:1a.0 gmfn d5d46 (XEN) root_entry = 83041ac07000 (XEN) root_entry[0] = 291cc9001 (XEN) context = 830291cc9000 (XEN) context[d0] = 2_2920d3001 (XEN) l4 = 8302920d3000 (XEN) l4_index = 0 (XEN) l4[0] = 2920d2003 (XEN) l3 = 8302920d2000 (XEN) l3_index = 3 (XEN) l3[3] = 291cc8003 (XEN) IOMMU Page fault! (XEN) IOMMU Page fault! (XEN) [VT-D]iommu.c:875: iommu_fault_status: Fault Overflow (XEN) [VT-D]iommu.c:877: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:855: DMAR:[DMA Read] Request device [:00:1d.0] fault addr d5d45000, iommu reg = 82c000203000 (XEN) DMAR:[fault reason 06h] PTE Read access is not set (XEN) print_vtd_entries: iommu 83041ac0c9b0 dev :00:1d.0 gmfn d5d45 (XEN) root_entry = 83041ac07000 (XEN) root_entry[0] = 291cc9001 (XEN) context = 830291cc9000 (XEN) context[e8] = 2_2920d3001 (XEN) l4 = 8302920d3000 (XEN) l4_index = 0 (XEN) l4[0] = 2920d2003 (XEN) l3 = 8302920d2000 (XEN) l3_index = 3 (XEN) l3[3] = 291cc8003 (XEN) l2 = 830291cc8000 (XEN) l2_index = ae (XEN) l2[ae] = 0 (XEN) l2[ae] not present Here are machine details: Following info collected using bare metal 3.18.0-rc7. Lenovo ThinkCentre M Series; dmesg: [0.00] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [0.00] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [0.00] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [0.00] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [0.00] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled) [0.00] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled) [0.00] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled) [0.00] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) [0.00] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [0.00] ACPI: IOAPIC (id[0x08] address[0xfec0] gsi_base[0]) [0.00] IOAPIC[0]: apic_id 8, version 32, address 0xfec0, GSI 0-23 [0.00] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [0.00] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [0.00] ACPI: IRQ0 used by override. [0.00] ACPI: IRQ9 used by override. [0.00] Using ACPI (MADT) for SMP configuration information [0.00] ACPI: HPET id: 0x8086a701 base: 0xfed0 [0.00] smpboot: Allowing 8 CPUs, 0 hotplug CPUs [0.00] PM: Registered nosave memory: [mem 0x-0x0fff] [0.00] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [0.00] PM: Registered nosave memory: [mem 0x0009e000-0x0009] [0.00] PM: Registered nosave memory: [mem 0x000a-0x000d] [0.00] PM: Registered nosave memory: [mem 0x000e-0x000f] [0.00] PM: Registered nosave memory: [mem 0xc27e-0xc27e6fff] [0.00] PM: Registered nosave memory: [mem 0xc2c2f000-0xc30d5fff] [0.00] PM: Registered nosave memory: [mem 0xd48ea000-0xd4975fff] [0.00] PM: Registered nosave memory: [mem 0xd4976000-0xd49d7fff] [0.00] PM: Registered nosave memory: [mem 0xd49d8000-0xd56e] [0.00] PM: Registered nosave memory: [mem 0xd56f-0xd5ffefff] [0.00] PM: Registered nosave memory: [mem 0xd600-0xd6ff] [0.00] PM: Registered nosave memory: [mem 0xd700-0xdf1f] [0.00] PM: Registered nosave memory: [mem 0xdf20-0xf7ff] [0.00] PM: Registered nosave memory: [mem 0xf800
