2016-06-10 0:57 GMT+08:00 Lutz Vieweg <l...@5t9.de>: > On 06/09/2016 06:03 PM, Alexander Duyck wrote: >>> >>> This time I'll reboot the machine, and also try "iommu=pt" as suggested >>> in different places for use with 10G NICs. >> >> >> That might be a good place to start. >> >> I'm adding, or at least attempting to, the mailing list and maintainer >> for the IOMMU code. You might want to check with the AMD-Vi IOMMU >> maintainers to see if they have any other advice as this seems like >> something that may have been introduced with changes to the IOMMU as >> the ixgbe driver hasn't had any updates to the DMA mapping/unmapping >> code in some time and it was working in the 4.4 kernel series and >> still works on my system which runs an Intel IOMMU so I am wondering >> if this may be something specifically related to changes in the AMD >> IOMMU code. > > > After having rebooted the system with "iommu=pt", the following change > of dmesg-output looks curious to me: > > Without "iommu=pt": >> >> [ 4.869591] iommu: Adding device 0000:04:00.0 to group 13 > > ... >> >> [ 4.873105] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 >> [ 4.873347] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 >> [ 4.873586] AMD-Vi: Interrupt remapping enabled >> [ 4.874108] AMD-Vi: Lazy IO/TLB flushing enabled >
Ok, so there are two iommus controller in your system. > > With "iommu=pt": >> >> [ 4.832580] iommu: Adding device 0000:04:00.0 to group 13 >> [ 4.832838] iommu: Using direct mapping for device 0000:04:00.0 > That is right, you will pass through AMD IOMMU when you set iommu=pt. > ... >> >> [ 4.837074] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 >> [ 4.837305] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 >> [ 4.837535] AMD-Vi: Interrupt remapping enabled >> [ 4.838062] AMD-Vi: Lazy IO/TLB flushing enabled >> [ 4.838291] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) >> [ 4.838533] software IO TLB [mem 0xd3e80000-0xd7e80000] (64MB) mapped >> at [ffff8800d3e80000-ffff8800d7e7ffff] > > > I hope that doesn't mean all my network data is now passing through > an additional copy-by-CPU... that would be kind of the opposite of what > "iommu=pt" seemed to promise :-) It depends. Firstly, I need to know if your ethernet card works well now or not after you set iommu=pt. If your ethernet card with 64bit(not 32bit) DMA addressable cap, that is ok, you will not be impacted by bounce buffer. But iommu=pt is a terrible option, that make all devices bypass the iommu. If you want to get further help, Please try: (1)Please add 'amd_iommu_dump' option in your kernel boot option, and send your full kernel logs, lspci info, don't add iommu=pt. (2) Add amd_iommu=fullflush option to kernel boot option, just try it. > > One more thing I find curious, but this didn't change with "iommu=pt": >> >> [ 0.000000] AGP: Checking aperture... >> [ 0.000000] AGP: No AGP bridge found >> [ 0.000000] AGP: Node 0: aperture [bus addr 0x00000000-0x01ffffff] >> (32MB) >> [ 0.000000] AGP: Your BIOS doesn't leave an aperture memory hole >> [ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup >> [ 0.000000] AGP: This costs you 64MB of RAM >> [ 0.000000] AGP: Mapping aperture over RAM [mem 0xcc000000-0xcfffffff] >> (65536KB) > > I checked and the IOMMU-option is definitely enabled in the BIOS setup. > So I assume right that these message are irrelevant (since AGP as a whole > is irrelevant on this server)? Please cat /proc/iomem, send the information. > > Regards, > > Lutz Vieweg > > > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu -- --- Vincent Wan(Zongshun) www.mcuos.com _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
