ccing bug for the record on testing. On Wed, May 10, 2017 at 10:07 AM, Nate Watterson <[email protected]> wrote: > Unless this is a PCIe attached SATA disk, you will not see any impact > from the iommu.passthrough setting since the SATA SMMUs are not even > enabled! Additionally, the FIO results you sent last week I think > showed rates beyond the physical capabilities of SATA so you should > probably make sure that file system caching is disabled or just > access the disk directly using a readonly profile. Also, the SMMU is > sufficiently fast that you will typically not even see the SMMU > impact on a relatively low BW peripheral like SATA. I would recommend > getting an NVME drive. > -Nate > > From: Manoj Iyer [mailto:[email protected]] > Sent: Wednesday, May 10, 2017 10:05 AM > To: Nate Watterson <[email protected]> > Cc: Timur Tabi <[email protected]> > Subject: RE: how to test iommu.passthough=0/1 ?? > > > > On Wed, May 10, 2017 at 8:33 AM, Nate Watterson > <[email protected]> wrote: > >> Ok, what is the physical device that ./fio.virtualfs is being >> created on? Is it and NVME or SATA (internal or PCI attached) disk? > > Ah! it was on the SATA drive. Sorry misunderstood your initial > question. > > >> -----Original Message----- From: Manoj Iyer >> [mailto:[email protected]] Sent: Wednesday, May 10, 2017 9:06 >> AM To: Nate Watterson <[email protected]> Cc: Manoj Iyer >> <[email protected]>; Timur Tabi <[email protected]> >> Subject: RE: how to test iommu.passthough=0/1 ?? On Wed, 10 May >> 2017, Nate Watterson wrote: >>> Manoj, What was the backing store for the file mounted as >>> /dev/loop0 in the results you sent? >> dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 mkfs.ext4 >> ./fio.virtualfs sudo losetup /dev/loop0 ./fio.virtualfs >>> For iperf, I typically only test the target device as the iperf >>> server and directly connect a host PC to run the client threads. I >>> test this way because it usually has the worst performance with >>> SMMU enabled. Target CMD: iperf -s Client CMD: iperf -c >>> 192.168.2.222 -fg -i10 -t60 -P8 -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Tuesday, May 9, 2017 10:29 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? Nate, Thanks for that info. I will give >>> that a try. Did you see my iperf results ? those were kind of odd >>> too.. I thought with the mlx card I might be testing with PCIe >>> device .. but may I was hitting a limitation with iperf.. Also, >>> I tested the same kernel on Cavium system, and with >>> iommu.passthough=1/0 was I seeing a significant difference in the >>> iops with a file mounted to /dev/loop0. Its attached here.. On >>> Tue, May 9, 2017 at 8:52 PM, Nate Watterson >>> <[email protected]> wrote: Hi Manoj, Sorry for the late >>> reply. I finally found some time to test passthrough mode and >>> verified using the SMMU performance counters that the >>> ‘iommu.passthrough’ param is working as I would expect it to (1 >>> == Passthrough / 0 == non-Passthrough). I am not sure why you’re >>> seeing this strange behavior but based on the logs you’ve sent, I >>> have a few comments: Your boot log is showing that only the PCIe >>> SMMUs are enabled (this is good, it means your FW is recent). >>> Because of this, you will need to do your testing with a PCIe >>> attached device instead of the SD card or SSD (assuming it was >>> connected to the onboard SATA). I would recommend using an NVME >>> drive as the impact of enabling passthrough mode will only really >>> be apparent with devices capable of sustaining high IO throughput. >>> I would recommend changing your FIO profile to access the device >>> directly instead of going through the filesystem. Additionally, I >>> would recommend using a readonly profile without randomization to >>> preserve the disk and get consistent results. The following is the >>> FIO command that I use: fio --name=global --readonly >>> --group_reporting \ --direct=1 --ioengine=libaio --rw=read >>> --eta-newline=1s \ --size=1T --blocksize=512k --iodepth=32 >>> --numjobs=1 --runtime=10s \ --name=nvme_0 --filename=/dev/nvme0n1 >>> *** iommu.passthrough=1 *** READ: io=25301MB, aggrb=2528.3MB/s, >>> minb=2528.3MB/s, maxb=2528.3MB/s, mint=10007msec, maxt=10007msec >>> *** iommu.passthrough=0 *** READ: io=22657MB, aggrb=2265.5MB/s, >>> minb=2265.5MB/s, maxb=2265.5MB/s, mint=10001msec, maxt=10001msec >>> Hopefully this helps. Let me know if you have any more questions. >>> -Nate From: Manoj Iyer [mailto:[email protected]] Sent: >>> Thursday, May 4, 2017 4:55 PM To: Nate Watterson >>> <[email protected]> Cc: Timur Tabi <[email protected]> >>> Subject: RE: how to test iommu.passthough=0/1 ?? Nate, I am >>> attaching fio output for the CAF kernel that is available in the >>> centriq PPA. The results are not that much different. This time I >>> ran fio on the loop back disk and also on the mmcblk0 SD card. >>> Please see the attached test report. Thanks Manoj Iyer On Thu, >>> May 4, 2017 at 2:46 PM, Manoj Iyer <[email protected]> >>> wrote: I have attached the boot log from the system with >>> iommu.passthrough=1. Also I ran iperf on the mlx card >>> interface.. and again the results don't make sense. I was expecting >>> see better numbers with iommu.passthrough=1. Please see the >>> attached test data for UDP and TCP. Figuring out what firmware I >>> have was always a challenge. I had brought this up with the fw >>> folks that there is no way to say I am on version X Y or Z.. >>> anyways.. FW on this SDP: XBL.DF.2.0.R1-00406 QDF2400_REL CRM >>> Type: FF6D3C53-2748-4017-838F-07BACCC91341 Version: 0000016C >>> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx >>> Which I think translates to FW: commit >>> 891a97c41712112ce38ba70220c9ca85c9e52869 Author: QC Publisher >>> <[email protected]> Date: Mon Apr 17 13:02:52 2017 >>> -0700 Commit label r00364.1 - N/A 0.0.364.1 >>> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad On Thu, May 4, >>> 2017 at 2:21 PM, Nate Watterson <[email protected]> wrote: >>> Ok that makes a bit more sense. I still need the boot log though. >>> Is the USB device connected to one of the onboard USB controllers >>> or a PCI based USB controller? Do you happen to know what FW >>> version you’re running? -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 2:56 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? Opps my bad.. I am terribly sorry.. >>> /dev/sdd is a USB device. *-usb:2 description: >>> Mass storage device product: ADATA USB Flash Drive >>> vendor: ADATA description: SCSI Disk >>> physical id: 0.0.0 bus info: scsi@9:0.0.0 >>> logical name: /dev/sdd On Thu, May 4, 2017 at >>> 1:30 PM, Nate Watterson <[email protected]> wrote: Please >>> send the boot log. How is the SD card attached? I would have >>> expected it to show up as an mmcblk device but it appears to be >>> showing up at /dev/sdd. -Nate From: Nate Watterson Sent: >>> Thursday, May 4, 2017 2:12 PM To: 'Manoj Iyer' >>> <[email protected]> Cc: Timur Tabi <[email protected]> >>> Subject: RE: how to test iommu.passthough=0/1 ?? Some of those >>> results are shocking. I’ll try your FIO profile here a bit later >>> and let you know what I get. -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 1:00 >>> PM To: Nate Watterson <[email protected]> Cc: Timur Tabi >>> <[email protected]> Subject: RE: how to test >>> iommu.passthough=0/1 ?? I tried fio on a file on SSD mounted to >>> loopback device, and on the SD card on the AW SDP. I dont have any >>> additional disk on it so I am limited to those options. But the >>> results are a bit puzzling.. I see lower iops for >>> iommu.passthough=1. I have attached the test and output, may be I >>> am looking at this wrong... Any thoughts? On Thu, May 4, 2017 >>> at 10:25 AM, Nate Watterson <[email protected]> wrote: >>> Manoj, I don’t think there will be anything obvious in dmesg you >>> can check. You’ll pretty much have to run an IO benchmark (eg: >>> iperf/fio) in the *host* kernel on a device with sufficient IO >>> throughput to show the effects of the change. You should see a >>> significant performance benefit to having iommu.passthrough==1. >>> Note that this will have absolutely no impact at all on IO >>> performance within a VM. Let me know if you have any other >>> questions. -Nate From: Manoj Iyer >>> [mailto:[email protected]] Sent: Thursday, May 4, 2017 1:26 >>> AM To: Timur Tabi <[email protected]>; Nate Watterson >>> <[email protected]> Subject: how to test >>> iommu.passthough=0/1 ?? Nate/Timur, I backported the SMMU pass >>> though using default domain patch series from linux-next. I have a >>> public bug filed >>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158 to SRU >>> this patch series. But I cant figure out a way of testing >>> iommu.passthough=0/1. Could you pls suggest a test and tell me what >>> to look for dmesg etc to compare both cases? I understand it is >>> supposed to improve performance is there a way of checking that ? >>> I compared dmesg on both cases, and also launched vms and looked >>> to see if I can spot anything obvious.. Really appreciate your >>> help with suggestions on testing this. I need to post some testing >>> data with my SRU request. Thanks Manoj Iyer. >> -- ============================ Manoj Iyer Ubuntu/Canonical ARM >> Servers - Cloud ============================
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1688158 Title: [SRU][Zesty] Support SMMU passthrough using the default domain Status in linux package in Ubuntu: Incomplete Bug description: [Impact] Have the SMMU come up in a passthrough configuration, and then allow subsequent translation for things such as VFIO. Rather than do this in each SMMU driver, it's much cleaner to allow the default domain to be configured to be something other than DMA. This patch series implements a command-line option to configure the default domain type. Currently, it supports "dma" and "identity" which is sufficient for the passthrough use-case. [Fix] The following patch series in linux-next adds this support to the kernel. 4a8d8a14c0d0 arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA fccb4e3b8ab0 iommu: Allow default domain type to be set on the kernel command line beb3c6a066bf iommu/arm-smmu-v3: Install bypass STEs for IOMMU_DOMAIN_IDENTITY domains 67560edcd8e5 iommu/arm-smmu-v3: Make arm_smmu_install_ste_for_dev return void 61bc671179f1 iommu/arm-smmu: Install bypass S2CRs for IOMMU_DOMAIN_IDENTITY domains 0834cc28fa56 iommu/arm-smmu: Restrict domain attributes to UNMANAGED domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
