> On Feb 4, 2019, at 2:13 PM, Josh Paetzel <[email protected]> wrote: > > Thanks for the detailed writeup. > > ZFS by default limits I/O to block devices in an attempt to not tip them > over. This works pretty well for spinning rust, but it's somewhat limiting > for NVMe drives. > > On FreeBSD anyways the following sysctls control the throttling: > > vfs.zfs.vdev.async_write_max_active: 10 > vfs.zfs.vdev.async_write_min_active: 1 > vfs.zfs.vdev.async_read_max_active: 3 > vfs.zfs.vdev.async_read_min_active: 1 > vfs.zfs.vdev.sync_write_max_active: 10 > vfs.zfs.vdev.sync_write_min_active: 10 > vfs.zfs.vdev.sync_read_max_active: 10 > vfs.zfs.vdev.sync_read_min_active: 10 > vfs.zfs.top_maxinflight: 32 > > In your case with 4 vdevs doing a read test the system was only dispatching > 12 I/Os simultaneously. (regardless of how many userland threads you had > running) > > I'd try with async_read set to 400 and maxinflight set to 100 and see what > happens. My suspicion is those values will be too low, but the trick is to > keep increasing them until performance plateaus and latency starts to climb. > I'm guessing you know the drill. (That was a pretty good writeup of your > testing)
async reads are prefetch. So you'll also want to look at sync_read_max_active for on-demand reads. See: https://github.com/zfsonlinux/zfs/wiki/ZIO-Scheduler <https://github.com/zfsonlinux/zfs/wiki/ZIO-Scheduler> https://github.com/zfsonlinux/zfs/wiki/ZFS-on-Linux-Module-Parameters#zio_scheduler <https://github.com/zfsonlinux/zfs/wiki/ZFS-on-Linux-Module-Parameters#zio_scheduler> To see the ZIO scheduler details in action, for ZFS on Linux CLI use "zpool iostat -q [interval]" And if you want to collect those queueing stats into a TSDB see https://github.com/richardelling/zpool_influxdb <https://github.com/richardelling/zpool_influxdb> or https://github.com/richardelling/zpool_prometheus <https://github.com/richardelling/zpool_prometheus> -- richard > > > On Mon, Feb 4, 2019, at 3:48 PM, batkinson via openzfs-developer wrote: >> We are currently seeing poor read performance with ZFS 0.7.12 with our >> Samsung PM1725a devices in a Dell PowerEdge R7425. Briefly describing our >> setup, we currently have four PM1724a devices attached to the PCIe root >> complex in NUMA domain 1 on an AMD EPYC 7401 processor. In order to measure >> read throughput of ZFS, XFS, and the Raw Devices, the XDD tool was used >> which is available at: >> >> >> >> git at github.com >> <http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-devel>:bwatkinson/xdd.git >> >> >> >> In all cases I am presenting, kernel 4.18.20-100 was used and I disabled all >> CPU's not on socket 0 within the kernel. I also issued asynchronous >> sequential reads to the file systems/devices while pinning all XDD threads >> to NUMA domain 1 and Socket 0's memory banks. I conducted four tests >> consisting of measuring throughput for the raw devices, XFS, and ZFS 0.7.12. >> For the Raw Device tests, I had 6 I/O threads per devices with a request >> sizes of 1 MB and a total of 32 GB read from each device using Direct I/O. >> In the XFS case, I created a single XFS file system on each of the 4 >> devices. In each of the XFS file systems, I read a 32 GB file of random data >> using 6 I/O threads per file with request sizes of 1 MB using Direct I/O. In >> the ZFS Single ZPool case I created a single ZPool composed of 4 VDEVs and >> read 128 GB file of random data using 24 I/O threads with request sizes of 1 >> MB. In the ZFS Multiple ZPool case I create 4 separate ZPools each >> consisting of a single VDEV. In each of the ZPools, I read a 32 GB file of >> random data using 6 I/O threads per file with request sizes of 1 MB. In both >> the Single ZPool and Multipl ZPool cases I set the record sizes for all >> pools to 1 MB and I set the primarycache=none. We decided to disable the ARC >> in all cases, because we were reading 128 GB of data, which was exactly >> equal to 2x available memory on Socket 0. Even with the ARC enabled we were >> seeing no performance benefits. Below are the throughput measurements I >> collected for each of these case. >> >> >> >> Performance Results: >> >> Raw Device - 12,7246.724 MB/s >> >> XFS Direct I/O - 12,734.633 MB/s >> >> ZFS Single Zpool - 6,452.946 MB/s >> >> ZFS Multiple Zpool – 6,344.915 MB/s >> >> >> >> In order to try and solve what was cutting ZFS read performance in half, I >> generated flame graphs using the following tool: >> >> >> >> http://www.brendangregg.com/flamegraphs.html >> <http://www.brendangregg.com/flamegraphs.html> >> >> >> >> In general I found most of the perf samples were occurring in zio_wait. It >> is in this call that io_scheudle is called. Comparing the ZFS flame graphs >> to XFS flame graphs, I found that the number of samples between the >> submit_bio and io_schedule was significantly larger in the ZFS case. I >> decided to take timestamps of each call to io_schedule for both ZFS and XFS >> to measure the latency between the calls. Below is a link to to histograms >> as well as total elapsed time in microseconds between io_schedule calls for >> the tests I described above. In total I collected 110,000 timestamps. In >> plotted data the first 10,000 timestamps were ignored to allow for the file >> systems to reach a steady state. >> >> >> https://drive.google.com/drive/folders/1t7tLXxyhurcxGjry4WYkw_wryh6eMqdh?usp=sharing >> >> <https://drive.google.com/drive/folders/1t7tLXxyhurcxGjry4WYkw_wryh6eMqdh?usp=sharing> >> >> >> In general, ZFS has a significant latency between io_scheudule calls. I have >> also verified that the output from iostat shows a larger await_r value for >> ZFS over XFS for these tests as well. In general it seems ZFS is letting >> request sit in the hardware queues longer than XFS and the Raw Devices >> causing a huge performance penalty in ZFS reads (effectively cutting the >> available device bandwidth in half). >> >> >> >> In general, has this issue been noticed with NVMe SSD’s and ZFS and is there >> a current fix to this issue? If there is no current fix, is this issue being >> worked on? >> >> >> >> Also, I have tried to duplicate the XFS results using ZFS 0.8.0-rc2 using >> Direct I/O, but the performance for 0.8.0 read almost exactly matched ZFS >> 0.7.12 read performance without Direct I/O. >> >> openzfs <https://openzfs.topicbox.com/latest> / openzfs-developer / see >> discussions <https://openzfs.topicbox.com/groups/developer> + participants >> <https://openzfs.topicbox.com/groups/developer/members> + delivery options >> <https://openzfs.topicbox.com/groups/developer/subscription> Permalink >> <https://openzfs.topicbox.com/groups/developer/Td9e139e299dd9e7c-M4812e00392b4956ef5daa674> > > -- > > Thanks, > > Josh Paetzel > > > openzfs <https://openzfs.topicbox.com/latest> / openzfs-developer / see > discussions <https://openzfs.topicbox.com/groups/developer> + participants > <https://openzfs.topicbox.com/groups/developer/members> + delivery options > <https://openzfs.topicbox.com/groups/developer/subscription>Permalink > <https://openzfs.topicbox.com/groups/developer/Td9e139e299dd9e7c-M396cc7ff279859e24881e121> ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Td9e139e299dd9e7c-Mac5e713508aa0d5a4414d7ed Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
