Josh and Richard, Thank you for your replies. I am current using a CentOS, so I adjusted the parameters under /sys/module/zfs/parameters. I incrementally increased the recommend values for the parameters to the following, but I seeing only a minimal read performance increase from the 4 ZPool's with a consisting of a single VDEV setup:
zfs_vdev_async_read_max_active = 2048 zfs_vdev_sync_read_max_active = 2048 zfs_top_maxinflight = 1024 Even with these values set, I get just around 7.1 MB/s. This is well below the total aggregate device rate of 12.3 MB/s for all four PM1725a NVMe devices for sequential reads. Maybe these values should be increased further, but I imagine that seems like a rather large value for these parameters (I may be completely wrong though). Monitoring the queues with `zpool iostat -q 1` I only see the syncq_read activ value fluctuate between 3-6 with zero pending operations. XDD issues reads using pread's, so I imagine that is why the read requests are landing in the sync read queues. I tried this both with the ARC disabled and enabled, but the values remain the same. Is there possibly other ZFS parameters/settings that might be hindering us from achieving close to device line rate? Depending on your feedback, I can again collect latencies between io_schedule calls with these updated parameter values to see if that still is the lingering issue. ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Td9e139e299dd9e7c-Mfa2eb7757e84bae7982f8be9 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
