[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
And if anyone is wondering why this is not the default, the answer is here: https://github.com/zfsonlinux/zfs/issues/443 ** Bug watch added: Github Issue Tracker for ZFS #443 https://github.com/zfsonlinux/zfs/issues/443 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Ah, that is a good solution. By default, ZFS on Linux will store xattrs in a hidden folder, so there are multiple seeks per xattr and it's poorly cached, so it will impact on performance especially on the slower block devices. The zfs set xattr=sa will store this in the inodes and will really help. Let's mark this as a suitable workaround and hence a fix. ** Changed in: zfs-linux (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, I set a couple of ZFS options a day or two ago, and have bee playing videos to see if we still have the problem. The options I set were: o zfs set xattr=sa storage o zfs set xattr=sa storage/vault o zfs set ralatime=on storage o zfs set ralatime=on storage/vault We've played about four hours of video, and we have not seen the hang. Do you still want to pursue this? I could set the options back and do the trace or leave the options and do the trace, or should we just consider this fixed? My only concern about the latter option is that the installation was a default install with Ubuntu 17.10 and it seems these options should be the default on any linux distro. Let me know what you want to do? Best regards, Colin. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
I've tried to simulate this with very slow media and can't yet reproduce this issue. Can you play the DVD image and also run the following command in another terminal: iostat 1 300 > iostat.log this will capture 5 minutes of activity. please then attach the log to the bug report. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
root@hetty:/home/crlb# blockdev --getpbsz /dev/sdb 512 root@hetty:/home/crlb# blockdev --getpbsz /dev/sdc 512 root@hetty:/home/crlb# -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Just one more check, what is the block size of the raw devices? One can check this using: sudo blockdev --getpbsz /dev/sdb sudo blockdev --getpbsz /dev/sdc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, I have a couple more screen shots showing the raw performance of both drives; the first using 512MiB as you suggested and the second with a larger sample and using the dd commands status=progress option to monitor for pauses. Also in the second ss, I performed the test on an iso file from the zfs file system. I did not see any pauses on the raw devices, but then I only saw momentary pauses (one second or less) on the zfs file. Regards, Colin. ** Attachment added: "raw device performance" https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5114647/+files/dd-1.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
** Attachment added: "raw device and zfs performance on a larger sample." https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5114648/+files/dd-2.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
The iostat -mx shows that the /dev/sdb /dev/sdc devices have a very high read I/O latency of ~80-110 milliseconds which may be the underlying issue. I suggest seeing how long it takes to read data from these raw devices to sanity check their maximum speed. One can do this by doing: sync echo 3 | sudo tee /proc/sys/vm/drop_caches dd if=/dev/sdb of=/dev/null bs=512 count=1048576 ..and wait for 512MB of data to be read and see what the read speed is of the raw device. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, Thank you for responding. To Answer your questions: 1. I have two external drives, sdb and sdc, in USB 3 Orico enclosure. The drives are mirrored. 2. Atom D525 with 4GiB RAM. The attached screenshot shows "vmstat 5 5" output while VLC is playing a movie. It shows 1.5GiB of free memory, no swap used. 3. Screenshot also shows output from "iostat -mx", and that I manually set the spl_kmem_cache_slab_limit to zero as suggested. I played a movie which paused repeatedly for 30 seconds and one time for several minutes. When the pause occurs, VLC is completely hung until the requested I/O completes. The end of pause is heralded by the I/O activity LED on the drive enclosure. I want to emphasize/re-iterate a couple of points from my original post: 1. This is not a VLC problem. I can recreate the problem with any app (cp, rsyn, etc) that reads sequentially from the ZFS file system. 2. I do not believe this is a drive/latency problem: I can dd the content of sdb and sdc (to /dev/null) without any hang. Regards, Colin. ** Attachment added: "spl_kmem_cache_slab_limit, vmstat, and iostat -mx" https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5113194/+files/spl.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
1. what is your ZFS configuration (number of drives?) 2. size of your system (total memory, free memory). 3. Do your drive(s) suffer from large I/O latencies? - what is the output from iostat -mx You may possibly get some benefit from setting spl_kmem_cache_slab_limit to zero as a module parameter for the spl ZoL kernel module. This avoids SLUB slab merging entirely and maybe helpful from anecdotal evidence I've seen. ** Changed in: zfs-linux (Ubuntu) Status: New => Triaged ** Changed in: zfs-linux (Ubuntu) Importance: Undecided => Medium ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu) Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Just updated to: root@hetty:~# apt list --installed | awk '/zpool|zfs/' libzfs2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic] libzpool2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed] zfs-zed/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic] zfsutils-linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed] root@hetty:~# A little better, but the problem still happens. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
