[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-23 Thread Tyson Key
Thanks, Colin. I intend to retest, with 20.10, but for now, with deduplication disabled, things seem stable. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1899249 Title: OpenZFS

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-14 Thread Tyson Key
And, after 18 hours+, I was able to archive a ZFS snapshot, using BorgBackup, with the WBT settings, suggested, earlier: tyson@ubuntu:~$ sudo zfs send Yaesu@Crucial-2TB-1951E22FA633 | borg create --stats BorgStore::Yaesu@Crucial-2TB-1951E22FA633 -

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
Needs more time, in terms of long-running I/O, but changing the WBT setting at least makes retrieving snapshots a little faster, with "zfs list -t all". -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu.

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
Set the WBT value to 0, for all of the devices, whilst another archive run takes place. The last one ran for just over 20 hours, before hitting the I/O problem. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu.

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
All of the drives in the pool are SSDs (2x Crucial BX500 (2TB), 1x SanDisk Ultra 3D (4TB), 1x Samsung 870 QVO (4TB), but obviously don't perfectly-align, in terms of performance characteristics - but, I'll test the WBT tuneable, to see if it makes a difference. I'm also suspecting that some power

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
Here's a full "dmesg" log, if it helps: tyson@ubuntu:~$ dmesg [0.00] Booting Linux on physical CPU 0x00 [0x410fd083] [0.00] Linux version 5.4.0-1018-raspi (buildd@bos02-arm64-052) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #20-Ubuntu SMP Sun Sep 6 05:11:16 UTC 2020

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
I'm using 16GB of swap, on a ZVOL, in the pool, but I can also test with swap, on the internal MicroSD card's EXT4 root partition, if it helps. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu.

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-13 Thread Tyson Key
Looks like I could archive about 1.3TB, of a 1.64TB snapshot, before things started to go bad, again: [110079.681102] INFO: task z_wr_iss_h:2171 blocked for more than 120 seconds. [110079.688123] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [110079.695167] "echo 0 >

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-12 Thread Tyson Key
Thanks. I was able to archive, and delete 400GB of data, without problems, earlier on, today, which reduced the "REFER" of my data set, a little - however, it looks like I probably need to focus on archiving, and removing some of the older snapshots, if I want to trim down the memory utilisation

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-12 Thread Tyson Key
Unfortunately, I haven't got a storage device large enough, to contain all of the data from the pool, and much of it cannot be recreated, or restored, from another source, so I won't be able to nuke the pool, and rebuild it. -- You received this bug notification because you are a member of

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-12 Thread Tyson Key
Also receive this, when trying to archive a large directory: [ 2055.147509] INFO: task z_wr_iss_h:2169 blocked for more than 120 seconds. [ 2055.154450] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [ 2055.161401] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-11 Thread Tyson Key
With deduplication disabled, it eventually gets further with the backup, but still stalls, after leaving it, overnight: [12084.274242] INFO: task z_wr_iss_h:2157 blocked for more than 120 seconds. [12084.281171] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.288126] "echo 0

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-11 Thread Tyson Key
In the meantime, I'll see if I can temporarily disable ZPool-level deduplication, and retry the backup run, again, on the Windows machine. Whilst it's not the perfect long-term solution, I might look into using offline deduplication, for older, infrequently-accessed data in the pool (probably

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-11 Thread Tyson Key
It looks like writing snapshots, with "zfs snapshot" will sometimes stall, even if other commands, like "zfs status" appear to work, occasionally, too. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu.

[Kernel-packages] [Bug 1899249] Re: OpenZFS writing stalls, under load

2020-10-11 Thread Tyson Key
Thanks for your reply (I expected this to go into the ether, since it seems to be a very common issue, with I/O, on this hardware platform). I did try switching from LZJB, to LZ4, and got slightly-better performance, and reliability, from at least testing with backing up a Windows 10 machine,

[Kernel-packages] [Bug 1899249] [NEW] OpenZFS writing stalls, under load

2020-10-10 Thread Tyson Key
Public bug reported: Using a QNAP 4-drive USB enclosure, with a set of SSDs, on a Raspberry Pi 8GB. ZFS deduplication, and LZJB compression is enabled. This issue seems to occur, intermittently, after some time (happens with both SMB access, via Samba, and when interacting with the system, via