Steps to reproduce
Simply copying a file into the file system can cause things to lock up. In this
case, the files will usually be thin-provisioned qcow2 disks for kvm vm's.
There is no detailed formula to force the lockup to occur, but it happens
regularly, sometimes multiple times in one day.
Files are often copied from a master by reference (cp --reflink), one per day
to perform a daily backup for up to 45 days. Removing older files is a
painfully slow process, even though there are only 45 files in the directory.
Doing a scrub is almost a sure way to lock up the system, especially if a copy
or delete operation is in progress. On two systems, crashes occur with 4.18 and
4.19 but not 4.17. On the other systems that crash, it does not seem to matter
if it is 4.17, 4.18 or 4.19.
Unless otherwise indicated
using qgroups: No
using compression: Yes, compress-force=zstd
number of snapshots: Zero
number of subvolumes: top level subvolume only
raid profile: None
using bcache: No
layered on md or lvm: No
vhost002
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data/datastore2 btrfs
rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
# smartctl -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
vhost003
# grep btrfs /etc/mtab
/dev/sdb4 /usr/local/data/datastore2 btrfs
rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
(RAID controller)
# smartctl -l scterc /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.17.0-0.bpo.1-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdb4].write_io_errs 0
[/dev/sdb4].read_io_errs 0
[/dev/sdb4].flush_io_errs 0
[/dev/sdb4].corruption_errs 0
[/dev/sdb4].generation_errs 0
vhost004
# grep btrfs /etc/mtab
/dev/sdb4 /usr/local/data/datastore2 btrfs
rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
(RAID controller)
# smartctl -l scterc /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.17.0-0.bpo.1-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, [ www.smartmontools.org
]( http://www.smartmontools.org )
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdb4].write_io_errs 0
[/dev/sdb4].read_io_errs 0
[/dev/sdb4].flush_io_errs 0
[/dev/sdb4].corruption_errs 0
[/dev/sdb4].generation_errs 0
vhost031
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data/datastore2 btrfs
rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
# smartctl -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
vhost032
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data/datastore2 btrfs
rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
# smartctl -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
vhost182
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data/datastore2 btrfs
rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
# smartctl -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
vhost241
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data/datastore2 btrfs
rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
# smartctl -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.17.0-0.bpo.1-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
# btrfs dev stats /usr/local/data/datastore2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
lxc008
number of subvolumes: 1416
# grep btrfs /etc/mtab
/dev/sdc1 /usr/local/data2 btrfs
rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
(RAID controller)
# smartctl -d megaraid,0 -l scterc /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Write SCT (Get) Error Recovery Control Command failed: ATA return descriptor
not supported by controller firmware
SCT (Get) Error Recovery Control command failed
# btrfs dev stats /usr/local/data2
[/dev/sdc1].write_io_errs 0
[/dev/sdc1].read_io_errs 0
[/dev/sdc1].flush_io_errs 0
[/dev/sdc1].corruption_errs 0
[/dev/sdc1].generation_errs 0
lxc009
# grep btrfs /etc/mtab
/dev/sda3 / btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
/dev/sdb1 /usr/local/data2 btrfs
rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
(RAID controller)
# smartctl -l scterc /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, [ www.smartmontools.org
]( http://www.smartmontools.org )
# btrfs dev stats /usr/local/data2
[/dev/sdb1].write_io_errs 0
[/dev/sdb1].read_io_errs 0
[/dev/sdb1].flush_io_errs 0
[/dev/sdb1].corruption_errs 0
[/dev/sdb1].generation_errs 0
