Hi,
I have a BTRFS filesystem that seems to have become very ill. After 4 hours of
being mounted, it will fail with every write attempt saying "No space left on
device".
Unmounting and remounting the filesystem clears the issue for another 4 hours
>From every check I have done, no messages are logged at the point of the
>failure to "dmesg" or any system log.
I'm over 99% sure there is not a space issue on the filesystem - it has over
100GB free, and I've run a full "balance" which has not changed the behaviour.
A "scrub" on the filesystem hasn't reported any issues.
The output of the three (why on earth are there three?) disk space commands on
the filesystem are:
--------------------------------------------------------------------------------------------------------------------------------------
$ sudo btrfs filesystem usage /home
Overall:
Device size: 450.00GiB
Device allocated: 319.06GiB
Device unallocated: 130.94GiB
Device missing: 0.00B
Used: 305.95GiB
Free (estimated): 131.77GiB (min: 66.30GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:299.00GiB, Used:298.16GiB
/dev/mapper/VG-HomeVol 299.00GiB
Metadata,DUP: Size:10.00GiB, Used:3.89GiB
/dev/mapper/VG-HomeVol 20.00GiB
System,DUP: Size:32.00MiB, Used:80.00KiB
/dev/mapper/VG-HomeVol 64.00MiB
Unallocated:
/dev/mapper/VG-HomeVol 130.94GiB
$ sudo btrfs filesystem df /home
Data, single: total=299.00GiB, used=298.16GiB
System, DUP: total=32.00MiB, used=80.00KiB
Metadata, DUP: total=10.00GiB, used=3.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
$ sudo btrfs filesystem show /home
Label: none uuid: 550e6e7c-d669-4128-9b0d-b61ef4f3f1c1
Total devices 1 FS bytes used 302.07GiB
devid 1 size 450.00GiB used 319.06GiB path
/dev/mapper/VG-HomeVol
--------------------------------------------------------------------------------------------------------------------------------------
>From my understanding of the output in this, there don't seem to be any areas
>that are even close to full. And if it was a genuine full condition, even due
>to running out of metadata or something, then I wouldn't expect unmounting and
>remounting to clear the issue.
Is there any known issue that may cause this behaviour?
Is there any way to get more debugging from what is going on?
My initial thought was that it might be related to snapshots, as I was
generating regular snapshots (for a 'previous versions' feature), and many of
the failures were just after a snapshot was created. However, I have now
disabled the snapshot creation and I am still seeing regular failures.
The system is running stock Debian 9 (Stretch). It was running their latest 4.9
kernel (Rev 4.9.144-3.1) when the problem first occurred. After two instances
of the problem, I rolled back to their previous kernel (Rev 4.9.130-2), which
the system had been running error free for several months, but the failures
have continued.
I'm happy to get any other information that would be needed to debug this, if
someone can point me to how to do it.
Currently my faith in BTRFS is approaching zero (it was knocked after a data
loss in October, but had grown again). It has a lot of nice features, but
(despite comments on the Wiki) really does not seem stable, at least not in
Debian.
Thanks
Michael
