On 2018-02-15 10:47, Qu Wenruo wrote:
On 2018年02月14日 22:19, Tomasz Chmielewski wrote:
Just FYI, how dangerous running btrfs can be - we had a fatal,
unrecoverable MySQL corruption when btrfs decided to do one of these
"I
have ~50 GB left, so let's do out of space (and corrupt some files at
the same time, ha ha!)".
I'm recently looking into unexpected corruption problem of btrfs.
Would you please provide some extra info about how the corruption
happened?
1) Is there any power reset?
Btrfs should be bullet proof, but in fact it's not, so I'm here to
get some clue.
No power reset.
2) Are MySQL files set with nodatacow?
If so, data corruption is more or less expected, but should be
handled by checkpoint of MySQL.
Yes, MySQL files were using "nodatacow".
I've seen many cases of "filesystem full" with ext4, but none lead to
database corruption (i.e. the database would always recover after
releasing some space)
On the other hand, I've seen a handful of "out of space" with gigabytes
of free space with btrfs, which lead to some light, heavy or
unrecoverable MySQL or mongo corruption.
Can it be because of of how "predictable" out of space situations are
with btrfs and other filesystems?
- in short, ext4 will report out of space when there is 0 bytes left
(perhaps slightly faster for non-root users) - the application trying to
write data will see "out of space" at some point, and it can stay like
this for hours (i.e. until some data is removed manually)
- on the other hand, btrfs can report out of space when there is still
10, 50 or 100 GB left, meaning, any capacity planning is close to
impossible; also, the application trying to write data can be seeing the
fs as transitioning between "out of space" and "data written
successfully" many times per minute/second?
3) Is the filesystem metadata corrupted? (AKA, btrfs check report
error)
If so, that should be the problem I'm looking into.
I don't think so, there are no scary things in dmesg. However, I didn't
unmount the filesystem to run btrfs check.
4) Metadata/data ratio?
"btrfs fi usage" could have quite good result about it.
And "btrfs fi df" also helps.
Here it is - however, that's after removing some 80 GB data, so most
likely doesn't reflect when the failure happened.
# btrfs fi usage /var/lib/lxd
Overall:
Device size: 846.25GiB
Device allocated: 840.05GiB
Device unallocated: 6.20GiB
Device missing: 0.00B
Used: 498.26GiB
Free (estimated): 167.96GiB (min: 167.96GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID1: Size:411.00GiB, Used:246.14GiB
/dev/sda3 411.00GiB
/dev/sdb3 411.00GiB
Metadata,RAID1: Size:9.00GiB, Used:2.99GiB
/dev/sda3 9.00GiB
/dev/sdb3 9.00GiB
System,RAID1: Size:32.00MiB, Used:80.00KiB
/dev/sda3 32.00MiB
/dev/sdb3 32.00MiB
Unallocated:
/dev/sda3 3.10GiB
/dev/sdb3 3.10GiB
# btrfs fi df /var/lib/lxd
Data, RAID1: total=411.00GiB, used=246.15GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=9.00GiB, used=2.99GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /var/lib/lxd
Label: 'btrfs' uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
Total devices 2 FS bytes used 249.15GiB
devid 1 size 423.13GiB used 420.03GiB path /dev/sda3
devid 2 size 423.13GiB used 420.03GiB path /dev/sdb3
Tomasz Chmielewski
https://lxadm.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
