On Sun, 2021-01-10 at 13:08 +0100, Forza wrote: > > On 2021-01-10 12:52, David Woodhouse wrote: > > I migrated a system to btrfs which was hosting virtual machins with > > qemu. > > > > Using it without disabling copy-on-write was a mistake, of course, and > > it became horribly fragmented and slow. > > > > So I tried copying it to a new file... but it has actual *errors* too, > > which I think are because it was using the 'directsync' caching mode > > for block I/O in qemu. > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1204569#c12 > > > > I filed https://bugzilla.redhat.com/show_bug.cgi?id=1914433 > > > > What I see is that *both* disks of the RAID-1 have data which is > > consistent, and does not match the checksum that btrfs expects: > > > > [ 6827.513630] BTRFS warning (device sda3): csum failed root 5 ino 24387997 > > off 2935152640 csum 0x81529887 expected csum 0xb0093af0 mirror 1 > > [ 6827.517448] BTRFS error (device sda3): bdev /dev/sdb3 errs: wr 0, rd 0, > > flush 0, corrupt 8286, gen 0 > > [ 6827.527281] BTRFS warning (device sda3): csum failed root 5 ino 24387997 > > off 2935152640 csum 0x81529887 expected csum 0xb0093af0 mirror 2 > > [ 6827.530817] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 0, > > flush 0, corrupt 9115, gen 0 > > > > It looks like an O_DIRECT bug where the data *do* get updated without > > updating the checksum. Which is kind of the worst of both worlds, I > > suppose, since I also did get the appalling performance of COW and > > fragmentation. > > With O_DIRECT Btrfs shouldn't do checksum or compression. This is one of > the issues with Direct IO for the moment. But it should not cause those > dmesg errors. I believe it should show in scrub as no_csum:
It showed up as errors. There appears to be a btrfs bug there but since I suspect it'll be easy to reproduce I'm more focused on recovery right now. > > In the short term, all I want to do is make a copy of the file, using > > the data which are in the disk regardless of the fact that btrfs thinks > > the checksum doesn't match. Is there a way I can turn off *checking* of > > the checksum for that specific file (or file descriptor?). > > > > Or is the only way to do it with something like FIBMAP, reading the > > offending blocks directly from the underlying disk and then writing > > them into the appropriate offset in (a copy of) the file? A plan which > > is slightly complicated by the fact that of course btrfs doesn't > > support FIBMAP. > > > > What's the best way to recover the data? > > > > You can use GNU ddrescue to copy files. It can skip the offending blocks > and replace the bad data with zeroes. Not sure how well qemu will handle > that though. Right. I've already copied the image with dd conv=sync,noerror to a new one with the +C flag. It passes 'qemu-img check', and in fact the guest is running just fine with it. I was expecting it to stop with catastrophic file system errors but I can't see anything wrong at all. I'm just paranoid that eventually I'll find out that the blocks belong to some file(s) I actually want, and I'd like to recover them. Right now I have a horribly fragmented image file with these 'errors' cluttering up my file system and making backups of the host go extremely slow. I'd like to get those blocks back so I can make a clean copy of the image, and keep it around for reference in case I later *do* discover that I need the contents of those blocks.
smime.p7s
Description: S/MIME cryptographic signature
