Re: Force recalculation of a data block checksum
On Sat, Jul 23, 2016 at 10:32 PM, Tomasz Melcer wrote: > On 24.07.2016 04:16, Chris Murphy wrote: >> >> On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer wrote: >>> >>> How can I ask btrfs to recompute the checksum of a data block as it is >>> stored on the drive? >> >> >> Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a >> whole new csum tree. > > > I can't make this command recalculate only checksum of that single file? The likelihood of the data block being wrong is very high. If the csum itself were wrong, the csum of the leaf it's stored in would catch it, except in the narrow edge case where the csum was computed wrong in memory, and then put into the leaf which is then checksummed correctly, i.e. the leaf csum says the leaf is good but it contains a bad csum for just one block. Possible but, pretty rare if there isn't a problem causing in-memory corruption. You'd see a lot more errors. A few bad sectors are exactly what this most likely is. The drive ECC is probably failing to correct the problem, but thinks it's right which why there's no read error from the drive. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Force recalculation of a data block checksum
24.07.2016 07:32, Tomasz Melcer пишет: > On 24.07.2016 04:16, Chris Murphy wrote: >> On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer wrote: >>> How can I ask btrfs to recompute the checksum of a data block as it is >>> stored on the drive? >> >> Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a >> whole new csum tree. > > I can't make this command recalculate only checksum of that single file? > > Rewriting single block in question should do it. As you can read from raw device and have both physical and logical block address something like dd if=/dev/sdd1 skip=222940168 count=8 | dd of=/path/to/dysk/dysk.bin seek=189669768 conv=notrunc count=8 If you have possibility to verify file content(s), this technique can be used to read over unreadable parts of file by combining multiple dd's from file and device. It probably makes sense to create (read-only) snapshot to make sure you have old content available. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Force recalculation of a data block checksum
On 24.07.2016 04:16, Chris Murphy wrote: On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer wrote: How can I ask btrfs to recompute the checksum of a data block as it is stored on the drive? Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a whole new csum tree. I can't make this command recalculate only checksum of that single file? -- Tomasz Melcer -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Force recalculation of a data block checksum
The negative of --init-csum-tree is that it's going to work on all data extents. It's going to take a while. It's probably faster to extract the suspect file with restore onto another file system and check its integrity by other means (compare to backup, database integrity check, etc). -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Force recalculation of a data block checksum
Tomasz Melcer posted on Sun, 24 Jul 2016 04:03:08 +0200 as excerpted: > How can I ask btrfs to recompute the checksum of a data block as it is > stored on the drive? I don't see any command doing an operation like > that, and I couldn't find anything on the topic on the internet. There's no "easy" btrfs method to recompute the checksum of an individual block (you'd have to do some deep debugging and hex editing to manually rewrite the correct sum), but there /is/ one that reinitiates and recalculates checksums for the entire filesystem. You don't mention what version of btrfs-progs you have, but you'll want a reasonably new one, as AFAIK the command is fairly new, and early versions of the command simply wiped the existing checksums without recalculating them. btrfs check --init-csum-tree ... as found in the btrfs-check manpage. Another alternative is using btrfs restore, again, see the btrfs-restore manpage. Normally this command is used to get what you can off of a filesystem damaged badly enough it won't mount, but you can also use it to scrape specific files off an unmounted filesystem, if the checksum's bad, for instance. In this case, you'd use the --path-regex option to restore only files matching that regex. Of course, for a TB-sized file, you'll need a separate filesystem (need not be btrfs, but will need to be a filesystem that can handle a file that size) with at least a TB free to write it to. I'd normally prefer the btrfs restore method, myself, particularly if the btrfs in question has other files that you don't want to risk in the process. If that was the only file on the filesystem, I'd still try the restore method first, at least if I had some place to write it to, because btrfs restore is read-only against the btrfs you're restoring from, and thus has no chance of causing further damage, while of course btrfs check --init-csum-tree writes to the filesystem in question and thus is a higher risk. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Force recalculation of a data block checksum
On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer wrote: > Hi, > > I've got a USB-connected HDD with a btrfs partition. The partition contains > a 1TB file, a disk image. The first `btrfs scrub` after writing that file > found 3 logical bad blocks that developed somewhere in the middle of that > file (logs below). > > The full area of the btrfs partition can be read without I/O error, so I > think there are two possible cases: either the data block was written > incorrectly or an incorrect checksum is stored. The first case is obviously > unrecoverable, but if it's the second case, fixing the problem should be as > simple as recomputing the checksum for what is already stored. > > How can I ask btrfs to recompute the checksum of a data block as it is > stored on the drive? I don't see any command doing an operation like that, > and I couldn't find anything on the topic on the internet. Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a whole new csum tree. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Force recalculation of a data block checksum
Hi, I've got a USB-connected HDD with a btrfs partition. The partition contains a 1TB file, a disk image. The first `btrfs scrub` after writing that file found 3 logical bad blocks that developed somewhere in the middle of that file (logs below). The full area of the btrfs partition can be read without I/O error, so I think there are two possible cases: either the data block was written incorrectly or an incorrect checksum is stored. The first case is obviously unrecoverable, but if it's the second case, fixing the problem should be as simple as recomputing the checksum for what is already stored. How can I ask btrfs to recompute the checksum of a data block as it is stored on the drive? I don't see any command doing an operation like that, and I couldn't find anything on the topic on the internet. Thanks, Logs: #v+ [ 7702.964265] BTRFS warning (device sdd1): checksum error at logical 5473719291904 on dev /dev/sdd1, sector 222940168, root 5, inode 1245769, offset 97110921216, length 4096, links 1 (path: dysk/dysk.bin) [ 7702.964274] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 17, gen 0 [ 7702.964278] BTRFS error (device sdd1): unable to fixup (regular) error at logical 5473719291904 on dev /dev/sdd1 […] [ 9588.625906] BTRFS warning (device sdd1): checksum error at logical 5241172611072 on dev /dev/sdd1, sector 691494312, root 5, inode 1245769, offset 310632271872, length 4096, links 1 (path: dysk/dysk.bin) [ 9588.625916] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 18, gen 0 [ 9588.625920] BTRFS error (device sdd1): unable to fixup (regular) error at logical 5241172611072 on dev /dev/sdd1 […] [20545.302898] BTRFS warning (device sdd1): checksum error at logical 3991747321856 on dev /dev/sdd1, sector 3185809200, root 5, inode 1245769, offset 907925676032, length 4096, links 1 (path: dysk/dysk.bin) [20545.302908] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 19, gen 0 [20545.302912] BTRFS error (device sdd1): unable to fixup (regular) error at logical 3991747321856 on dev /dev/sdd1 #v- -- Tomasz Melcer -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Chances to recover with bad partition table?
On Sat, Jul 23, 2016 at 7:46 PM, Chris Murphy wrote: > Something like this: > [root@f24s ~]# dd if=/dev/sda | hexdump -C | egrep '5f 42 48 52 66 53 5f' > 00110040 5f 42 48 52 66 53 5f 4d 8d 4f 04 00 00 00 00 00 |_BHRfS_M.O..| Ha so originally I was planning on putting in the dd portion a count limit, like count=20480 for a 10MiB search from the start of the drive. You could just do # hexdump -C /dev/sda | egrep '5f 42 48 52 66 53 5f' And then control-C to cancel it once the first super shows up. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Chances to recover with bad partition table?
On Sat, Jul 23, 2016 at 2:37 AM, Hendrik Friedel wrote: > Hello, > > this morning I had to face an unusual prompt on my machine. > > I found that the partition table of /dev/sda had vanished. > > I restored it with testdisk. It found one partition, but I am quite sure > there was a /boot partition in front of that which was not found. > > Now, running btrfsck fails: > > root@homeserver:~# fdisk -l /dev/sda > > WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk > doesn't support GPT. Use GNU Parted. > > > Disk /dev/sda: 120.0 GB, 120034123776 bytes > 255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x > >Device Boot Start End Blocks Id System > /dev/sda1 * 1026048 234440703 116707328 83 Linux > root@homeserver:~# btrfsck /dev/sda1 > checksum verify failed on 20987904 found E4E3BDB6 wanted > checksum verify failed on 20987904 found E4E3BDB6 wanted > checksum verify failed on 20987904 found E4E3BDB6 wanted > checksum verify failed on 20987904 found E4E3BDB6 wanted > bytenr mismatch, want=20987904, have=0 > Couldn't read chunk root > Couldn't open file system > > Is there a way to let btrfs search for the start of the partiton? > > I do have a backup; thus it is not fatal. But some data on the disk is more > recent than my back up (of course) Something like this: [root@f24s ~]# dd if=/dev/sda | hexdump -C | egrep '5f 42 48 52 66 53 5f' 00110040 5f 42 48 52 66 53 5f 4d 8d 4f 04 00 00 00 00 00 |_BHRfS_M.O..| 04100040 5f 42 48 52 66 53 5f 4d 8d 4f 04 00 00 00 00 00 |_BHRfS_M.O..| Since the magic appears 0x40 after the start of the super, and there's a 64KiB, or 0x1, pad for a bootloader between the start of the partition and the start of the superblock, subtract 0x10040 to find the start of the partition in bytes. 0x110040 - 0x10040 = 0x10 = 1048576 Divide by 512 to find the LBA to start the partition at 1048576 / 512 = 2048 And in fact that's what my partition is set to start at... [root@f24s ~]# gdisk -l /dev/sda GPT fdisk (gdisk) version 1.0.1 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/sda: 1953525168 sectors, 931.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 3D9880B5-D4DF-41F8-9EFF-40155A1D3DD5 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 1953525134 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector)End (sector) Size Code Name 12048 377489407 180.0 GiB 8300 fedora-root Note that Btrfs silently tolerates the end of the partition being either shorter or longer than the file system claims it is, so to find the proper ending you'll have to first guess, so that you can point btrfs-show-super at the newly established partition, and then look for dev_item.total_bytes. In my case: dev_item.total_bytes193273528320 193273528320 / 512 = 377487360 377487360 + 2048 - 1 = 377489407 -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: qcow2 becomes 37P in size while qemu crashes
Kindof a rudimentary way of making a debug tree smaller, by filtering by inode... [root@f24m images]# btrfs-debug-tree -t 583 /dev/sda5 | grep -A 50 -B 50 994163 > inode994163_39PBfile.txt 381K gzip file https://drive.google.com/open?id=0B_2Asp8DGjJ9Rk1qMG54MUNDWkE But I'm going to rm this file so I can see if the problem is reproducible. --- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: qcow2 becomes 37P in size while qemu crashes
Using btrfs-debug-tree, I'm finding something a bit odd about some of the items in this 39P file. Seems normal item 71 key (994163 EXTENT_DATA 43689967616) itemoff 12467 itemsize 53 extent data disk byte 617349906432 nr 80805888 extent data offset 0 nr 80805888 ram 80805888 extent compression(none) Seems not normal item 58 key (994163 EXTENT_DATA 38345535488) itemoff 13156 itemsize 53 extent data disk byte 0 nr 0 extent data offset 394752000 nr 61440 ram 34626881742770176 extent compression(none) There are quite a large number of items that take the 2nd form, and in each case the ram value is the same as above. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Chances to recover with bad partition table?
Hendrik Friedel posted on Sat, 23 Jul 2016 13:15:50 +0200 as excerpted: > this morning I had to face an unusual prompt on my machine. > > I found that the partition table of /dev/sda had vanished. > > I restored it with testdisk. It found one partition, but I am quite sure > there was a /boot partition in front of that which was not found. > > Now, running btrfsck fails: > > root@homeserver:~# fdisk -l /dev/sda > > WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util > fdisk doesn't support GPT. Use GNU Parted. There are two partition types in common usage on x86, the older MBR type and the newer GPT type, which you're running. As it says, fdisk doesn't support the latter well, tho it does warn about it (and I think new fdisk /might/ actually support it, but if so you're not using new enough, obviously). But you're in luck, because gpt stores a second copy of the partition table at the /end/ of the device, as well as the primary copy at the beginning, and good gpt tools will be able to restore the primary copy from the secondary one. Try gdisk, aka gptfdisk. It both supports gpt partition tables and has functionality that allows you to restore the primary from the secondary as above. It has a good manpage, and there's more info about it on the home page as well. http://www.rodsbooks.com/gdisk/ -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
It seems like I accidentally managed to break my Btrfs/RAID5 filesystem, yet again, in a similar fashion. This time around, I ran into some random libata driver issue (?) instead of a faulty hardware part but the end result is quiet similar. I issued the command (replacing X with valid letters for every hard-drives in the system): # echo 1 > /sys/block/sdX/device/queue_depth and I ended up with read-only filesystems. I checked dmesg and saw write errors on every disks (not just those in RAID-5). I tried to reboot immediately without success. My root filesystem with a single-disk Btrfs (which is an SSD, so it has "single" profile for both data and metadata) was unmountable, thus the kernel was stuck in a panic-reboot cycle. I managed to fix this one by booting from an USB stick and trying various recovery methods (like mounting it with "-o clear_cache,nospace_cache,recovery" and running "btrfs rescue chunk-recovery") until everything seemed to be fine (it can now be mounted read-write without error messages in the kernel-log, can be fully scrubbed without errors reported, it passes in "btrfs check", files can be actually written and read, etc). Once my system was up and running (well, sort of), I realized my /data is also un-mountable. I tried the same recovery methods on this RAID-5 filesystem but nothing seemed to help (there is an exception with the recovery attempts: the system drive was a small and fast SSD so "chunk-recovery" was a viable option to try but this one consists of huge slow HDDs - so, I tried to run it as a last-resort over-night but I found an unresponsive machine on the morning with the process stuck relatively early in the process). I can always mount it read-only and access files on it, seemingly without errors (I compared some of the contents with backups and it looks good) but as soon as I mount it read-write, all hell breaks loose and it falls into read-only state in no time (with some files seemingly disappearing from the filesystem) and the kernel log is starting to get spammed with various kind of error messages (including missing csums, etc). After mounting it like this: # mount /dev/sdb /data -o rw,noatime,nospace_cache and doing: # btrfs scrub start /data the result is: scrub status for 7d4769d6-2473-4c94-b476-4facce24b425 scrub started at Sat Jul 23 13:50:55 2016 and was aborted after 00:05:30 total bytes scrubbed: 18.99GiB with 16 errors error details: read=16 corrected errors: 0, uncorrectable errors: 16, unverified errors: 0 The relevant dmesg output is: [ 1047.709830] BTRFS info (device sdc): disabling disk space caching [ 1047.709846] BTRFS: has skinny extents [ 1047.895818] BTRFS info (device sdc): bdev /dev/sdc errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1047.895835] BTRFS info (device sdc): bdev /dev/sdb errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1065.764352] BTRFS: checking UUID tree [ 1386.423973] BTRFS error (device sdc): parent transid verify failed on 24431936729088 wanted 585936 found 586145 [ 1386.430922] BTRFS error (device sdc): parent transid verify failed on 24431936729088 wanted 585936 found 586145 [ 1411.738955] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1411.948040] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.040964] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.040980] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.041134] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.042628] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.042748] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1499.45] BTRFS error (device sdc): parent transid verify failed on 24432312270848 wanted 585779 found 586143 [ 1499.230264] BTRFS error (device sdc): parent transid verify failed on 24432312270848 wanted 585779 found 586143 [ 1525.865143] BTRFS error (device sdc): parent transid verify failed on 24432367730688 wanted 585779 found 586144 [ 1525.880537] BTRFS error (device sdc): parent transid verify failed on 24432367730688 wanted 585779 found 586144 [ 1552.434209] BTRFS error (device sdc): parent transid verify failed on 24432415821824 wanted 585781 found 586144 [ 1552.437325] BTRFS error (device sdc): parent transid verify failed on 24432415821824 wanted 585781 found 586144 btrfs check /dev/sdc results in: Checking filesystem on /dev/sdc UUID: 7d4769d6-2473-4c94-b476-4facce24b425 checking extents parent transid verify failed on 24431859855360 wanted 585941 found 586144 parent transid verify failed on 24431859855360 wanted 585941 found 586144 checksum verify failed on 24431859855360 found 3F0C0853 wanted 165308D5 par
Chances to recover with bad partition table?
Hello, this morning I had to face an unusual prompt on my machine. I found that the partition table of /dev/sda had vanished. I restored it with testdisk. It found one partition, but I am quite sure there was a /boot partition in front of that which was not found. Now, running btrfsck fails: root@homeserver:~# fdisk -l /dev/sda WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sda: 120.0 GB, 120034123776 bytes 255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x Device Boot Start End Blocks Id System /dev/sda1 * 1026048 234440703 116707328 83 Linux root@homeserver:~# btrfsck /dev/sda1 checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted bytenr mismatch, want=20987904, have=0 Couldn't read chunk root Couldn't open file system Is there a way to let btrfs search for the start of the partiton? I do have a backup; thus it is not fatal. But some data on the disk is more recent than my back up (of course) Regards, Hendrik --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chances to recover with bad partition table?
Hello, this morning I had to face an unusual prompt on my machine. I found that the partition table of /dev/sda had vanished. I restored it with testdisk. It found one partition, but I am quite sure there was a /boot partition in front of that which was not found. Now, running btrfsck fails: root@homeserver:~# fdisk -l /dev/sda WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sda: 120.0 GB, 120034123776 bytes 255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x Device Boot Start End Blocks Id System /dev/sda1 * 1026048 234440703 116707328 83 Linux root@homeserver:~# btrfsck /dev/sda1 checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted checksum verify failed on 20987904 found E4E3BDB6 wanted bytenr mismatch, want=20987904, have=0 Couldn't read chunk root Couldn't open file system Is there a way to let btrfs search for the start of the partiton? I do have a backup; thus it is not fatal. But some data on the disk is more recent than my back up (of course) Regards, Hendrik --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
