2012-01-13 4:12, Jim Klimov wrote:
As I recently wrote, my data pool has experienced some "unrecoverable errors". It seems that a userdata block of deduped data got corrupted and no longer matches the stored checksum. For whatever reason, raidz2 did not help in recovery of this data, so I rsync'ed the files over from another copy. Then things got interesting...
Well, after some crawling over my data with zdb, od and dd, I guess ZFS was right about finding checksum errors - the metadata's checksum matched that of a block on original system, and the data block was indeed erring. Just in case it helps others, the SHA256 checksums can be tested with openssl as I show below. I still search for a command-line fletcher4/fletcher2 checker (as that weak hash is used on metadata; I wonder why). Here's a tail from on-disk blkptr_t, bytes with checksum: # tail -2 /tmp/osx.l0+1100000.blkptr.txt 000460 1f 6f 4c 73 5d c1 ab 15 00 cc 56 90 38 8e b4 dd 000470 a9 8e 54 6f f1 a7 db 43 7d 61 9e 01 23 45 2e 70 In byte 0x435 I have value 0x8 - SHA256. And here is the SHA256 hash for the excerpt from original file (128Kb cut out with dd): # dd if=osx.zip of=/tmp/osx.l0+1100000.bin.orig bs=512 skip=34816 count=256 # openssl dgst -sha256 < /tmp/osx.l0+1100000.bin.orig 15abc15d734c6f1fddb48e389056cc0043dba7f16f548ea9702e4523019e617d As my x86 is little-endian, the four 8-byte words of the checksum appear reversed. But you can see it matches, so my source file is okay. I did not find the DDT entries (yet), so I don't know what hash is there or what addresses it references for how many files. The block pointer has the dedup bit set, though. However, of all my files with errors, there are no DVA overlaps. I hexdumped (with od) the two 128Kb excerpts (one from the original file, another fetched with zdb) and diffed them, and while some lines matched, others did not. What is more interesting, is that most of the error area contains a repeating pattern like this, sometimes with "extra" chars thrown in: fc 42 fc 42 fc 42 fc 42 fc 42 fc fc 42 1f fc 42 fc 42 42 ff fc 42 fc 42 fc 42 fc 42 I have seen similar patterns when I zdb-dumped compressed blocks without decompression, so I guess this could be a miswrite of compressed data and/or parity destined for another file (which also did not get it). The erroneous data starts and ends at "round" offsets like 0x1000-0x2000, 0x9000-0xa000, 0x11000-0x12000 (step 0x8000 between both sets of mismatches, size 4kb is my disk sector size), which also suggests a non-coincidental problem. However, part of the differing data is "normal-looking random noise", while some part is that pattern above, starting and ending at a seemingly random location mid-sector. Here's about all I have to say and share so far :) Open to suggestions how to compute fletcher checksums on blocks... Thanks, //Jim _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
