On Mon, Apr 18, 2016 at 01:18:31AM +0200, Dmitry Katsubo wrote:
> On 2016-04-14 22:30, Dmitry Katsubo wrote:
> > Dear btrfs community,
> >
> > I have the following setup:
> >
> > # btrfs fi show /home
> > Label: none uuid: 865f8cf9-27be-41a0-85a4-6cb4d1658ce3
> > Total devices 3 FS bytes used 55.68GiB
> > devid 1 size 52.91GiB used 0.00B path /dev/sdd2
> > devid 2 size 232.89GiB used 59.03GiB path /dev/sda
> > devid 3 size 111.79GiB used 59.03GiB path /dev/sdc1
> >
> > btrfs volume was created in raid1 mode both for data and metadata and
> > mounted
> > with compress=lzo option.
> >
> > Unfortunately, two drives (sda and sdc1) started to fail at the same time.
> > This
> > leads to system crash if I start the system in runlevel 3 (see crash1.log).
> >
> > After I have started the system in single mode, volume can be mounted in rw
> > mode and I can write some data into it. Unfortunately when I tried to read
> > a certain file, the system crashed (see crash2.log).
> >
> > I have started scrub on the volume and here is the report:
> >
> > # btrfs scrub status /home
> > scrub status for 865f8cf9-27be-41a0-85a4-6cb4d1658ce3
> > scrub started at Tue Apr 12 20:39:20 2016 and finished after 02:40:09
> > total bytes scrubbed: 55.68GiB with 1767 errors
> > error details: verify=175 csum=1592
> > corrected errors: 1110, uncorrectable errors: 657, unverified errors: 0
> >
> > Obviously, some data is lost. However due to above crash, I cannot just copy
> > the data from the volume. I would assume that I still can access the data,
> > but
> > the files for which data is lost, should result I/O error (I would then
> > recover
> > them from my backup).
> >
> > I have decided to attach another drive and remove failing devices
> > one-by-one.
> > However that does not work:
> >
> > # btrfs dev delete /dev/sda /home
> > [ 168.680057] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> > [ 168.684236] ata3.00: BMDMA stat 0x25
> > [ 168.688464] ata3.00: failed command: READ DMA
> > [ 168.692681] ata3.00: cmd c8/00:08:68:4b:84/00:00:00:00:00/e7 tag 0 dma
> > 4096 in
> > [ 168.692681] res 51/40:08:68:4b:84/40:08:07:00:00/e7 Emask 0x9
> > (media error)
> > [ 168.701281] ata3.00: status: { DRDY ERR }
> > [ 168.705600] ata3.00: error: { UNC }
> > [ 168.724446] blk_update_request: I/O error, dev sda, sector 126110568
> > [ 168.728860] BTRFS error (device sdc1): bdev /dev/sda errs: wr 0, rd 43,
> > flush 0, corrupt 0, gen 0
> > [ 172.824043] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> > [ 172.828651] ata3.00: BMDMA stat 0x25
> > [ 172.833281] ata3.00: failed command: READ DMA
> > [ 172.837876] ata3.00: cmd c8/00:08:50:4b:84/00:00:00:00:00/e7 tag 0 dma
> > 4096 in
> > [ 172.837876] res 51/40:08:50:4b:84/40:08:07:00:00/e7 Emask 0x9
> > (media error)
> > [ 172.847296] ata3.00: status: { DRDY ERR }
> > [ 172.852054] ata3.00: error: { UNC }
> > [ 172.872404] blk_update_request: I/O error, dev sda, sector 126110544
> > [ 172.877241] BTRFS error (device sdc1): bdev /dev/sda errs: wr 0, rd 44,
> > flush 0, corrupt 0, gen 0
> > ERROR: error removing device '/dev/sda': Input/output error
> >
> > The same happens when I try to delete /dev/sdc1 from the volume. Is there
> > any
> > btrfs "force" option so that btrfs balances only chunks that are
> > accessible? I
> > can potentially physically disconnect /dev/sda, but the loss will be greater
> > I believe.
> >
> > How can I proceed except btrfs restore?
> >
> > During scrub operation the following was recorded in the logs:
> >
> > [Tue Apr 12 23:10:20 2016] BTRFS warning (device sdc1): checksum error at
> > logical 126952947712 on dev /dev/sdc1, sector 126150176, root 258, inode
> > 879324, offset 308256768, length 4096, links 1 (path: lib/mysql/ibdata1)
> >
> > If I collect all the messages like this, will it give a full picture of
> > damaged files?
> >
> > Many thanks in advance.
> >
> > P.S. Linux kernel v4.4.2, btrfs-progs v4.4.
>
> I have decided to try "btrfs restore". Actually I have discovered two
> usability
> points about it:
>
> 1. I cannot run this utility as following:
>
> btrfs -i restore /dev/sda /mnt/usb &> log
>
> because this command is interactive and may read something from the terminal.
> It would be nice if there is a flag -y (answer "yes" to all questions) so that
> no input is required from user. The example of the question is:
>
> We seem to be looping a lot on ..., do you want to keep going on? [y/N/a]
>
> In general this question puzzles me. What does it mean? As far as I understood
> it prevents btrfs restore from looping forever. Should I consider those files
> as lost? I have also hit the same problem as discussed in [1]: answer
> "a" (always) still causes the questions to be asked.
>
> 2. btrfs restore does not print a final statistics: how many files are
> successfully restored, and how many have failed.
Thanks for trying 'restore', but I was wondering, does btrfsck work for you?
Thanks,
-liubo
>
> [1] https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg36458.html
>
> --
> With best regards,
> Dmitry
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
