Re: Infinite loop/hang in in btrfs_find_space_cluster()

[Liu Bo]

On Tue, Apr 19, 2016 at 02:51:33PM -0700, Eric Wheeler wrote:
> On Wed, 13 Apr 2016, Eric Wheeler wrote:
> 
> > Hello all,
> > 
> > We just got this backtrace in 4.4.6 on an ARM AM335x (beaglebone 
> > compatible).  The trace looks similar to this one:
> >   http://permalink.gmane.org/gmane.comp.file-systems.btrfs/54734 
> 
> We solved the problem so I thought I would report back in case anyone else 
> runs in to the same issue.  
> 
> This is running on a memory constrained 700mhz ARM (single core) so we had 
> these mount options that worked fine until we went from a single volume to 
> raid1, at which time it hung after the first reboot (or remount?):
>   
> ssd,ssd_spread,thread_pool=1,noatime,compress-force=lzo,nospace_cache,commit=37
> 
> so we switched to these on the raid1 system:
>   noatime,compress-force=lzo,commit=37
> and it stopped hanging.
> 
> Ultimately my guess is either of thread_pool=1 causing deadlock (no 
> progress), or nospace_cache causing space churn.  
> 
> We used thread_pool=1 which worked for a long time on a single volume.  
> We just added a 2nd volume and went to raid1, so perhaps the minimum 
> thread_pool should be the number of volumes?  I cannot be certain, so FYI 
> in case others run into this.

Thanks for giving us this feedback, I think 'nospace-cache' can be that
cause since 'thread_pool' is really just affecting workqueue part and
suggesting how many works it processes, it's supposed to work with any
sane numbers.

Can you please add back nospace-cache and see if we can get the hang
again?

Thanks,

-liubo

> 
> --
> Eric Wheeler
> 
> 
> 
> 
> 
> 
> but I 
> > don't have nice backtraces on this hardware (maybe hang traces are a 
> > compile option?).
> > 
> > In my case, the system appeared to hang but sysrq functions still worked 
> > and I was able to send a sysrq-(c)rash over serial.  The filesystem was 
> > just formatted in RAID1, and while I cannot access it because this hangs 
> > at boot, there can't be very much data yet.
> > 
> > This looks like the top of the relevant section, full trace below:
> > 
> > [   80.738518] [] (setup_cluster_no_bitmap [btrfs]) from 
> > [] (btrfs_find_space_cluster+0x10c/0x1dc [btrfs])
> > 
> > Any help you can offer would be greatly appreciated!
> > 
> > -Eric
> > 
> > [   80.005546] sysrq: SysRq : Trigger a crash
> > [   80.018339] pgd = c0004000
> > [   80.021160] [] *pgd=
> > [   80.024897] Internal error: Oops: 817 [#1] ARM
> > [   80.029531] Modules linked in: btrfs raid10 raid456 async_raid6_recov 
> > async_memcpy async_pq async_xor async_tx xor
> >  raid6_pq raid0 multipath linear zram lz4_compress zsmalloc dm_thin_pool 
> > dm_persistent_data dm_bio_prison dm_snapshot
> >  dm_bufio dm_zero dm_mod raid1 md_mod
> > [   80.054411] CPU: 0 PID: 6 Comm: kworker/u2:0 Not tainted 
> > 4.4.6-vr3-4-g2dfa78e #15
> > [   80.062579] Hardware name: Generic AM33XX (Flattened Device Tree)
> > [   80.069476] Workqueue: btrfs-delalloc btrfs_delalloc_helper [btrfs]
> > [   80.076026] task: cc8344c0 ti: cc858000 task.ti: cc858000
> > [   80.081669] PC is at sysrq_handle_crash+0x28/0x30
> > [   80.086576] LR is at sysrq_handle_crash+0x24/0x30
> > [   80.091484] pc : []lr : []psr: 6193
> > [   80.091484] sp : cc8599e8  ip : cc8599e8  fp : cc8599fc
> > [   80.103460] r10: cc8ecb80  r9 : c06008ce  r8 : 0001
> > [   80.108909] r7 : 0007  r6 : 0063  r5 : c05d6308  r4 : 0001
> > [   80.115717] r3 :   r2 : c05d62c8  r1 :   r0 : 0063
> > [   80.122529] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  
> > Segment none
> > [   80.130063] Control: 10c5387d  Table: 8a42c019  DAC: 0051
> > [   80.136057] Process kworker/u2:0 (pid: 6, stack limit = 0xcc858210)
> > [   80.142595] Stack: (0xcc8599e8 to 0xcc85a000)
> > [   80.147145] 99e0:   c02936cc c05e34f4 cc859a24 cc859a00 
> > c0293d00 c02936d8
> > [   80.155683] 9a00: c05b2bc4 cca49010  0101 0002 0005 
> > cc859a34 cc859a28
> > [   80.164220] 9a20: c0294208 c0293c64 cc859a74 cc859a38 c02ad3fc c02941e4 
> >  
> > [   80.172758] 9a40: 0001 0001 00015200 cca22380 cc8ecb90  
> >  009b
> > [   80.181296] 9a60: c06008ce cc8ecb80 cc859aac cc859a78 c006652c c02ad050 
> > c00409f4 c00405e0
> > [   80.189833] 9a80:  cc8ecb80 cc8ecb90   cc804000 
> > cc15ec50 ccd02518
> > [   80.198373] 9aa0: cc859ac4 cc859ab0 c008 c00664d4 00022000 cc8ecb80 
> > cc859adc cc859ac8
> > [   80.206912] 9ac0: c00690ec c0066644 009b c05de0a0 cc859aec cc859ae0 
> > c0065e58 c006905c
> > [   80.215451] 9ae0: cc859b14 cc859af0 c0065f7c c0065e44 cc859b30 c06342c0 
> > 6013 
> > [   80.223990] 9b00: cc859b64 00201000 cc859b2c cc859b18 c00093b8 c0065f34 
> > bf187974 bf18799c
> > [   80.232529] 9b20: cc859bcc cc859b30 c0412e14 c000938c cc15ec50 cc15fba8 
> > 00201000 
> > [   80.241066] 9b40: 5000  cc859bf8 ca144880 

Re: Question: raid1 behaviour on failure

[Qu Wenruo]

Matthias Bodenbinder wrote on 2016/04/21 07:22 +0200:

Am 20.04.2016 um 09:25 schrieb Qu Wenruo:



Unfortunately, this is the designed behavior.

The fs is rw just because it doesn't hit any critical problem.

If you try to touch a file and then sync the fs, btrfs will become RO 
immediately.





Btrfs fails to read space cache, nor make a new dir.

The failure on cow_block in mkdir is ciritical, and btrfs become RO.

All expected behavior so far.

You may try use degraded mount option, but AFAIK it may not handle case like 
yours.


This really scares me. "Expected bevahour"?
So you are saying: If one of the drives in the raid1 is going dead without 
noticing btrfs, the redundancy is lost.

Lets say, the power unit of a disc is going dead. This disc will disappear from 
the raid1 pretty much as suddenly as in my test case here. No difference.

You are saying that in this case, btrfs should exactly behave like this? If 
that is the case I eventually need to rethink my interpretation of redundancy.

Matthias



The "expected behavior" just means the abort transaction behavior for 
critical error is expected.


And you should know, btrfs is not doing full block level RAID1, it's 
doing RAID at chunk level.
Which needs to consider more things than full block level RAID1, and 
it's more flex than block level raid1.
(For example, you can use 3 devices with different sizes to do btrfs 
RAID1 and get more available size than mdadm raid1)


You may think the behavior is totally insane for btrfs RAID1, but don't 
forget, btrfs can have different metdata/data profile.
(And even more, there is already plan to support different profile for 
different subvolumes)


In case your metadata is RAID1, your data can still be RAID0, and in 
that case a missing devices can still cause huge problem.


There are already unmerged patches which will partly do the mdadm level 
behavior, like automatically change to degraded mode without making the 
fs RO.


The original patchset:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/48335

Or the latest patchset inside Anand Jain's auto-replace patchset:
http://thread.gmane.org/gmane.comp.file-systems.btrfs/55446

Thanks,
Qu



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KERNEL PANIC + CORRUPTED BTRFS?

[Liu Bo]

On Wed, Apr 20, 2016 at 10:09:07PM +0200, lenovomi wrote:
> Hi Chris,
> 
> please find below attached the complete log while executing all the
> brtrfs commands, all of them failed.
> 
> ;-(
> 
> 
> https://bpaste.net/show/4d8877a49b80
> https://bpaste.net/show/7e2e5aa30741
> https://bpaste.net/show/482e91b25fc5
> https://bpaste.net/show/5093cc3daa5a
> https://bpaste.net/show/a24935eb5a1b

It's not that easy to corrupt both of metadata copies which are located
in two different drives by an unclean shutdown because we always do COW.

>From what I can tell from the above results, the two copies of raid1
remain consistent and indentical, but somehow there are some problems in
checksum field. 

-
root@heap-unreal:/home/heap/btrfs-progs# ./btrfs check --readonly /dev/sda
checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC, dev 
bytenr 2972268003328, devid 2
checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC, dev 
bytenr 2972268003328, devid 2
checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC, dev 
bytenr 2973311336448, devid 1
checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC, dev 
bytenr 2972268003328, devid 2
-

In order to verify that, please follow this and show us what you get.

1. dd if=/dev/sdb of=/tmp/corrupt-dev2.txt bs=1 skip=2972268003327 count=16384
2. dd if=/dev/sdd of=/tmp/corrupt-dev1.txt bs=1 skip=2973311336447 count=16384
3. od -x /tmp/corrupt-dev2.txt
4. od -x /tmp/corrupt-dev1.txt

Thanks,

-liubo

> 
> Thanks
> 
> On Tue, Apr 12, 2016 at 9:58 PM, Chris Murphy  wrote:
> > On Tue, Apr 12, 2016 at 9:48 AM, lenovomi  wrote:
> >
> >> root@ubuntu:/home/ubuntu# btrfs restore -D -v  /dev/sda /mnt/usb/
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> Csum didn't match
> >> Couldn't read tree root
> >> Could not open root, trying backup super
> >> warning, device 2 is missing
> >> warning devid 2 not found already
> >> warning devid 3 not found already
> >> warning devid 4 not found already
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> Csum didn't match
> >> Couldn't read tree root
> >> Could not open root, trying backup super
> >> warning, device 2 is missing
> >> warning devid 2 not found already
> >> warning devid 3 not found already
> >> warning devid 4 not found already
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
> >> Csum didn't match
> >> Couldn't read tree root
> >> Could not open root, trying backup super
> >>
> >
> > Why are devices 2, 3, 4 missing? I think there's a known issue where
> > btrfs fi show might see drives as available that other tools won't
> > see. Try 'btrfs dev scan' and then repeat the restore command with -D
> > just to see if the missing device warnings go away. If devices are
> > missing, it's kinda hard to do a restore.
> >
> >
> > If these are hard drives, there should be supers 0, 1, 2 and they
> > should all be the same. But they may not be the same on each drive, so
> > it's worth checking:
> >
> > btrfs-show-super -f 
> >
> > And then also btrfs-find-root 
> >
> >
> >
> >
> > --
> > Chris Murphy
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question: raid1 behaviour on failure

[Matthias Bodenbinder]

Am 20.04.2016 um 09:25 schrieb Qu Wenruo:

> 
> Unfortunately, this is the designed behavior.
> 
> The fs is rw just because it doesn't hit any critical problem.
> 
> If you try to touch a file and then sync the fs, btrfs will become RO 
> immediately.
> 


> Btrfs fails to read space cache, nor make a new dir.
> 
> The failure on cow_block in mkdir is ciritical, and btrfs become RO.
> 
> All expected behavior so far.
> 
> You may try use degraded mount option, but AFAIK it may not handle case like 
> yours.

This really scares me. "Expected bevahour"? 
So you are saying: If one of the drives in the raid1 is going dead without 
noticing btrfs, the redundancy is lost. 

Lets say, the power unit of a disc is going dead. This disc will disappear from 
the raid1 pretty much as suddenly as in my test case here. No difference.

You are saying that in this case, btrfs should exactly behave like this? If 
that is the case I eventually need to rethink my interpretation of redundancy.

Matthias



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question: raid1 behaviour on failure

[Matthias Bodenbinder]

Am 20.04.2016 um 15:32 schrieb Anand Jain:
>> 1. mount the raid1 (2 disc with different size)
> 
>> 2. unplug the biggest drive (hotplug)
> 
>   Btrfs won't know that you have plugged-out a disk.
>   Though it experiences IO failures, it won't close the bdev.

Well, as far as I can tell mdadm can handle this use case. I tested that. I 
have an mdadm raid5 running. I accidentially unplugged a sata cable from one of 
the devices and the raid still worked. I did not even notice that the cable was 
unplugged until a few hours later. Then I plugged in the cable agaib and that 
was it. mdadm recovered the raid5 without any problem. -> This is redunancy!


> 
>> 3. try to copy something to the degraded raid1
> 
>   This will work as long as you do _not_ run unmount/mount.
 
I did not umount the raid1 when I tried to copy something. As you can see from 
the sequence of events: I removed the drive and immdiately afterwards tried to 
copy something to the degraded array. This copy failed with a crash of the 
btrfs module. -> This is NOT redundancy.

The ummount and mount operations are coming afterwards.

In a nutshell I have to say that the btrfs behaviour is by no means compliant 
with my understanding of redundancy.


Matthias



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel crash if both devices in raid1 are failing

[Liu Bo]

On Mon, Apr 18, 2016 at 01:18:31AM +0200, Dmitry Katsubo wrote:
> On 2016-04-14 22:30, Dmitry Katsubo wrote:
> > Dear btrfs community,
> > 
> > I have the following setup:
> > 
> > # btrfs fi show /home
> > Label: none  uuid: 865f8cf9-27be-41a0-85a4-6cb4d1658ce3
> > Total devices 3 FS bytes used 55.68GiB
> > devid1 size 52.91GiB used 0.00B path /dev/sdd2
> > devid2 size 232.89GiB used 59.03GiB path /dev/sda
> > devid3 size 111.79GiB used 59.03GiB path /dev/sdc1
> > 
> > btrfs volume was created in raid1 mode both for data and metadata and 
> > mounted
> > with compress=lzo option.
> > 
> > Unfortunately, two drives (sda and sdc1) started to fail at the same time. 
> > This
> > leads to system crash if I start the system in runlevel 3 (see crash1.log).
> > 
> > After I have started the system in single mode, volume can be mounted in rw
> > mode and I can write some data into it. Unfortunately when I tried to read
> > a certain file, the system crashed (see crash2.log).
> > 
> > I have started scrub on the volume and here is the report:
> > 
> > # btrfs scrub status /home
> > scrub status for 865f8cf9-27be-41a0-85a4-6cb4d1658ce3
> > scrub started at Tue Apr 12 20:39:20 2016 and finished after 02:40:09
> > total bytes scrubbed: 55.68GiB with 1767 errors
> > error details: verify=175 csum=1592
> > corrected errors: 1110, uncorrectable errors: 657, unverified errors: 0
> > 
> > Obviously, some data is lost. However due to above crash, I cannot just copy
> > the data from the volume. I would assume that I still can access the data, 
> > but
> > the files for which data is lost, should result I/O error (I would then 
> > recover
> > them from my backup).
> > 
> > I have decided to attach another drive and remove failing devices 
> > one-by-one.
> > However that does not work:
> > 
> > # btrfs dev delete /dev/sda /home
> > [  168.680057] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> > [  168.684236] ata3.00: BMDMA stat 0x25
> > [  168.688464] ata3.00: failed command: READ DMA
> > [  168.692681] ata3.00: cmd c8/00:08:68:4b:84/00:00:00:00:00/e7 tag 0 dma 
> > 4096 in
> > [  168.692681]  res 51/40:08:68:4b:84/40:08:07:00:00/e7 Emask 0x9 
> > (media error)
> > [  168.701281] ata3.00: status: { DRDY ERR }
> > [  168.705600] ata3.00: error: { UNC }
> > [  168.724446] blk_update_request: I/O error, dev sda, sector 126110568
> > [  168.728860] BTRFS error (device sdc1): bdev /dev/sda errs: wr 0, rd 43, 
> > flush 0, corrupt 0, gen 0
> > [  172.824043] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> > [  172.828651] ata3.00: BMDMA stat 0x25
> > [  172.833281] ata3.00: failed command: READ DMA
> > [  172.837876] ata3.00: cmd c8/00:08:50:4b:84/00:00:00:00:00/e7 tag 0 dma 
> > 4096 in
> > [  172.837876]  res 51/40:08:50:4b:84/40:08:07:00:00/e7 Emask 0x9 
> > (media error)
> > [  172.847296] ata3.00: status: { DRDY ERR }
> > [  172.852054] ata3.00: error: { UNC }
> > [  172.872404] blk_update_request: I/O error, dev sda, sector 126110544
> > [  172.877241] BTRFS error (device sdc1): bdev /dev/sda errs: wr 0, rd 44, 
> > flush 0, corrupt 0, gen 0
> > ERROR: error removing device '/dev/sda': Input/output error
> > 
> > The same happens when I try to delete /dev/sdc1 from the volume. Is there 
> > any
> > btrfs "force" option so that btrfs balances only chunks that are 
> > accessible? I
> > can potentially physically disconnect /dev/sda, but the loss will be greater
> > I believe.
> > 
> > How can I proceed except btrfs restore?
> > 
> > During scrub operation the following was recorded in the logs:
> > 
> > [Tue Apr 12 23:10:20 2016] BTRFS warning (device sdc1): checksum error at 
> > logical 126952947712 on dev /dev/sdc1, sector 126150176, root 258, inode 
> > 879324, offset 308256768, length 4096, links 1 (path: lib/mysql/ibdata1)
> > 
> > If I collect all the messages like this, will it give a full picture of 
> > damaged files?
> > 
> > Many thanks in advance.
> > 
> > P.S. Linux kernel v4.4.2, btrfs-progs v4.4.
> 
> I have decided to try "btrfs restore". Actually I have discovered two 
> usability
> points about it:
> 
> 1. I cannot run this utility as following:
> 
> btrfs -i restore /dev/sda /mnt/usb &> log
> 
> because this command is interactive and may read something from the terminal.
> It would be nice if there is a flag -y (answer "yes" to all questions) so that
> no input is required from user. The example of the question is:
> 
> We seem to be looping a lot on ..., do you want to keep going on? [y/N/a]
> 
> In general this question puzzles me. What does it mean? As far as I understood
> it prevents btrfs restore from looping forever. Should I consider those files
> as lost? I have also hit the same problem as discussed in [1]: answer
> "a" (always) still causes the questions to be asked.
> 
> 2. btrfs restore does not print a final statistics: how many files are
> successfully restored, and how many have failed.

Thanks for 

BTRFS: assertion failed: num_extents, file: fs/btrfs/extent-tree.c, line: 5584

[Dave Jones]

Don't think I've reported this one before. It's on the same box I've been
seeing the btrfs_destroy_inode WARN_ON's on though.

Dave

BTRFS: assertion failed: num_extents, file: fs/btrfs/extent-tree.c, line: 5584
[ cut here ]
kernel BUG at fs/btrfs/ctree.h:4320!
invalid opcode:  [#1] SMP DEBUG_PAGEALLOC KASAN
CPU: 1 PID: 9320 Comm: trinity-c21 Not tainted 4.6.0-rc4-think+ #5
task: 880453bdb7c0 ti: 88045893 task.ti: 88045893
RIP: 0010:[]  [] 
assfail.constprop.88+0x1c/0x1e [btrfs]
RSP: :880458937860  EFLAGS: 00010282
RAX: 004e RBX:  RCX: 
RDX:  RSI: 0001 RDI: ed008b126f02
RBP: 880458937860 R08: 0001 R09: 0001
R10: 0003 R11: 0001 R12: 88045ef14548
R13: 88045066e048 R14: 88045066dc58 R15: 008f
FS:  7f6465c7b700() GS:88046880() knlGS:
CS:  0010 DS:  ES:  CR0: 80050033
CR2: 7f6465c81000 CR3: 000454e05000 CR4: 001406e0
Stack:
 8804589378a0 c02407f3 008f 
 88045ef14548 88045066e048 88045066dc58 008f
 8804589378f0 c0253a3c 880453bdc048 880453bdb7c0
Call Trace:
 [] drop_outstanding_extent+0x153/0x1a0 [btrfs]
 [] btrfs_delalloc_release_metadata+0x9c/0x5d0 [btrfs]
 [] btrfs_delalloc_release_space+0x1f/0x50 [btrfs]
 [] __btrfs_buffered_write+0xb05/0xea0 [btrfs]
 [] ? btrfs_dirty_pages+0x2d0/0x2d0 [btrfs]
 [] ? local_clock+0x1c/0x20
 [] ? local_clock+0x1c/0x20
 [] ? debug_lockdep_rcu_enabled+0x77/0x90
 [] ? btrfs_file_write_iter+0xa07/0x1570 [btrfs]
 [] btrfs_file_write_iter+0x631/0x1570 [btrfs]
 [] ? __might_fault+0x166/0x1b0
 [] ? __might_fault+0xcb/0x1b0
 [] do_iter_readv_writev+0x134/0x230
 [] ? vfs_iter_read+0x260/0x260
 [] ? rcu_sync_lockdep_assert+0x78/0xb0
 [] ? percpu_down_read+0x5c/0xa0
 [] ? __sb_start_write+0xb4/0xf0
 [] do_readv_writev+0x39c/0x6a0
 [] ? btrfs_sync_file+0xd00/0xd00 [btrfs]
 [] ? vfs_write+0x4a0/0x4a0
 [] ? local_clock+0x1c/0x20
 [] ? __context_tracking_exit.part.6+0x52/0x220
 [] ? enter_from_user_mode+0x50/0x50
 [] vfs_writev+0x75/0xb0
 [] do_pwritev+0x12a/0x170
 [] ? SyS_pwritev+0x20/0x20
 [] SyS_pwritev2+0x17/0x30
 [] do_syscall_64+0x19b/0x4a0
 [] ? trace_hardirqs_on_thunk+0x1b/0x1d
 [] entry_SYSCALL64_slow_path+0x25/0x25
Code: d3 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 89 f1 48 c7 c2 e0 c0 43 
c0 48 89 fe 48 89 e5 48 c7 c7 20 c4 43 c0 e8 92 06 01 d3 <0f> 0b 55 48 89 e5 41 
57 41 56 41 55 41 54 53 48 83 ec 18 48 89 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel crash if both devices in raid1 are failing

[Dmitry Katsubo]

On 2016-04-19 09:58, Duncan wrote:
> Dmitry Katsubo posted on Tue, 19 Apr 2016 07:45:40 +0200 as excerpted:
> 
>> Actually btrfs restore has recovered many files, however I was not able
>> to run in fully unattended mode as it complains about "looping a lot".
>> Does it mean that files are corrupted / not correctly restored?
> 
> As long as you tell it to keep going each time, the loop complaints 
> shouldn't be an issue.  The problem is that the loop counter is measuring 
> loops on a particular directory, because that's what it has available to 
> measure.  But if you had a whole bunch of files in that dir, it's /going/ 
> to loop a lot, to restore all of them.
> 
> I have one cache directory with over 200K files in it.  They're all text 
> messages from various technical lists and newsgroups (like this list, 
> which I view as a newsgroup using gmane.org's list2news service) so 
> they're quite small, about 5 KiB on average by my quick calculation, but 
> that's still a LOT of files for a single dir, even if they're only using 
> just over a GiB of space.
> 
> I ended up doing a btrfs restore on that filesystem (/home), because 
> while I had a backup, restore was getting more recent copies of stuff 
> back, and that dir looped a *LOT* the first time it happened, now several 
> years ago, before they actually added the always option.

I have the same situation here: there is a backup, but the most recent
modifications in files are preferable.

> The second time it happened, about a year ago, restore worked much 
> better, and I was able to use the always option.  But AFAIK, always only 
> applies to that dir.  If you have multiple dirs with the problem, you'll 
> still get asked for the next one.  But it did vastly improve the 
> situation for me, giving me only a handful of prompts instead of the very 
> many I had before the option was there.

Yes, this is exactly the problem discussed a while ago. Would be nice if
"btrfs restore -i" applies "(a)lways" option to all questions or there is
a separate option for that ("-y").

For me personally "looping" is too low-level problem. System administrators
(that are going to use this utility) should operate with some more reasonable
terms. If "looping" is some analogy of "time consumption" then I would say
that during restore time does not matter so much: I am ready to wait for 1
minute until a specific file is restored. So I think not the number of loops
but number of time spent should be measured.

Also I have difficulties in finding out what files have not been restored
due to uncorrectable errors. As I cannot redirect the output of
"btrfs restore" and it does not print the final stats, I cannot tell what
files have to be restored from backup.

> (The main problem triggering the need to run restore for me, turned out 
> to be hardware.  I've had no issues since I replaced that failing ssd, 
> and with a bit of luck, won't be running restore again for a few years, 
> now.)

I would be happy if I am able to replace the failing drive on the fly, without
stopping the system. Unfortunately I cannot do that due to kernel crashes :(
btrfs is still not resistant to these corner cases.

-- 
With best regards,
Dmitry
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KERNEL PANIC + CORRUPTED BTRFS?

[Chris Murphy]

On Wed, Apr 20, 2016 at 2:09 PM, lenovomi  wrote:
> Hi Chris,
>
> please find below attached the complete log while executing all the
> brtrfs commands, all of them failed.
>

> https://bpaste.net/show/482e91b25fc5

>warning, device 1 is missing
>warning, device 2 is missing

Restore probably can't work if too many devices are missing. I can't
tell you if this is a bug, or if they're really missing, or why you're
getting this message. But at least as far as the command is concerned,
it's not seeing two devices. So restore is a problem. I forget what
version of btrfs-progs you're using?





> https://bpaste.net/show/5093cc3daa5a
> https://bpaste.net/show/a24935eb5a1b

Try

btrfs-debug-tree -b 17802801315840

btrfs-restore -v -i -D -t 17802801315840

with the proper endings on those...

You might have to really constrain the restore parameters, maybe use
-f or -d or even -path-regex to find specific files you want to
scrape, and hope for the best. The less you're trying to grab, the
fewer errors, so it's a kind of triage at this point I think, short of
someone else with better ideas.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KERNEL PANIC + CORRUPTED BTRFS?

[lenovomi]

Hi Chris,

please find below attached the complete log while executing all the
brtrfs commands, all of them failed.

;-(


https://bpaste.net/show/4d8877a49b80
https://bpaste.net/show/7e2e5aa30741
https://bpaste.net/show/482e91b25fc5
https://bpaste.net/show/5093cc3daa5a
https://bpaste.net/show/a24935eb5a1b

Thanks

On Tue, Apr 12, 2016 at 9:58 PM, Chris Murphy  wrote:
> On Tue, Apr 12, 2016 at 9:48 AM, lenovomi  wrote:
>
>> root@ubuntu:/home/ubuntu# btrfs restore -D -v  /dev/sda /mnt/usb/
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> Csum didn't match
>> Couldn't read tree root
>> Could not open root, trying backup super
>> warning, device 2 is missing
>> warning devid 2 not found already
>> warning devid 3 not found already
>> warning devid 4 not found already
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> Csum didn't match
>> Couldn't read tree root
>> Could not open root, trying backup super
>> warning, device 2 is missing
>> warning devid 2 not found already
>> warning devid 3 not found already
>> warning devid 4 not found already
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> checksum verify failed on 17802818387968 found FF45E2D3 wanted BFB02AEC
>> Csum didn't match
>> Couldn't read tree root
>> Could not open root, trying backup super
>>
>
> Why are devices 2, 3, 4 missing? I think there's a known issue where
> btrfs fi show might see drives as available that other tools won't
> see. Try 'btrfs dev scan' and then repeat the restore command with -D
> just to see if the missing device warnings go away. If devices are
> missing, it's kinda hard to do a restore.
>
>
> If these are hard drives, there should be supers 0, 1, 2 and they
> should all be the same. But they may not be the same on each drive, so
> it's worth checking:
>
> btrfs-show-super -f 
>
> And then also btrfs-find-root 
>
>
>
>
> --
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs-progs confusing message

[Konstantin Svist]

Pretty much all commands print out the usage message when no device is
specified:

[root@host ~]# btrfs scrub start
btrfs scrub start: too few arguments
usage: btrfs scrub start [-BdqrRf] [-c ioprio_class -n ioprio_classdata]
|
...

However, balance doesn't

[root@host ~]# btrfs balance start
ERROR: can't access 'start': No such file or directory




--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v8 00/27][For 4.7] Btrfs: Add inband (write time) de-duplication framework

[Chris Mason]

On Wed, Apr 20, 2016 at 10:02:27AM +0800, Qu Wenruo wrote:
> Hi David,
> 
> Any new comment about the ondisk format and ioctl interface?

Hi Qu,

I'm at LSF this week but will dig through again on the way home.
Thanks!

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4] btrfs: qgroup: Fix qgroup accounting when creating snapshot

[Holger Hoffstätte]

On Tue, 19 Apr 2016 15:19:46 -0700, Mark Fasheh wrote:

> On Fri, Apr 15, 2016 at 05:08:22PM +0800, Qu Wenruo wrote:
>> Current btrfs qgroup design implies a requirement that after calling
>> btrfs_qgroup_account_extents() there must be a commit root switch.
>> 
>> Normally this is OK, as btrfs_qgroup_accounting_extents() is only called
>> inside btrfs_commit_transaction() just be commit_cowonly_roots().
>> 
>> However there is a exception at create_pending_snapshot(), which will
>> call btrfs_qgroup_account_extents() but no any commit root switch.
>> 
>> In case of creating a snapshot whose parent root is itself (create a
>> snapshot of fs tree), it will corrupt qgroup by the following trace:
>> (skipped unrelated data)
>> ==
>> btrfs_qgroup_account_extent: bytenr = 29786112, num_bytes = 16384, 
>> nr_old_roots = 0, nr_new_roots = 1
>> qgroup_update_counters: qgid = 5, cur_old_count = 0, cur_new_count = 1, rfer 
>> = 0, excl = 0
>> qgroup_update_counters: qgid = 5, cur_old_count = 0, cur_new_count = 1, rfer 
>> = 16384, excl = 16384
>> btrfs_qgroup_account_extent: bytenr = 29786112, num_bytes = 16384, 
>> nr_old_roots = 0, nr_new_roots = 0
>> ==
>> 
>> The problem here is in first qgroup_account_extent(), the
>> nr_new_roots of the extent is 1, which means its reference got
>> increased, and qgroup increased its rfer and excl.
>> 
>> But at second qgroup_account_extent(), its reference got decreased, but
>> between these two qgroup_account_extent(), there is no switch roots.
>> This leads to the same nr_old_roots, and this extent just got ignored by
>> qgroup, which means this extent is wrongly accounted.
>> 
>> Fix it by call commit_cowonly_roots() after qgroup_account_extent() in
>> create_pending_snapshot(), with needed preparation.
>> 
>> Reported-by: Mark Fasheh 
>> Signed-off-by: Qu Wenruo 
> 
> Ok, this version seems to be giving me the right numbers. I'll send a test
> for it shortly. I'd still like to know if this patch introduces an
> unintended side effects but otherwise, thanks Qu.
>   --Mark

Hi Mark,

Can't speak about other side effects since I have not observed any so far,
but I can confirm that the previously failing case of deleting a renamed
snapshot [1] now works properly with v4 without getting the commit roots
in a twist. So:

Tested-by: holger.hoffstae...@googlemail.com

cheers
Holger

[1] http://thread.gmane.org/gmane.comp.file-systems.btrfs/55052

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question: raid1 behaviour on failure

[Anand Jain]

1. mount the raid1 (2 disc with different size)



2. unplug the biggest drive (hotplug)


  Btrfs won't know that you have plugged-out a disk.
  Though it experiences IO failures, it won't close the bdev.


3. try to copy something to the degraded raid1


  This will work as long as you do _not_ run unmount/mount.

  However once you umount/mount you won't be able to mount
  even with -o degraded option. (there are some workaround
  patches in the ML)


4. plugin the device again (hotplug)


  This is a bad test case.

  - Since btrfs didn't close the device, at #2 above, the block
  layer will create a new device instance and path when you plug-in
  the device.

  And when btrfs will promptly scan the device and update its
  records. But note that its still using the old bdev. And
  you will continue to see the IO errors. And no IO will go
  to the new device instance.

  There are patches in the ML under tests which will force
  close the device upon loosing access to the device. As a
  first step.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-image and btrfs send related queries

[sri]

Sorry. Its typo I used original disk /dev/sdb where filesystem is 
created and seeing these errors.



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question: raid1 behaviour on failure

[Qu Wenruo]

Matthias Bodenbinder wrote on 2016/04/20 07:17 +0200:

Am 18.04.2016 um 09:22 schrieb Qu Wenruo:

BTW, it would be better to post the dmesg for better debug.


So here we. I did the same test again. Here is a full log of what i did. It 
seems to be mean like a bug in btrfs.
Sequenz of events:
1. mount the raid1 (2 disc with different size)
2. unplug the biggest drive (hotplug)
3. try to copy something to the degraded raid1
4. plugin the device again (hotplug)

This scenario does not work. The disc array is NOT redundant! I can not work 
with it while a drive is missing and I can not reattach the device so that 
everything works again.

The btrfs module crashes during the test.

I am using LMDE2 with backports:
btrfs-tools 4.4-1~bpo8+1
linux-image-4.4.0-0.bpo.1-amd64

Matthias


rakete - root - /root
1# mount /mnt/raid1/

Journal:

Apr 20 07:01:16 rakete kernel: BTRFS info (device sdi): enabling auto defrag
Apr 20 07:01:16 rakete kernel: BTRFS info (device sdi): disk space caching is 
enabled
Apr 20 07:01:16 rakete kernel: BTRFS: has skinny extents

rakete - root - /mnt/raid1
3# ll
insgesamt 0
drwxrwxr-x 1 root root   36 Nov 14  2014 AfterShot2(64-bit)
drwxrwxr-x 1 root root 5082 Apr 17 09:06 etc
drwxr-xr-x 1 root root  108 Mär 24 07:31 var

4# btrfs fi show
Label: none  uuid: 16d5891f-5d52-4b29-8591-588ddf11e73d
Total devices 3 FS bytes used 1.60GiB
devid1 size 698.64GiB used 3.03GiB path /dev/sdg
devid2 size 465.76GiB used 3.03GiB path /dev/sdh
devid3 size 232.88GiB used 0.00B path /dev/sdi


unplug device sdg:

Apr 20 07:03:05 rakete kernel: Buffer I/O error on dev sdf1, logical block 
243826688, lost sync page write
Apr 20 07:03:05 rakete kernel: JBD2: Error -5 detected when updating journal 
superblock for sdf1-8.
Apr 20 07:03:05 rakete kernel: Aborting journal on device sdf1-8.
Apr 20 07:03:05 rakete kernel: Buffer I/O error on dev sdf1, logical block 
243826688, lost sync page write
Apr 20 07:03:05 rakete kernel: JBD2: Error -5 detected when updating journal 
superblock for sdf1-8.
Apr 20 07:03:05 rakete umount[16405]: umount: /mnt/raid1: target is busy
Apr 20 07:03:05 rakete umount[16405]: (In some cases useful info about 
processes that
Apr 20 07:03:05 rakete umount[16405]: use the device is found by lsof(8) or 
fuser(1).)
Apr 20 07:03:05 rakete systemd[1]: mnt-raid1.mount mount process exited, 
code=exited status=32
Apr 20 07:03:05 rakete systemd[1]: Failed unmounting /mnt/raid1.
Apr 20 07:03:24 rakete kernel: usb 3-1: new SuperSpeed USB device number 3 
using xhci_hcd
Apr 20 07:03:24 rakete kernel: usb 3-1: New USB device found, idVendor=152d, 
idProduct=0567
Apr 20 07:03:24 rakete kernel: usb 3-1: New USB device strings: Mfr=10, 
Product=11, SerialNumber=5
Apr 20 07:03:24 rakete kernel: usb 3-1: Product: USB to ATA/ATAPI Bridge
Apr 20 07:03:24 rakete kernel: usb 3-1: Manufacturer: JMicron
Apr 20 07:03:24 rakete kernel: usb 3-1: SerialNumber: 152D00539000
Apr 20 07:03:24 rakete kernel: usb-storage 3-1:1.0: USB Mass Storage device 
detected
Apr 20 07:03:24 rakete kernel: usb-storage 3-1:1.0: Quirks match for vid 152d 
pid 0567: 500
Apr 20 07:03:24 rakete kernel: scsi host9: usb-storage 3-1:1.0
Apr 20 07:03:24 rakete mtp-probe[16424]: checking bus 3, device 3: 
"/sys/devices/pci:00/:00:1c.5/:04:00.0/usb3/3-1"
Apr 20 07:03:24 rakete mtp-probe[16424]: bus: 3, device: 3 was not an MTP device
Apr 20 07:03:25 rakete kernel: scsi 9:0:0:0: Direct-Access WDC WD20 
02FAEX-007BA00125 PQ: 0 ANSI: 6
Apr 20 07:03:25 rakete kernel: scsi 9:0:0:1: Direct-Access WDC WD50 
01AALS-00L3B20125 PQ: 0 ANSI: 6
Apr 20 07:03:25 rakete kernel: scsi 9:0:0:2: Direct-Access SAMSUNG  SP2504C 
 0125 PQ: 0 ANSI: 6
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: Attached scsi generic sg6 type 0
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: Attached scsi generic sg7 type 0
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] 3907029168 512-byte logical 
blocks: (2.00 TB/1.82 TiB)
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Write Protect is off
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Mode Sense: 67 00 10 08
Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: Attached scsi generic sg8 type 0
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] 976773168 512-byte logical 
blocks: (500 GB/466 GiB)
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] No Caching mode page found
Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Assuming drive cache: write 
through
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Write Protect is off
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Mode Sense: 67 00 10 08
Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] 488395055 512-byte logical 
blocks: (250 GB/233 GiB)
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] No Caching mode page found
Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Assuming drive cache: write 
through
Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] Write Protect is off
Apr 20 07:03:25 rakete 

[PATCH] btrfs-progs: prop: remove an unnecessary condition on parse_args

[Satoru Takeuchi]

>From commit c742debab11f ('btrfs-progs: fix a regression that
"property" with -t option doesn't work'), the number of arguments
is checked strictly. So the following condition never be
satisfied.

Signed-off-by: Satoru Takeuchi 
---
 cmds-property.c | 5 -
 1 file changed, 5 deletions(-)

diff --git a/cmds-property.c b/cmds-property.c
index eed5f4a..48a8945 100644
--- a/cmds-property.c
+++ b/cmds-property.c
@@ -352,11 +352,6 @@ static void parse_args(int argc, char **argv,
if (value && optind < argc)
*value = argv[optind++];

-   if (optind != argc) {
-   error("unexpected agruments found");
-   usage(usage_str);
-   }
-
if (!*types && object && *object) {
ret = autodetect_object_types(*object, types);
if (ret < 0) {
-- 
2.5.5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html