[PATCH] Btrfs: fix possible deadlock in btrfs_cleanup_transaction

2014-02-10 Thread Liu Bo
[13654.480669] == [13654.480905] [ INFO: possible circular locking dependency detected ] [13654.481003] 3.12.0+ #4 Tainted: GW O [13654.481060] --- [13654.481060] btrfs-transacti/9347 is

Re: [btrfs] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038

2014-02-10 Thread Fengguang Wu
Hi Filipe, > If you disable CONFIG_BTRFS_FS_RUN_SANITY_TESTS, does it still crash? I tried disabling CONFIG_BTRFS_FS_RUN_SANITY_TESTS in the reported 3 randconfigs and they all boot fine. Thanks, Fengguang -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a

Re: Provide a better free space estimate on RAID1

2014-02-10 Thread Roman Mamedov
On Mon, 10 Feb 2014 00:02:38 + (UTC) Duncan <1i5t5.dun...@cox.net> wrote: > Meanwhile, you said it yourself, users aren't normally concerned about > this. I think you're being mistaken here, the point that "users aren't looking at the free space, hence it is not important to provide a correc

Re: [btrfs] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038

2014-02-10 Thread Fengguang Wu
On Sat, Feb 08, 2014 at 03:10:37PM -0500, Tejun Heo wrote: > Hello, David, Fengguang, Chris. > > On Fri, Feb 07, 2014 at 01:13:06PM -0800, David Rientjes wrote: > > On Fri, 7 Feb 2014, Fengguang Wu wrote: > > > > > On Fri, Feb 07, 2014 at 02:13:59AM -0800, David Rientjes wrote: > > > > On Fri, 7

[PATCH v2] Btrfs: avoid warning bomb of btrfs_invalidate_inodes

2014-02-10 Thread Liu Bo
So after transaction is aborted, we need to cleanup inode resources by calling btrfs_invalidate_inodes(), and btrfs_invalidate_inodes() hopes roots' refs to be zero in old times and sets a WARN_ON(), however, this is not always true within cleaning up transaction, so we get to detect transaction ab

Re: Issue with btrfs balance

2014-02-10 Thread Imran Geriskovan
I've experienced the following with balance: Setup: - Kernel 3.12.9 - 11 DVD sized (4.3GB) loopback devices. (9 Read-Only Seed devices + 2 Read/Write devices) - 9 device seed created with "-m single -d single" and made Read-only with "btrfstune -S 1 ..." - 2 devices was added at different dates. N

[PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Wang Shilong
From: Wang Shilong So i was wandering why test 004 could pass my previous wrong kernel patch while it defenitely should not. By some debugging, i found here perl script is wrong, we did not filter out anything and this unit test did not work acutally.so it came out we will never fail this test.

[PATCH] Btrfs-progs: receive: don't output normal message into stderr

2014-02-10 Thread Wang Shilong
From: Wang Shilong Don't output normal messages into stderr, this make xfstests filter output easier. Signed-off-by: Wang Shilong --- cmds-receive.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmds-receive.c b/cmds-receive.c index cce37a7..2d55c53 100644 --- a/cmds-rece

btrfs send runs out of memory and file handles

2014-02-10 Thread Frank Kingswood
Hi, I'm attempting to back up a btrfs subvolume with $ btrfs send /path/to/subvol | nc and the receiving end does $ nc -l | btrfs receive /path/to/volume This subvolume holds ~250 GB of data, about half full, and uses RAID1. Doing so runs out of file descriptors on the sending machine (h

Re: Issue with btrfs balance

2014-02-10 Thread Brendan Hide
On 2014/02/10 04:33 AM, Austin S Hemmelgarn wrote: Apparently, trying to use -mconvert=dup or -sconvert=dup on a multi-device filesystem using one of the RAID profiles for metadata fails with a statement to look at the kernel log, which doesn't show anything at all about the failure. ^ If this i

Re: btrfs send runs out of memory and file handles

2014-02-10 Thread Hugo Mills
On Mon, Feb 10, 2014 at 01:28:38PM +, Frank Kingswood wrote: > Hi, > > I'm attempting to back up a btrfs subvolume with > > $ btrfs send /path/to/subvol | nc > > and the receiving end does > > $ nc -l | btrfs receive /path/to/volume > > This subvolume holds ~250 GB of data, about half

Re: btrfs send runs out of memory and file handles

2014-02-10 Thread Frank Kingswood
On 10/02/14 13:47, Hugo Mills wrote: On Mon, Feb 10, 2014 at 01:28:38PM +, Frank Kingswood wrote: I'm attempting to back up a btrfs subvolume >> [...] Doing so runs out of file descriptors on the sending machine (having over 100k files open) and eventually runs out of memory and gets kill

scrub crashed?

2014-02-10 Thread Johan Kröckel
root@fortknox:~# uname -a Linux fortknox 3.12-0.bpo.1-amd64 #1 SMP Debian 3.12.9-1~bpo70+1 (2014-02-07) x86_64 GNU/Linux root@fortknox:~# btrfs version Btrfs v3.12 root@fortknox:~# btrfs scrub status -d /bunker scrub status for 11312131-3372-4637-b526-35a4ef0c31eb scrub device /dev/mapper/bunkerA (

Re: lost with degraded RAID1

2014-02-10 Thread Johan Kröckel
Thanks, that explains something. There was indeed a BIOS-Problem (the drive vanished was disabled in BIOS suddenly and was only useable again after reactivating it in the BIOS again). So should have been a BIOS-problem. 2014-02-09 Duncan <1i5t5.dun...@cox.net>: > Johan Kröckel posted on Sat, 08 F

Re: Issue with btrfs balance

2014-02-10 Thread Austin S Hemmelgarn
On 2014-02-10 08:41, Brendan Hide wrote: > On 2014/02/10 04:33 AM, Austin S Hemmelgarn wrote: >> >> Apparently, trying to use -mconvert=dup or -sconvert=dup on a >> multi-device filesystem using one of the RAID profiles for metadata >> fails with a statement to look at the kernel log, which doesn'

Re: scrub crashed?

2014-02-10 Thread Shilong Wang
Hello Johan, This should be a known problem. The problem seemed that scrub log file is corrupt, so i added an option -f something like: btrfs scrub start -f <..> You can update latest btrfs-progs from david's latest integration branch and try it. if you don't want to do that, just rm /var/lib/b

Re: scrub crashed?

2014-02-10 Thread Johan Kröckel
Thank you Shilong, that was the problem. 2014-02-10 Shilong Wang : > Hello Johan, > > This should be a known problem. > > The problem seemed that scrub log file is corrupt, so i added an option > -f something like: > > btrfs scrub start -f <..> > > You can update latest btrfs-progs from david's la

Re: system stuck with flush-btrfs-4 at 100% after filesystem resize

2014-02-10 Thread John Navitsky
As a follow-up, at some point over the weekend things did finish on their own: romulus:/vms/johnn-sles11sp3 # df -h /vms Filesystem Size Used Avail Use% Mounted on /dev/dm-4 2.6T 1.6T 1.1T 60% /vms romulus:/vms/johnn-sles11sp3 # I'd still be interested in any comments about what

Re: [RFC PATCH 2/2] Revert "Btrfs: remove transaction from btrfs send"

2014-02-10 Thread Josef Bacik
On 02/08/2014 10:46 AM, Wang Shilong wrote: From: Wang Shilong This reverts commit 41ce9970a8a6a362ae8df145f7a03d789e9ef9d2. Previously i was thinking we can use readonly root's commit root safely while it is not true, readonly root may be cowed with the following cases. 1.snapshot send root

What to do about df and btrfs fi df

2014-02-10 Thread Josef Bacik
Hello, So first of all this is going to get a lot of responses, so straight away I'm only going to consider your opinion if I recognize your name and think you are a sane person. This basically means any big contributors and we'll make sanity exceptions for cwillu. These are just broad stro

Re: system stuck with flush-btrfs-4 at 100% after filesystem resize

2014-02-10 Thread Josef Bacik
On 02/08/2014 01:36 PM, John Navitsky wrote: Hello, I have a large file system that has been growing. We've resized it a couple of times with the following approach: lvextend -L +800G /dev/raid/virtual_machines btrfs filesystem resize +800G /vms I think the FS started out at 200G, we

Re: system stuck with flush-btrfs-4 at 100% after filesystem resize

2014-02-10 Thread John Navitsky
On 2/10/2014 8:43 AM, Josef Bacik wrote: On 02/08/2014 01:36 PM, John Navitsky wrote: romulus:/home/users/johnn # cat /etc/SuSE-release openSUSE 12.3 (x86_64) VERSION = 12.3 CODENAME = Dartmouth romulus:/home/users/johnn # uname -a Linux romulus.us.redacted.com 3.7.10-1.16-desktop #1 SMP PREEM

Re: [PATCH 3/4][RFC] btrfs: export global block reserve size as space_info

2014-02-10 Thread Chris Mason
On 02/07/2014 08:34 AM, David Sterba wrote: Introduce a block group type bit for a global reserve and fill the space info for SPACE_INFO ioctl. This should replace the newly added ioctl (01e219e8069516cdb98594d417b8bb8d906ed30d) to get just the 'size' part of the global reserve, while the actual

Re: What to do about df and btrfs fi df

2014-02-10 Thread Hugo Mills
tl;dr: Yes to proposed df changes. Keep btrfs fi df as-is. On Mon, Feb 10, 2014 at 11:41:51AM -0500, Josef Bacik wrote: [snip] > = What to do moving forward = > > Flip what both of these do. Do not multiply for normal df, and > multiply for btrfs fi df. > > = New and improved df ==

Re: What to do about df and btrfs fi df

2014-02-10 Thread cwillu
I concur. The regular df data used number should be the amount of space required to hold a backup of that content (assuming that the backup maintains reflinks and compression and so forth). There's no good answer for available space; the statfs syscall isn't rich enough to cover all the bases eve

Re: What to do about df and btrfs fi df

2014-02-10 Thread Josef Bacik
On 02/10/2014 01:24 PM, cwillu wrote: I concur. The regular df data used number should be the amount of space required to hold a backup of that content (assuming that the backup maintains reflinks and compression and so forth). There's no good answer for available space; the statfs syscall is

Re: What to do about df and btrfs fi df

2014-02-10 Thread cwillu
IMO, used should definitely include metadata, especially given that we inline small files. I can convince myself both that this implies that we should roll it into b_avail, and that we should go the other way and only report the actual used number for metadata as well, so I might just plead insani

Re: What to do about df and btrfs fi df

2014-02-10 Thread Josef Bacik
On 02/10/2014 01:36 PM, cwillu wrote: IMO, used should definitely include metadata, especially given that we inline small files. I can convince myself both that this implies that we should roll it into b_avail, and that we should go the other way and only report the actual used number for meta

Re: What to do about df and btrfs fi df

2014-02-10 Thread cwillu
>> IMO, used should definitely include metadata, especially given that we >> inline small files. >> >> I can convince myself both that this implies that we should roll it >> into b_avail, and that we should go the other way and only report the >> actual used number for metadata as well, so I might

Re: What to do about df and btrfs fi df

2014-02-10 Thread Hugo Mills
On Mon, Feb 10, 2014 at 01:41:23PM -0500, Josef Bacik wrote: > > > On 02/10/2014 01:36 PM, cwillu wrote: > >IMO, used should definitely include metadata, especially given that we > >inline small files. > > > >I can convince myself both that this implies that we should roll it > >into b_avail, and

Re: [PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Josef Bacik
On 02/10/2014 07:10 AM, Wang Shilong wrote: From: Wang Shilong So i was wandering why test 004 could pass my previous wrong kernel patch while it defenitely should not. By some debugging, i found here perl script is wrong, we did not filter out anything and this unit test did not work acutal

Re: Error: could not do orphan cleanup -22

2014-02-10 Thread Pavel Volkov
On Monday 10 February 2014 00:20:54 you wrote: > There was a similar discussion about an error in January 2013 but it related > to some kernel panic. I don't know if I encountered the same thing. > > These errors from system journal bother me: > > 2月 09 22:18:53 melforce kernel: BTRFS error (dev

Re: [PATCH v2] xfstests: Btrfs: add test for large metadata blocks

2014-02-10 Thread Josef Bacik
On 02/08/2014 03:30 AM, Koen De Wit wrote: Tests Btrfs filesystems with all possible metadata block sizes, by setting large extended attributes on files. Signed-off-by: Koen De Wit --- v1->v2: - Fix indentation: 8 spaces instead of 4 - Move _scratch_unmount to end of loop, add _che

Re: Error: could not do orphan cleanup -22

2014-02-10 Thread Pavel Volkov
Some more update. I checked the FS with btrfsck: checking extents ref mismatch on [17018880 8192] extent item 1, found 2 Incorrect local backref count on 17018880 root 258 owner 826 offset 0 found 2 wanted 1 back 0x961b268 backpointer mismatch on [17018880 8192] ref mismatch on [17027072 8192] ex

Re: Error: could not do orphan cleanup -22

2014-02-10 Thread Josef Bacik
On 02/10/2014 03:53 PM, Pavel Volkov wrote: Some more update. I checked the FS with btrfsck: Build a kernel with this patch applied http://ur1.ca/glslj and re-run the mount and when it fails attach dmesg to this email. Thanks, Josef -- To unsubscribe from this list: send the line "unsubsc

Re: [PATCH v4] xfstests/btrfs: add a regression test for running snapshot and send concurrently

2014-02-10 Thread Josef Bacik
On 02/07/2014 09:00 AM, Wang Shilong wrote: From: Wang Shilong Btrfs would fail to send if snapshot run concurrently, this test is to make sure we have fixed the bug. Looks reasonable, ran it with and without the patch and it did as expected. Reviewed-by: Josef Bacik Thanks, Josef --

Re: [PATCH] xfstests: add test for btrfs data corruption when using compression

2014-02-10 Thread Josef Bacik
On 02/08/2014 10:50 AM, Filipe David Borba Manana wrote: Test for a btrfs data corruption when using compressed files/extents. Under certain cases, it was possible for reads to return random data (content from a previously used page) instead of zeroes. This also caused partial updates to those

Re: [PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Dave Chinner
On Mon, Feb 10, 2014 at 08:10:56PM +0800, Wang Shilong wrote: > From: Wang Shilong > > So i was wandering why test 004 could pass my previous wrong > kernel patch while it defenitely should not. > > By some debugging, i found here perl script is wrong, we did not > filter out anything and this u

[PATCH v3] xfstests: Btrfs: add test for large metadata blocks

2014-02-10 Thread Koen De Wit
Tests Btrfs filesystems with all possible metadata block sizes, by setting large extended attributes on files. Signed-off-by: Koen De Wit --- v1->v2: - Fix indentation: 8 spaces instead of 4 - Move _scratch_unmount to end of loop, add _check_scratch_fs - Sending failure messages of m

Re: [PATCH] xfstests: Btrfs: add test for large metadata blocks

2014-02-10 Thread Koen De Wit
On 02/10/2014 12:02 AM, Dave Chinner wrote: On Sat, Feb 08, 2014 at 09:30:51AM +0100, Koen De Wit wrote: On 02/07/2014 11:49 PM, Dave Chinner wrote: On Fri, Feb 07, 2014 at 06:14:45PM +0100, Koen De Wit wrote: echo -n "$xattr_value" | md5sum ${ATTR_PROG} -Lq -s attr_$char -V $x

Re: Error: could not do orphan cleanup -22

2014-02-10 Thread Pavel Volkov
On Monday 10 February 2014 16:13:40 Josef Bacik wrote: > Build a kernel with this patch applied > > http://ur1.ca/glslj > > and re-run the mount and when it fails attach dmesg to this email. Thanks, I don't see these new messages nor the previous "-22" messages in dmesg now. Only the access pro

Re: [PATCH v3] xfstests: Btrfs: add test for large metadata blocks

2014-02-10 Thread Dave Chinner
On Mon, Feb 10, 2014 at 10:39:22PM +0100, Koen De Wit wrote: > Tests Btrfs filesystems with all possible metadata block sizes, by > setting large extended attributes on files. > > Signed-off-by: Koen De Wit > + > +_test_illegal_leafsize() { > +_scratch_mkfs -l $1 >>$seqres.full 2>&1

Re: What to do about df and btrfs fi df

2014-02-10 Thread Goffredo Baroncelli
On 02/10/2014 06:06 PM, Hugo Mills wrote: >Biggest multiplier leads to the pessimistic estimate, which is what > I'd prefer to see here, so that's good. Agree with this. I would prefer to use as "raid multiplier" the ratio total data block groups + total metadata block group ---

Re: What to do about df and btrfs fi df

2014-02-10 Thread Goffredo Baroncelli
On 02/10/2014 05:41 PM, Josef Bacik wrote: > = New and improved btrfs fi df = > > Since people using this tool are already going to be better informed > and since we are already given the block group flags we can go ahead > and do the raid multiplier in btrfs-progs and spit out the adjuste

Re: What to do about df and btrfs fi df

2014-02-10 Thread cwillu
> In the past [1] I proposed the following approach. > > $ sudo btrfs filesystem df /mnt/btrfs1/ > Disk size: 400.00GB > Disk allocated:8.04GB > Disk unallocated:391.97GB > Used: 11.29MB > Free (Estimated):250.45GB (Max: 396.99GB,

Re: What to do about df and btrfs fi df

2014-02-10 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/02/14 10:24, cwillu wrote: > The regular df data used number should be the amount of space required > to hold a backup of that content (assuming that the backup maintains > reflinks and compression and so forth). > > There's no good answer for

Re: [PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Wang Shilong
Hi Josef, On 02/11/2014 03:18 AM, Josef Bacik wrote: On 02/10/2014 07:10 AM, Wang Shilong wrote: From: Wang Shilong So i was wandering why test 004 could pass my previous wrong kernel patch while it defenitely should not. By some debugging, i found here perl script is wrong, we did not fil

Re: [PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Wang Shilong
On 02/11/2014 05:39 AM, Dave Chinner wrote: On Mon, Feb 10, 2014 at 08:10:56PM +0800, Wang Shilong wrote: From: Wang Shilong So i was wandering why test 004 could pass my previous wrong kernel patch while it defenitely should not. By some debugging, i found here perl script is wrong, we did n

Re: btrfsck does not fix

2014-02-10 Thread Chris Murphy
On Feb 9, 2014, at 1:36 AM, Hendrik Friedel wrote: > > Yes, but I can create that space. > So, for me the next steps would be to: > -generate enough room on the filesystem > -btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/BTRFS/Video > -btrfs device delete /dev/sdc1 /mnt/BTRFS/Video >

Re: [PATCH] xfstests: btrfs/004: fix to make test really work

2014-02-10 Thread Josef Bacik
On 02/10/2014 08:22 PM, Wang Shilong wrote: Hi Josef, On 02/11/2014 03:18 AM, Josef Bacik wrote: On 02/10/2014 07:10 AM, Wang Shilong wrote: From: Wang Shilong So i was wandering why test 004 could pass my previous wrong kernel patch while it defenitely should not. By some debugging, i f

Re: BTRFS with RAID1 cannot boot when removing drive

2014-02-10 Thread Chris Murphy
On Feb 9, 2014, at 2:40 PM, Saint Germain wrote: > > Then I added another drive for a RAID1 configuration (with btrfs > balance) and I installed grub on the second hard drive with > "grub-install /dev/sdb". That can't work on UEFI. UEFI firmware effectively requires a GPT partition map and som

Re: btrfsck does not fix

2014-02-10 Thread Chris Murphy
On Feb 10, 2014, at 6:45 PM, Chris Murphy wrote: > > On Feb 9, 2014, at 1:36 AM, Hendrik Friedel wrote: >> >> Yes, but I can create that space. >> So, for me the next steps would be to: >> -generate enough room on the filesystem >> -btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/BTR

Re: BTRFS with RAID1 cannot boot when removing drive

2014-02-10 Thread Saint Germain
Hello Duncan, What an amazing extensive answer you gave me ! Thank you so much for it. See my comments below. On Mon, 10 Feb 2014 03:34:49 + (UTC), Duncan <1i5t5.dun...@cox.net> wrote : > > I am experimenting with BTRFS and RAID1 on my Debian Wheezy (with > > backported kernel 3.12-0.bpo.1-

Re: What to do about df and btrfs fi df

2014-02-10 Thread cwillu
On Mon, Feb 10, 2014 at 7:02 PM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 10/02/14 10:24, cwillu wrote: >> The regular df data used number should be the amount of space required >> to hold a backup of that content (assuming that the backup maintains >> reflinks a

Re: BTRFS with RAID1 cannot boot when removing drive

2014-02-10 Thread Saint Germain
Hello ! On Mon, 10 Feb 2014 19:18:22 -0700, Chris Murphy wrote : > > On Feb 9, 2014, at 2:40 PM, Saint Germain wrote: > > > > Then I added another drive for a RAID1 configuration (with btrfs > > balance) and I installed grub on the second hard drive with > > "grub-install /dev/sdb". > > That

Re: What to do about df and btrfs fi df

2014-02-10 Thread ronnie sahlberg
On Mon, Feb 10, 2014 at 7:13 PM, cwillu wrote: > On Mon, Feb 10, 2014 at 7:02 PM, Roger Binns wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 10/02/14 10:24, cwillu wrote: >>> The regular df data used number should be the amount of space required >>> to hold a backup of that c

Re: system stuck with flush-btrfs-4 at 100% after filesystem resize

2014-02-10 Thread Duncan
John Navitsky posted on Mon, 10 Feb 2014 07:35:32 -0800 as excerpted: [I rearranged your upside-down posting so the reply comes in context after the quote.] > On 2/8/2014 10:36 AM, John Navitsky wrote: >> I have a large file system that has been growing. We've resized it a >> couple of times w

Re: BTRFS with RAID1 cannot boot when removing drive

2014-02-10 Thread Duncan
Saint Germain posted on Tue, 11 Feb 2014 04:15:27 +0100 as excerpted: > Ok I need to really understand how my motherboard works (new Z87E-ITX). > It is written "64Mb AMI UEFI Legal BIOS", so I thought it was really > UEFI. I expect it's truly UEFI. But from what I've read most UEFI based firmwa

Re: BTRFS with RAID1 cannot boot when removing drive

2014-02-10 Thread Duncan
Saint Germain posted on Tue, 11 Feb 2014 04:15:27 +0100 as excerpted: > I understand. Normally the swap will only be used for hibernating. I > don't expect to use it except perhaps in some extreme case. If hibernate is your main swap usage, you might consider the noauto fstab option as well, the