Re: [PATCH v3] xfstests: btrfs: add test case for qgroup account on shared extents

2014-12-19 Thread Liu Bo
On Thu, Dec 18, 2014 at 11:05:30AM +1100, Dave Chinner wrote: On Wed, Dec 17, 2014 at 04:30:47PM +0800, Liu Bo wrote: This is a regression test of 'commit fcebe4562dec (Btrfs: rework qgroup accounting)' It can produce qgroup related warnings. The fix is

[PATCH v4] xfstests: btrfs: add test case for qgroup account on shared extents

2014-12-19 Thread Liu Bo
This is a regression test of 'commit fcebe4562dec (Btrfs: rework qgroup accounting)', it's used to verify that removing shared extents can end up incorrect qgroup accounting. It can produce qgroup related warnings. The fix is https://patchwork.kernel.org/patch/5499981/ Btrfs: fix a warning of

Re: Oddly slow read performance with near-full largish FS

2014-12-19 Thread Satoru Takeuchi
Hi, Sorry for late reply. Let me ask some questions. On 2014/12/17 11:42, Charles Cazabon wrote: Hi, I've been running btrfs for various filesystems for a few years now, and have recently run into problems with a large filesystem becoming *really* slow for basic reading. None of the

Re: [PATCH v4] xfstests: btrfs: add test case for qgroup account on shared extents

2014-12-19 Thread Satoru Takeuchi
Hi Liu, On 2014/12/19 17:31, Liu Bo wrote: This is a regression test of 'commit fcebe4562dec (Btrfs: rework qgroup accounting)', it's used to verify that removing shared extents can end up incorrect qgroup accounting. It can produce qgroup related warnings. The fix is

Re: [PATCH 0/5] Cleanup warnings from clang

2014-12-19 Thread Satoru Takeuchi
Hi, On 2014/12/19 15:13, Qu Wenruo wrote: Cleanup warning when compile btrfs-progs with clang. Clang analyser also reports about 60+ errors, but it will be another patchset fixing it later. Qu Wenruo (5): btrfs-progs: Makefile: Move linker only option to LDFLAGS btrfs-progs: Fix a

Re: [PATCH v4] xfstests: btrfs: add test case for qgroup account on shared extents

2014-12-19 Thread Liu Bo
Hi Satoru san, On Fri, Dec 19, 2014 at 06:21:30PM +0900, Satoru Takeuchi wrote: Hi Liu, On 2014/12/19 17:31, Liu Bo wrote: This is a regression test of 'commit fcebe4562dec (Btrfs: rework qgroup accounting)', it's used to verify that removing shared extents can end up incorrect qgroup

Re: [PATCH 3/6] btrfs-progs: fi usage, update manpage

2014-12-19 Thread Satoru Takeuchi
Hi David, On 2014/12/18 23:27, David Sterba wrote: Signed-off-by: David Sterba dste...@suse.cz --- Documentation/btrfs-filesystem.txt | 28 1 file changed, 28 insertions(+) diff --git a/Documentation/btrfs-filesystem.txt

[PATCH] Btrfs: generic checksum framework

2014-12-19 Thread Liu Bo
This changes the original crc32c specific checksum functions into more generic ones, so that converting to a new checksum algorithm can be transparent to btrfs internal code. Note that file names' lookup and extent_data_ref's hashing use crc32c with their own seed instead of the default ~0, so

Re: [PATCH 3/6] btrfs-progs: fi usage, update manpage

2014-12-19 Thread David Sterba
On Fri, Dec 19, 2014 at 06:56:43PM +0900, Satoru Takeuchi wrote: +There are some options to set unit. See the description of *df* subcommand +from '-b' option to '-t' option. The unit options exist only for very few subcommands so I found it more convenient to list all of them near to the

Re: btrfs-progs: integration-20141218 possible corruption test regression

2014-12-19 Thread David Sterba
On Fri, Dec 19, 2014 at 02:23:12PM +0800, Qu Wenruo wrote: In fact, it's not a regression. The 013 testcase is a special case that uses a script to corrupt the image and then do the btrfsck test. There is a patch before the commit, to allow btrfs-progs test script call corruption

Re: [PATCH 0/5] Cleanup warnings from clang

2014-12-19 Thread David Sterba
On Fri, Dec 19, 2014 at 06:27:36PM +0900, Satoru Takeuchi wrote: Qu Wenruo (5): btrfs-progs: Makefile: Move linker only option to LDFLAGS btrfs-progs: Fix a clang dead-judgement warning in disk-io.c. btrfs-progs: Remove a unused function root_gtp_mask(). btrfs-progs: Remove a

[GIT PULL] Btrfs pull part two

2014-12-19 Thread Chris Mason
Hi Linus, Please pull my for-linus branch: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus It has part two of our merge window patches. These are all from Filipe, and fix some really hard to find races that can cause corruptions. Most of them involved block

[PATCH 0/6] Btrfs progs, coverity fixes for 3.18

2014-12-19 Thread David Sterba
A few straightforward fixes. David Sterba (6): btrfs-progs: corrupt block, add missing break to option I btrfs-progs: corrupt block, add break after option U btrfs-progs: fragments, close output file on error btrfs-progs: check result of first_cache_extent btrfs-progs: check allocation

[PATCH 1/6] btrfs-progs: corrupt block, add missing break to option I

2014-12-19 Thread David Sterba
Using -I would imply -d. Resolves-Coverity-CID: 1258792 Signed-off-by: David Sterba dste...@suse.cz --- btrfs-corrupt-block.c | 1 + 1 file changed, 1 insertion(+) diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c index af9ae4d4047c..aeeb1b298f66 100644 --- a/btrfs-corrupt-block.c +++

[PATCH 2/6] btrfs-progs: corrupt block, add break after option U

2014-12-19 Thread David Sterba
Resolves-Coverity-CID: 1258793 Signed-off-by: David Sterba dste...@suse.cz --- btrfs-corrupt-block.c | 1 + 1 file changed, 1 insertion(+) diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c index aeeb1b298f66..b477e878376b 100644 --- a/btrfs-corrupt-block.c +++ b/btrfs-corrupt-block.c @@

[PATCH 4/6] btrfs-progs: check result of first_cache_extent

2014-12-19 Thread David Sterba
If the tree's empty, we'll get NULL and dereference it. Resolves-Coverity-CID: 1248828 Signed-off-by: David Sterba dste...@suse.cz --- cmds-check.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cmds-check.c b/cmds-check.c index 6eea36c2f52c..3e7a4ebdce44 100644 --- a/cmds-check.c +++

[PATCH 6/6] btrfs-progs: let btrfs_free_path accept NULL

2014-12-19 Thread David Sterba
Same in kernel and matches semantics of free(). Resolves-Coverity-CID: 1054881 Signed-off-by: David Sterba dste...@suse.cz --- ctree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ctree.c b/ctree.c index bd6cb125b2a2..589efa3db17e 100644 --- a/ctree.c +++ b/ctree.c @@ -48,6 +48,8 @@

[PATCH 5/6] btrfs-progs: check allocation result in add_clone_source

2014-12-19 Thread David Sterba
Resolves-Coverity-CID: 1054894 Signed-off-by: David Sterba dste...@suse.cz --- cmds-send.c | 25 + 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/cmds-send.c b/cmds-send.c index b17b5e2ca666..9b32c1f0e624 100644 --- a/cmds-send.c +++ b/cmds-send.c @@

[PATCH 3/6] btrfs-progs: fragments, close output file on error

2014-12-19 Thread David Sterba
Resolves-Coverity-CID: 1258794 Signed-off-by: David Sterba dste...@suse.cz --- btrfs-fragments.c | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/btrfs-fragments.c b/btrfs-fragments.c index d03c2c3e7319..360f10f87bfa 100644 --- a/btrfs-fragments.c +++ b/btrfs-fragments.c

Re: btrfs receive being very slow

2014-12-19 Thread Nick Dimov
Hello. So I split the job in 2 tasks as per your suggestion. I create the differential snapshot with btrfs send and save it on SSD - so far this is very efficient and the sending happens almost at full SSD speed. When I try to receive the snapshot on the HDD - the speed is just as low as before

Re: [PATCH 2/6] btrfs-progs: corrupt block, add break after option U

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: Resolves-Coverity-CID: 1258793 Signed-off-by: David Sterba dste...@suse.cz Reviewed-by: Eric Sandeen sand...@redhat.com --- btrfs-corrupt-block.c | 1 + 1 file changed, 1 insertion(+) diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c

Re: [PATCH 1/6] btrfs-progs: corrupt block, add missing break to option I

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: Using -I would imply -d. Resolves-Coverity-CID: 1258792 Signed-off-by: David Sterba dste...@suse.cz Reviewed-by: Eric Sandeen sand...@redhat.com --- btrfs-corrupt-block.c | 1 + 1 file changed, 1 insertion(+) diff --git

Re: [PATCH 3/6] btrfs-progs: fragments, close output file on error

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: Resolves-Coverity-CID: 1258794 Signed-off-by: David Sterba dste...@suse.cz Reviewed-by: Eric Sandeen sand...@redhat.com --- btrfs-fragments.c | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/btrfs-fragments.c

Re: Oddly slow read performance with near-full largish FS

2014-12-19 Thread Charles Cazabon
Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com wrote: Let me ask some questions. Sure - thanks for taking an interest. On 2014/12/17 11:42, Charles Cazabon wrote: There's roughly 16TB of data in this filesystem (the filesystem is ~17TB). The btrfs filesystem is a simple single volume, no

Re: [PATCH 4/6] btrfs-progs: check result of first_cache_extent

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: If the tree's empty, we'll get NULL and dereference it. Hm, but this is under an explicit check for not empty: while (!cache_tree_empty(roots_info_cache)) { sooo? Maybe it's just defensive? Nothing really wrong with being defensive, I

Re: [PATCH 5/6] btrfs-progs: check allocation result in add_clone_source

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: Resolves-Coverity-CID: 1054894 Signed-off-by: David Sterba dste...@suse.cz Reviewed-by: Eric Sandeen sand...@redhat.com --- cmds-send.c | 25 + 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/cmds-send.c

Re: [PATCH 6/6] btrfs-progs: let btrfs_free_path accept NULL

2014-12-19 Thread Eric Sandeen
On 12/19/14 10:06 AM, David Sterba wrote: Same in kernel and matches semantics of free(). Resolves-Coverity-CID: 1054881 Signed-off-by: David Sterba dste...@suse.cz Reviewed-by: Eric Sandeen sand...@redhat.com --- ctree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ctree.c

Re: btrfs-progs: integration-20141218 possible corruption test regression

2014-12-19 Thread WorMzy Tykashi
On 19 December 2014 at 13:48, David Sterba dste...@suse.cz wrote: On Fri, Dec 19, 2014 at 02:23:12PM +0800, Qu Wenruo wrote: In fact, it's not a regression. The 013 testcase is a special case that uses a script to corrupt the image and then do the btrfsck test. There is a patch before the

Re: [PATCH 4/6] btrfs-progs: check result of first_cache_extent

2014-12-19 Thread David Sterba
On Fri, Dec 19, 2014 at 10:56:41AM -0600, Eric Sandeen wrote: On 12/19/14 10:06 AM, David Sterba wrote: If the tree's empty, we'll get NULL and dereference it. Hm, but this is under an explicit check for not empty: while (!cache_tree_empty(roots_info_cache)) { sooo? Maybe it's

Re: Oddly slow read performance with near-full largish FS

2014-12-19 Thread Duncan
Charles Cazabon posted on Fri, 19 Dec 2014 10:58:49 -0600 as excerpted: This configuration is one I've been using for many years. It's only recently that I've noticed it being particularly slow with btrfs -- I don't know if that's because the filesystem has filled up past some critical

[PULL] [PATCH 0/4] Updates in message levels

2014-12-19 Thread David Sterba
This is motivated by the ERR level when skinny metadata are used, this has been reported several times. Patch tagged for stable. The rest is taken from a SLES patch that I forgot to forward upstream. You can pull the branch from git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git

[PATCH 4/4] btrfs: set proper message level for skinny metadata

2014-12-19 Thread David Sterba
This has been confusing people for too long, the message is really just informative. CC: sta...@vger.kernel.org # 3.10+ Signed-off-by: David Sterba dste...@suse.cz --- fs/btrfs/disk-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c

[PATCH 3/4] btrfs: update message levels after checksum errors

2014-12-19 Thread David Sterba
The errors are worth noting and might get missed with INFO level. Signed-off-by: David Sterba dste...@suse.cz --- fs/btrfs/disk-io.c | 2 +- fs/btrfs/inode.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index

[PATCH 1/4] btrfs: update message levels for errors

2014-12-19 Thread David Sterba
Several messages that point to some internal problem, level INFO is wrong here. Signed-off-by: David Sterba dste...@suse.cz --- fs/btrfs/disk-io.c | 9 + fs/btrfs/inode.c | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c

[PATCH 2/4] btrfs: update message levels during failed mount

2014-12-19 Thread David Sterba
All error conditions from open_ctree shall be ERR. Warning would suggest that something's wrong and we can continue. Signed-off-by: David Sterba dste...@suse.cz --- fs/btrfs/disk-io.c | 16 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/disk-io.c

Btrfs progs pre-release 3.18-rc2

2014-12-19 Thread David Sterba
Hi another step towards 3.18, the changes are limited in scope and mostly cleanups or docs. I'd like to see more test images for new fsck code, there are 2 new that I missed earlier, so they don't count. The timing with end of the year is not good so if I'm not confident that the release is in a

[PATCH] Btrfs: track dirty block groups on their own list V2

2014-12-19 Thread Josef Bacik
Currently any time we try to update the block groups on disk we will walk _all_ block groups and check for the -dirty flag to see if it is set. This function can get called several times during a commit. So if you have several terabytes of data you will be a very sad panda as we will loop

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Phillip Susi
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/18/2014 9:59 AM, Daniele Testa wrote: As seen above, I have a 410GB SSD mounted at /opt/drives/ssd. On that partition, I have one single starse file, taking 302GB of space (max 315GB). The snapshots directory is completely empty. So you

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Daniele Testa
No, I don't have any snapshots or subvolumes. Only that single file. The file has both checksums and datacow on it. I will do chattr +C on the parent dir and re-create the file to make sure all files are marked as nodatacow. Should I also turn off checksums with the mount-flags if this

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Phillip Susi
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/19/2014 2:59 PM, Daniele Testa wrote: No, I don't have any snapshots or subvolumes. Only that single file. The file has both checksums and datacow on it. I will do chattr +C on the parent dir and re-create the file to make sure all files

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Josef Bacik
On 12/18/2014 09:59 AM, Daniele Testa wrote: Hey, I am hoping you guys can shed some light on my issue. I know that it's a common question that people see differences in the disk used when running different calculations, but I still think that my issue is weird. root@s4 / # mount /dev/md3 on

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Josef Bacik
On 12/19/2014 02:59 PM, Daniele Testa wrote: No, I don't have any snapshots or subvolumes. Only that single file. The file has both checksums and datacow on it. I will do chattr +C on the parent dir and re-create the file to make sure all files are marked as nodatacow. Should I also turn off

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Josef Bacik
On 12/19/2014 04:10 PM, Josef Bacik wrote: On 12/18/2014 09:59 AM, Daniele Testa wrote: Hey, I am hoping you guys can shed some light on my issue. I know that it's a common question that people see differences in the disk used when running different calculations, but I still think that my

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-19 Thread Josef Bacik
On 12/12/2014 09:37 AM, Tomasz Chmielewski wrote: FYI, still seeing this with 3.18 (scrub passes fine on this filesystem). # time btrfs balance start /mnt/lxc2 Segmentation fault Ok now I remember why I haven't fix this yet, the images you gave me restore but then they don't mount because

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Phillip Susi
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/19/2014 4:15 PM, Josef Bacik wrote: Please God don't turn off of checksums. Checksums are tracked in metadata anyway, they won't show up in the data accounting. Our csums are 8 bytes per block, so basic math says you are going to max out

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Josef Bacik
On 12/19/2014 04:53 PM, Phillip Susi wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/19/2014 4:15 PM, Josef Bacik wrote: Please God don't turn off of checksums. Checksums are tracked in metadata anyway, they won't show up in the data accounting. Our csums are 8 bytes per block, so

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-19 Thread Tomasz Chmielewski
On 2014-12-19 22:47, Josef Bacik wrote: On 12/12/2014 09:37 AM, Tomasz Chmielewski wrote: FYI, still seeing this with 3.18 (scrub passes fine on this filesystem). # time btrfs balance start /mnt/lxc2 Segmentation fault Ok now I remember why I haven't fix this yet, the images you gave me

kernel BUG at /home/apw/COD/linux/fs/btrfs/inode.c:3123!

2014-12-19 Thread Tomasz Chmielewski
Get this BUG with 3.18.1 (pasted at the bottom of the email). Below all actions from creating the fs to BUG. I did not attempt to reproduce. # mkfs.btrfs /dev/vdb Btrfs v3.17.3 See http://btrfs.wiki.kernel.org for more information. Turning ON incompat feature 'extref': increased hardlink

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Duncan
Daniele Testa posted on Sat, 20 Dec 2014 03:59:42 +0800 as excerpted: The file has both checksums and datacow on it. I will do chattr +C on the parent dir and re-create the file to make sure all files are marked as nodatacow. Should I also turn off checksums with the mount-flags if this

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Duncan
Josef Bacik posted on Fri, 19 Dec 2014 16:17:08 -0500 as excerpted: tl;dr: Cow means you can in the worst case end up using 2 * filesize - blocksize of data on disk and the file will appear to be filesize. Thanks for the tl;dr /and/ the very sensible longer explanation. That's a very nice

Can BTRFS handle XATTRs larger than 4K?

2014-12-19 Thread Richard Sharpe
Hi folks, I need a Linux file system that supports XATTRs up to 64K. Can BTRFS support that or is XFS the only Linux file system with such support? -- Regards, Richard Sharpe (何以解憂?唯有杜康。--曹操) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Zygo Blaxell
On Fri, Dec 19, 2014 at 04:17:08PM -0500, Josef Bacik wrote: And for your inode you now have this inode 256, file offset 0, size 4k, offset 0, diskebytenr (123+302g), disklen 4k inode 256, file offset 4k, size 302g-4k, offset 4k, diskbytenr 123, disklen 302g and in your extent tree you

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Daniele Testa
But I read somewhere that compression should be turned off on mounts that just store large VM-images. Is that wrong? Btw, I am not pre-allocation space for the images. I use sparse files with: dd if=/dev/zero of=drive.img bs=1 count=1 seek=300G It creates the file in a few ms. Is it better to

Re: btrfs is using 25% more disk than it should

2014-12-19 Thread Duncan
Daniele Testa posted on Sat, 20 Dec 2014 14:18:31 +0800 as excerpted: Anyways, would disabling CoW (by putting +C on the parent dir) prevent the performance issues and 2*filesize issue? It should, provided you don't then start snapshotting the file (which I don't believe you intend to do but