[PATCH v6 3/8] btrfs-progs: dedupe: Add disable support for inband dedupelication

2016-03-22 Thread Qu Wenruo
Add disable subcommand for dedupe command group. Signed-off-by: Qu Wenruo --- Documentation/btrfs-dedupe.asciidoc | 5 + btrfs-completion| 2 +- cmds-dedupe.c | 42 + 3 files changed, 48 insertions(+), 1 deletion

[PATCH v6 2/8] btrfs-progs: dedupe: Add enable command for dedupe command group

2016-03-22 Thread Qu Wenruo
Add enable subcommand for dedupe commmand group. Signed-off-by: Qu Wenruo --- Documentation/btrfs-dedupe.asciidoc | 105 +++- btrfs-completion| 6 +- cmds-dedupe.c | 155 ioctl.h

[PATCH v6 1/8] btrfs-progs: Basic framework for dedupe command group

2016-03-22 Thread Qu Wenruo
Add basic ioctl header and command group framework for later use. Alone with basic man page doc. Signed-off-by: Qu Wenruo --- Documentation/Makefile.in | 1 + Documentation/btrfs-dedupe.asciidoc | 39 ++ Documentation/btrfs.asciidoc| 4 Makefi

[PATCH v6 5/8] btrfs-progs: Add dedupe feature for mkfs and convert

2016-03-22 Thread Qu Wenruo
Add new DEDUPE ro compat flag and corresponding mkfs/convert flag 'dedupe'. Since dedupe tree is completely isolated from fs tree, so even old kernel could do read mount. So add it to RO compat flag instead of common incompat flags Signed-off-by: Qu Wenruo --- Documentation/mkfs.btrfs.asciidoc

[PATCH v6 6/8] btrfs-progs: Add show-super support for new DEDUPE flag

2016-03-22 Thread Qu Wenruo
Now btrfs-show-super can handle DEDUPE ro compat flag. Signed-off-by: Qu Wenruo --- cmds-inspect-dump-super.c | 18 ++ 1 file changed, 18 insertions(+) diff --git a/cmds-inspect-dump-super.c b/cmds-inspect-dump-super.c index 3e09ee8..6a939c9 100644 --- a/cmds-inspect-dump-super.

[PATCH v6 4/8] btrfs-progs: dedupe: Add status subcommand

2016-03-22 Thread Qu Wenruo
Add status subcommand for dedupe command group. Signed-off-by: Qu Wenruo --- Documentation/btrfs-dedupe.asciidoc | 3 ++ btrfs-completion| 2 +- cmds-dedupe.c | 84 + 3 files changed, 88 insertions(+), 1 deletion(-)

[PATCH v6 8/8] btrfs-progs: property: add a dedupe property

2016-03-22 Thread Qu Wenruo
From: Wang Xiaoguang Normally if we enable online dedupe for a fs, it's filesystem wide de-duplication. With this property, we can explicitly disable data de-duplication for specified files. Signed-off-by: Wang Xiaoguang --- Documentation/btrfs-property.asciidoc | 2 + props.c

[PATCH v6 7/8] btrfs-progs: debug-tree: Add dedupe tree support

2016-03-22 Thread Qu Wenruo
Add dedupe tree support for btrfs-debug-tree. Signed-off-by: Qu Wenruo --- cmds-inspect-dump-tree.c | 4 ++ ctree.h | 7 print-tree.c | 105 +++ 3 files changed, 116 insertions(+) diff --git a/cmds-inspect-dump-tr

csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Hello! Since one of the last kernel updates (I don't know which exactly), I'm experiencing csum errors within VDI files when running VirtualBox. A side effect of this is, as soon as dmesg shows these errors, commands like "du" and "df" hang until reboot. I've now restored the file from backup but

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Am Tue, 22 Mar 2016 09:03:42 +0100 schrieb Kai Krakow : > Hello! > > Since one of the last kernel updates (I don't know which exactly), I'm > experiencing csum errors within VDI files when running VirtualBox. A > side effect of this is, as soon as dmesg shows these errors, commands > like "du" an

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Qu Wenruo
Hi, Kai Krakow wrote on 2016/03/22 09:03 +0100: Hello! Since one of the last kernel updates (I don't know which exactly), I'm experiencing csum errors within VDI files when running VirtualBox. A side effect of this is, as soon as dmesg shows these errors, commands like "du" and "df" hang until

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Am Tue, 22 Mar 2016 09:03:42 +0100 schrieb Kai Krakow : > Hello! > > Since one of the last kernel updates (I don't know which exactly), I'm > experiencing csum errors within VDI files when running VirtualBox. A > side effect of this is, as soon as dmesg shows these errors, commands > like "du" an

Re: [PATCH] Btrfs: fix invalid reference in replace_path

2016-03-22 Thread David Sterba
On Mon, Mar 21, 2016 at 02:59:53PM -0700, Liu Bo wrote: > Dan Carpenter's static checker has found this error, it's introduced by > commit 64c043de466d > ("Btrfs: fix up read_tree_block to return proper error") > > It's really supposed to 'break' the loop on error like others. > > Cc: Dan Carpent

Re: [PATCH] btrfs: fix build warning

2016-03-22 Thread Geert Uytterhoeven
On Tue, Feb 16, 2016 at 9:02 AM, Sudip Mukherjee wrote: > We were getting build warning about: > fs/btrfs/extent-tree.c:7021:34: warning: ‘used_bg’ may be used > uninitialized in this function > > It is not a valid warning as used_bg is never used uninitilized since > locked is initially f

Re: [PATCH] btrfs: fix build warning

2016-03-22 Thread David Sterba
On Tue, Mar 22, 2016 at 10:39:59AM +0100, Geert Uytterhoeven wrote: > On Tue, Feb 16, 2016 at 9:02 AM, Sudip Mukherjee > wrote: > > We were getting build warning about: > > fs/btrfs/extent-tree.c:7021:34: warning: ‘used_bg’ may be used > > uninitialized in this function > > > > It is not a

Re: [PULL] Btrfs, misc for 4.6

2016-03-22 Thread David Sterba
On Mon, Mar 21, 2016 at 07:42:09PM +0100, David Sterba wrote: > Hi, > > a few low-risk patches that appeared after I sent my main pull for 4.6. I > think > they still apply to 4.6, the freezer patches help the livepatch to clean up > the > mis-use of the freezer API, the rest makes sure we don't

[PATCH] btrfs-progs: utils: make sure set_label_mounted uses correct length buffers

2016-03-22 Thread Petros Angelatos
When `btrfs filesystem label /foo bar` command is invoked, it will pass the buffer allocated in the argv array directly to set_label_mounted() and then to the BTRFS_IOC_SET_FSLABEL ioctl. However, the kernel code handling the ioctl will always try to copy BTRFS_LABEL_SIZE bytes[1] from the userlan

Re: [PATCH V15 00/15] Btrfs: Subpagesize-blocksize: Allow I/O on blocks whose size is less than page size

2016-03-22 Thread David Sterba
On Thu, Feb 11, 2016 at 11:17:38PM +0530, Chandan Rajendra wrote: > this patchset temporarily disables the commit > f82c458a2c3ffb94b431fc6ad791a79df1b3713e. > > The commits for the Btrfs kernel module can be found at > https://github.com/chandanr/linux/tree/btrfs/subpagesize-blocksize. The branc

Re: [PATCH v2] fstests: test fsync on overlayfs merged directory

2016-03-22 Thread Filipe Manana
On Tue, Mar 22, 2016 at 3:00 AM, Eryu Guan wrote: > On Mon, Mar 21, 2016 at 05:50:25PM +, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Test that calling fsync against a file using the merged directory does >> not result in a crash nor fails unexpectedly. >> >> This is motivated by

Re: [PATCH v2] fstests: test fsync on overlayfs merged directory

2016-03-22 Thread Eryu Guan
On Tue, Mar 22, 2016 at 11:07:06AM +, Filipe Manana wrote: > On Tue, Mar 22, 2016 at 3:00 AM, Eryu Guan wrote: ... > >> + > >> +# real QA test starts here > >> +_supported_fs generic > > > > The supported fs should be "overlay"? overlay/001 has the same issue > > though. > > Yeah, I copied it

[PATCH v3] fstests: test fsync on overlayfs merged directory

2016-03-22 Thread fdmanana
From: Filipe Manana Test that calling fsync against a file using the merged directory does not result in a crash nor fails unexpectedly. This is motivated by a change in overlayfs that resulted in a crash (invalid memory access) when the lower or upper directory belonged to a btrfs file system.

Re: [PATCH 13/13] btrfs: optimize check for stale device

2016-03-22 Thread David Sterba
On Fri, Feb 19, 2016 at 03:10:16PM +0800, Anand Jain wrote: > > I see crashes with btrfs/011 on a non-debugging config > > > > [ 641.714363] BUG: unable to handle kernel NULL pointer dereference at > > 0068 > > [ 641.716057] IP: [] scrub_setup_ctx.isra.19+0x1f6/0x260 > > [btrfs] > >

Re: [PATCH V15 00/15] Btrfs: Subpagesize-blocksize: Allow I/O on blocks whose size is less than page size

2016-03-22 Thread Chandan Rajendra
On Tuesday 22 Mar 2016 12:04:23 David Sterba wrote: > On Thu, Feb 11, 2016 at 11:17:38PM +0530, Chandan Rajendra wrote: > > this patchset temporarily disables the commit > > f82c458a2c3ffb94b431fc6ad791a79df1b3713e. > > > > The commits for the Btrfs kernel module can be found at > > https://github

Unable to rescue super-recover and/or restore

2016-03-22 Thread Thomas Kuther
Hi everybody, I could need some advice on recovering a sort of broken filesystem (power cycle death). It seems the first superblock is damaged, but I'm unable to recover from a good one. Running on a livecd with latest kernel and btrfs-tools I could get my hands on, this is all the info I can giv

Re: [PATCH v8 00/27][For 4.7] Btrfs: Add inband (write time) de-duplication framework

2016-03-22 Thread David Sterba
On Tue, Mar 22, 2016 at 09:35:25AM +0800, Qu Wenruo wrote: > This updated version of inband de-duplication has the following features: > 1) ONE unified dedup framework. > 2) TWO different back-end with different trade-off The on-disk format is defined in code, would be good to give some overview h

Re: overlay file to test btrfs repairs

2016-03-22 Thread Austin S. Hemmelgarn
On 2016-03-21 13:13, Chris Murphy wrote: On Mon, Mar 21, 2016 at 5:22 AM, Austin S. Hemmelgarn wrote: On 2016-03-21 05:55, Duncan wrote: Chris Murphy posted on Sun, 20 Mar 2016 21:43:52 -0600 as excerpted: Hi folks, So I just ran into this: https://raid.wiki.kernel.org/index.php/ Recover

Re: [RFC] Experimental btrfs encryption

2016-03-22 Thread David Sterba
On Thu, Mar 03, 2016 at 09:58:53AM +0800, Anand Jain wrote: > . (I received couple of private emails on this, so looks like > I confused you and I'm writing again to clear the air on this). > > > - Uses btrfs compression framework, so compression and then > >encryption is not possible. Howev

Re: Btrfsck memory usage reduce idea

2016-03-22 Thread David Sterba
On Mon, Mar 21, 2016 at 10:15:55AM +0800, Qu Wenruo wrote: > > IOW, there will be two options for the use to choose from, right? That's > > what I'd expect. Be able to check the filesystem on a machine with less > > memory at the cost of IO, but also do the faster check on a different > > machine.

Re: [PATCH] btrfs-progs: build: fix static standalone utilities

2016-03-22 Thread David Sterba
On Mon, Mar 21, 2016 at 10:16:07AM -0400, Noah Massey wrote: > commit b5e7979 "btrfs-progs: build: extend per-binary objects" allows > the standalone utilities to link against object files shared with the > main binary. However, the btrfs-*.static targets need to be adjusted > to build against the

Re: [PATCH] btrfs-progs: utils: make sure set_label_mounted uses correct length buffers

2016-03-22 Thread David Sterba
On Tue, Mar 22, 2016 at 03:40:29AM -0700, Petros Angelatos wrote: > When `btrfs filesystem label /foo bar` command is invoked, it will pass > the buffer allocated in the argv array directly to set_label_mounted() > and then to the BTRFS_IOC_SET_FSLABEL ioctl. > > However, the kernel code handling

Re: [PATCH 0/6 v3] btrfs-progs: subvolume functions reorg

2016-03-22 Thread David Sterba
On Mon, Mar 21, 2016 at 03:20:59PM +0800, Anand Jain wrote: > We need subvolume helper functions easily accessible for features > around subvolume. This patch set is just a cleanup of subvolume > functions. > > This is tested fine with fstests group subvol and subvol hand tests. > > v3: Separate

Re: [PATCH 13/13] btrfs: optimize check for stale device

2016-03-22 Thread Anand Jain
On 03/22/2016 08:21 PM, David Sterba wrote: On Fri, Feb 19, 2016 at 03:10:16PM +0800, Anand Jain wrote: I see crashes with btrfs/011 on a non-debugging config [ 641.714363] BUG: unable to handle kernel NULL pointer dereference at 0068 [ 641.716057] IP: [] scrub_setup_ctx.isra.1

[PATCH] btrfs-progs: fix build of standalone utilities after clean

2016-03-22 Thread David Sterba
$ make clean $ make btrfs-debug-tree will fail because the dependency from $(btrfs_debug_tree_objects) is missing. The variable standalone_deps magically collects all the deps and will build them in advance. The simple fix to use the existing substitution based on $@ does not work for pattern rule

Re: overlay file to test btrfs repairs

2016-03-22 Thread Duncan
Austin S. Hemmelgarn posted on Tue, 22 Mar 2016 10:21:57 -0400 as excerpted: > OTOH, if we could add some way to tell the code (both userspace and > in-kernel) to explicitly ignore specific devices when trying to assemble > filesystems, that would allow us to use DM snapshots (or something > simil

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Am Tue, 22 Mar 2016 16:47:10 +0800 schrieb Qu Wenruo : > Hi, > > Kai Krakow wrote on 2016/03/22 09:03 +0100: > > Hello! > > > > Since one of the last kernel updates (I don't know which exactly), > > I'm experiencing csum errors within VDI files when running > > VirtualBox. A side effect of this i

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Chris Murphy
This is kinda confusing. So the gist is that the guest OS is Windows, so the VDI contains an NTFS file system. Correct? And that VDI is on a Btrfs formatted bcache device. Correct? Does the VDI have +C xattr set? Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btr

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Henk Slager
On Tue, Mar 22, 2016 at 9:03 AM, Kai Krakow wrote: > Hello! > > Since one of the last kernel updates (I don't know which exactly), I'm > experiencing csum errors within VDI files when running VirtualBox. A > side effect of this is, as soon as dmesg shows these errors, commands > like "du" and "df"

RAID Assembly with Missing Empty Drive

2016-03-22 Thread John Marrett
I recently had a drive failure in a file server running btrfs. The failed drive was completely non-functional. I added a new drive to the filesystem successfully, when I attempted to remove the failed drive I encountered an error. I discovered that I actually experienced a dual drive failure, the s

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Am Tue, 22 Mar 2016 13:42:00 -0600 schrieb Chris Murphy : > This is kinda confusing. Err yes maybe... ;-D > So the gist is that the guest OS is Windows, so the VDI contains an > NTFS file system. Correct? And that VDI is on a Btrfs formatted bcache > device. Correct? Does the VDI have +C xattr s

Re: overlay file to test btrfs repairs

2016-03-22 Thread Henk Slager
On Mon, Mar 21, 2016 at 4:43 AM, Chris Murphy wrote: > Hi folks, > > So I just ran into this: > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file > > This is a device mapper overlay file - not overlayfs. > > For the repair

Re: RAID Assembly with Missing Empty Drive

2016-03-22 Thread Henk Slager
On Tue, Mar 22, 2016 at 9:19 PM, John Marrett wrote: > I recently had a drive failure in a file server running btrfs. The > failed drive was completely non-functional. I added a new drive to the I asume you did btrfs device add ? Or did you do this withbtrfs replace ? > filesystem succ

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Kai Krakow
Am Tue, 22 Mar 2016 21:07:35 +0100 schrieb Henk Slager : > On Tue, Mar 22, 2016 at 9:03 AM, Kai Krakow > wrote: > > Hello! > > > > Since one of the last kernel updates (I don't know which exactly), > > I'm experiencing csum errors within VDI files when running > > VirtualBox. A side effect of thi

Re: RAID Assembly with Missing Empty Drive

2016-03-22 Thread John Marrett
After further discussion in #btrfs: I left out the raid level, it's raid1: ubuntu@ubuntu:~$ sudo btrfs filesystem df /mnt Data, RAID1: total=6.04TiB, used=5.46TiB System, RAID1: total=32.00MiB, used=880.00KiB Metadata, RAID1: total=14.00GiB, used=11.59GiB GlobalReserve, single: total=512.00MiB, u

Re: RAID Assembly with Missing Empty Drive

2016-03-22 Thread John Marrett
Henk, > I asume you did btrfs device add ? > Or did you do this withbtrfs replace ? Just realised I missed this question, sorry, I performed an add followed by a (failed) delete. -JohnF > >> filesystem successfully, when I attempted to remove the failed drive I >> encountered an error

Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes

2016-03-22 Thread Liu Bo
On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote: > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > > This is to test if COW enabled btrfs can end up with single 4k extents > > when doing subpagesize buffered writes. > > What happens if btrfs is mounted with "nodatacow" option?

Re: [PATCH] fstests: _fail the tests if _scratch_mount failed to avoid fully filling root fs

2016-03-22 Thread Dave Chinner
On Mon, Mar 21, 2016 at 03:23:41PM +0800, Eryu Guan wrote: > btrfs failed to mount small fs on ppc64 host with error ENOSPC, even > creating such small fs succeeded, then generic/027 consumed all free > space on root fs not on SCRATCH_DEV and test harness cannot create tmp > files and continue othe

Re: Btrfsck memory usage reduce idea

2016-03-22 Thread Qu Wenruo
David Sterba wrote on 2016/03/22 15:49 +0100: On Mon, Mar 21, 2016 at 10:15:55AM +0800, Qu Wenruo wrote: IOW, there will be two options for the use to choose from, right? That's what I'd expect. Be able to check the filesystem on a machine with less memory at the cost of IO, but also do the fa

Re: moving btrfs subvolumes to new disk

2016-03-22 Thread Ryan Erato
Finally got around to running the suggested commands. Same error with the send, but not much output to help. The check operation did seem to reveal some potential issues. Here's the play-by-play along with the file output from check: [liveuser@localhost /]$ sudo btrfs check /dev/sda6 > /home/live

reflink copy 'invalid argument'

2016-03-22 Thread Chris Murphy
kernel is 4.5.0 The source is on one subvolume, destination on another subvolume, the full path from top-level id5 is being used for the copy. I've not previously seen this error before so I'm not sure what's going on. The suspicious part to me in the strace is this line: ioctl(4, BTRFS_IOC_CLONE

Re: reflink copy 'invalid argument'

2016-03-22 Thread Chris Murphy
OK OK, Mr. Short Term Memory here. The target images/ directory has +C xattr. When removed the problem doesn't happen. I've run into this before, and asked about it on the list before. So yeah... Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body o

Re: [PATCH v8 00/27][For 4.7] Btrfs: Add inband (write time) de-duplication framework

2016-03-22 Thread Qu Wenruo
Thank you for your interest in dedupe patchset first. In fact I'm quite afraid if there is no one interest in the patchset, it may be delayed again to 4.8. David Sterba wrote on 2016/03/22 14:38 +0100: On Tue, Mar 22, 2016 at 09:35:25AM +0800, Qu Wenruo wrote: This updated version of inband

Re: [PATCH] fstests: _fail the tests if _scratch_mount failed to avoid fully filling root fs

2016-03-22 Thread Eryu Guan
On Wed, Mar 23, 2016 at 11:08:56AM +1100, Dave Chinner wrote: > On Mon, Mar 21, 2016 at 03:23:41PM +0800, Eryu Guan wrote: > > btrfs failed to mount small fs on ppc64 host with error ENOSPC, even > > creating such small fs succeeded, then generic/027 consumed all free > > space on root fs not on SC

RAID-1 refuses to balance large drive

2016-03-22 Thread Brad Templeton
I have a RAID 1, and was running a bit low, so replaced a 2TB drive with a 6TB. The other drives are a 3TB and a 4TB.After switching the drive, I did a balance and ... essentially nothing changed. It did not balance clusters over to the 6TB drive off of the other 2 drives. I found it odd, an

Re: moving btrfs subvolumes to new disk

2016-03-22 Thread Chris Murphy
On Tue, Mar 22, 2016 at 7:40 PM, Ryan Erato wrote: > Finally got around to running the suggested commands. Same error with > the send, but not much output to help. The check operation did seem > to reveal some potential issues. Here's the play-by-play along with > the file output from check: > >

[PATCH] btrfs: Output more info for enospc_debug mount option

2016-03-22 Thread Qu Wenruo
As one user in mail list report reproducible balance ENOSPC error, it's better to add more debug info for enospc_debug mount option. Reported-by: Marc Haber Signed-off-by: Qu Wenruo --- changelog: v2: Add output for block group bytenr --- fs/btrfs/extent-tree.c | 21 +++-- 1 f

Re: [PATCH 06/12] xfs/030: fix output on newer filesystems

2016-03-22 Thread Dave Chinner
On Sat, Mar 05, 2016 at 12:20:50PM -0800, Christoph Hellwig wrote: > Still fails for me: > > --- tests/xfs/030.out 2016-03-03 07:55:58.556427678 + > +++ /root/xfstests/results//xfs/030.out.bad 2016-03-05 20:20:17.561433837 > + > @@ -231,8 +231,6 @@ > bad agbno AGBNO in agfl, agno 0 >

Re: RAID-1 refuses to balance large drive

2016-03-22 Thread Qu Wenruo
Brad Templeton wrote on 2016/03/22 17:47 -0700: I have a RAID 1, and was running a bit low, so replaced a 2TB drive with a 6TB. The other drives are a 3TB and a 4TB.After switching the drive, I did a balance and ... essentially nothing changed. It did not balance clusters over to the 6TB

Re: csum errors in VirtualBox VDI files

2016-03-22 Thread Qu Wenruo
Kai Krakow wrote on 2016/03/22 19:48 +0100: Am Tue, 22 Mar 2016 16:47:10 +0800 schrieb Qu Wenruo : Hi, Kai Krakow wrote on 2016/03/22 09:03 +0100: Hello! Since one of the last kernel updates (I don't know which exactly), I'm experiencing csum errors within VDI files when running VirtualBox

Re: RAID-1 refuses to balance large drive

2016-03-22 Thread Brad Templeton
That's rather counter intuitive behaviour. In most FSs, resizes are needed when you do things like change the size of an underlying partition, or you weren't using all the partition. When you add one drive with device add, and you then remove another with device delete, why and how would the adde

Re: RAID-1 refuses to balance large drive

2016-03-22 Thread Chris Murphy
On Tue, Mar 22, 2016 at 10:47 PM, Brad Templeton wrote: > That's rather counter intuitive behaviour. In most FSs, resizes are > needed when you do things like change the size of an underlying > partition, or you weren't using all the partition. When you add one > drive with device add, and you t

Re: RAID-1 refuses to balance large drive

2016-03-22 Thread Chris Murphy
On Tue, Mar 22, 2016 at 11:54 PM, Brad Templeton wrote: > Actually, the URL suggests that all the space will be used, which is > what I had read about btrfs, that it handled this. It will. But it does this by dominating writes to the devices that have the most free space, until all devices have t