[RFC 00/11] xfstests: test the btrfs/xfs reflink/dedupe ioctls

2015-06-29 Thread Darrick J. Wong
Hi all, This is a RFC-quality pass at making xfstests perform more rigorous testing of the btrfs/xfs file clone, reflink, and dedupe ioctls. There are now tests of the basic functionality of the three ioctls; tests to ensure that the filesystem exhibits the expected copy on write semantics; tests

[PATCH 01/11] fuzz XFS and ext4 filesystems

2015-06-29 Thread Darrick J. Wong
Introduce tests for XFS and ext4 which format a filesystem, populate it, then uses blocktrash and e2fuzz to corrupt the metadata. The FS is remounted, modified, and unmounted. Following that, xfs_repair or e2fsck are run until it no longer finds errors to correct, after which the FS is mounted ye

[PATCH 02/11] move btrfs reflink tests to generic

2015-06-29 Thread Darrick J. Wong
Move the cp --reflink tests from btrfs/ to generic/ since xfs now supports that ioctl. Signed-off-by: Darrick J. Wong --- tests/btrfs/026 | 92 - tests/btrfs/026.out | 16 --- tests/btrfs/027 | 109 ---

[PATCH 03/11] generic/32[6-8]: support xfs in addition to btrfs

2015-06-29 Thread Darrick J. Wong
Modify the reflink tests to support xfs. Signed-off-by: Darrick J. Wong --- common/rc | 37 + tests/generic/800 |2 +- tests/generic/801 |2 +- tests/generic/802 |2 +- 4 files changed, 40 insertions(+), 3 deletions(-) diff --git a/comm

[PATCH 04/11] basic tests of the reflink and dedupe ioctls

2015-06-29 Thread Darrick J. Wong
Test the operation of the btrfs (and now xfs) reflink and dedupe ioctls at various file offsets and with matching and nonmatching files. Signed-off-by: Darrick J. Wong --- common/rc | 44 + tests/generic/803 | 81 +++ tests/generic/

[PATCH 05/11] test CoW behaviors of reflinked files

2015-06-29 Thread Darrick J. Wong
Ensure that CoW happens correctly with buffered, directio, and mmap writes. Signed-off-by: Darrick J. Wong --- tests/generic/808 | 138 + tests/generic/808.out | 16 ++ tests/generic/809 | 138 +++

[PATCH 06/11] reflink fallocate tests

2015-06-29 Thread Darrick J. Wong
Check that the variants of fallocate (allocate, punch, zero range, collapse range, insert range) do the right thing when they're run against a range of reflinked blocks. Signed-off-by: Darrick J. Wong --- tests/generic/811 | 104 +++ tests/generic

[PATCH 07/11] reflink concurrent operations tests

2015-06-29 Thread Darrick J. Wong
Make sure that running reflink ops while other IO is ongoing doesn't break the filesystem. Signed-off-by: Darrick J. Wong --- tests/generic/821 | 95 tests/generic/821.out |6 +++ tests/generic/822 | 95

[PATCH 08/11] test reflink for accuracy in free block counts

2015-06-29 Thread Darrick J. Wong
Check that the free block counts seem to be handled correctly in the reflink operation and subsequent attempts to rewrite reflinked copies. Signed-off-by: Darrick J. Wong --- tests/generic/830 | 88 + tests/generic/830.out |5 + tests/generic/831 | 120

[PATCH 09/11] test error conditions on reflink

2015-06-29 Thread Darrick J. Wong
Check that we can feed bad inputs to reflink and it'll reject them. Signed-off-by: Darrick J. Wong --- tests/generic/839 | 123 + tests/generic/839.out | 30 tests/generic/group |1 3 files changed, 154 insertions(+) cre

[PATCH 10/11] test xfs-specific reflink pieces

2015-06-29 Thread Darrick J. Wong
Check that growfs and xfs_fsr still work properly on reflinked fses. Signed-off-by: Darrick J. Wong --- tests/xfs/800 | 77 tests/xfs/800.out |5 ++ tests/xfs/801 | 114 + tests/xfs/801.out |

[PATCH 11/11] reflink: test what happens when we hit resource limits

2015-06-29 Thread Darrick J. Wong
Signed-off-by: Darrick J. Wong --- tests/generic/840 | 107 + tests/generic/840.out |0 tests/generic/841 | 87 tests/generic/841.out |5 ++ tests/generic/group |2 + 5 files changed

Re: [PATCH 1/2] [btrfs] btrfs_rename: abort transaction in case of error.

2015-06-29 Thread Davide Italiano
On Mon, Jun 29, 2015 at 4:59 AM, Filipe David Manana wrote: > On Sun, Jun 28, 2015 at 10:47 PM, Davide C. C. Italiano > wrote: >> From: Davide Italiano >> >> btrfs_insert_inode_ref() may fail and we want to make sure >> the transaction is aborted before calling btrfs_end_transaction(), >> as it

[PATCH 2/2] [btrfs] btrfs_rename(): don't ignore btrfs_end_transaction() return

2015-06-29 Thread Davide C. C. Italiano
From: Davide Italiano btrfs_end_transaction() can return an error -- this happens, e.g. if it tries to commit and the transaction was aborted in the meanhwile. Swallowing the error is wrong, so explicitly return it. Signed-off-by: Davide Italiano --- fs/btrfs/inode.c | 3 ++- 1 file changed, 2

Re: [PATCH 4/4] btrfs: Fix data checksum error cause by replace with io-load.

2015-06-29 Thread Qu Wenruo
To Chris: Would you consider merging these patchset for late 4.2 merge window? If it's OK to merge it into 4.2 late rc, we'll start our test and send pull request after our test, eta this Friday or next Monday. I know normally we should submit it early especially when such fix is not small. B

[PATCH][RESEND] btrfs: fix search key advancing condition

2015-06-29 Thread Naohiro Aota
The search key advancing condition used in copy_to_sk() is loose. It can advance the key even if it reaches sk->max_*: e.g. when the max key = (512, 1024, -1) and the current key = (512, 1025, 10), it increments the offset by 1, continues hopeless search from (512, 1025, 11). This issue make ioctl(

[GIT PULL] Small cleanup and fix from Fujitsu

2015-06-29 Thread Qu Wenruo
Hi Chris, This is the small cleanup and fixes for 4.2. Compared to the previous pull, this one is quite small, only 2 cleanup and one small quota fix. Yang Dongsheng (1): btrfs: qgroup: allow user to clear the limitation on qgroup Zhao Lei (2): btrfs: cleanup noused initialization of dev

[GIT PULL] Btrfs

2015-06-29 Thread Chris Mason
Hi Linus, Please pull my for-linus-4.2 branch: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus-4.2 Outside of our usual batch of fixes, this integrates the subvolume quota updates that Qu Wenruo from Fujitsu has been working on for a few releases now. He gets an e

Re: [PATCH 5/5] btrfs: don't update mtime on deduped inodes

2015-06-29 Thread Mark Fasheh
On Mon, Jun 29, 2015 at 03:35:02PM -0400, Zygo Blaxell wrote: > On Mon, Jun 29, 2015 at 10:52:41AM -0700, Mark Fasheh wrote: > > On Sat, Jun 27, 2015 at 05:44:28PM -0400, Zygo Blaxell wrote: > > > On Fri, Jun 26, 2015 at 02:01:01PM -0700, Mark Fasheh wrote: > > > > One issue users have reported is

Re: [PATCH 5/5] btrfs: don't update mtime on deduped inodes

2015-06-29 Thread Zygo Blaxell
On Mon, Jun 29, 2015 at 10:52:41AM -0700, Mark Fasheh wrote: > On Sat, Jun 27, 2015 at 05:44:28PM -0400, Zygo Blaxell wrote: > > On Fri, Jun 26, 2015 at 02:01:01PM -0700, Mark Fasheh wrote: > > > One issue users have reported is that dedupe changes mtime on files, > > > resulting in tools like rsyn

Re: [PATCH 5/5] btrfs: don't update mtime on deduped inodes

2015-06-29 Thread Mark Fasheh
On Sat, Jun 27, 2015 at 05:44:28PM -0400, Zygo Blaxell wrote: > On Fri, Jun 26, 2015 at 02:01:01PM -0700, Mark Fasheh wrote: > > One issue users have reported is that dedupe changes mtime on files, > > resulting in tools like rsync thinking that their contents have changed when > > in fact the data

Re: Linux 4.1.0: BTRFS: error (device sdb) in btrfs_run_delayed_refs:2821: errno=-17 Object already exists

2015-06-29 Thread Martin Tippmann
Some more data: After unmounting the read-only mounted partition, this notice appears in the kernel log: VFS: Busy inodes after unmount of sdb. Self-destruct in 5 seconds. Have a nice day... Here is a the btrfs check output.. btrfs check --repair seems to fix the issues: # btrfs check /dev/sdb

Re: [PATCH] Btrfs-progs: add feature to get mininum size for resizing a fs/device

2015-06-29 Thread Filipe Manana
On Mon, Jun 29, 2015 at 4:42 PM, David Sterba wrote: > On Wed, Jun 17, 2015 at 12:44:55PM +0100, fdman...@kernel.org wrote: >> Currently there is not way for a user to know what is the minimum size a >> device of a btrfs filesystem can be resized to. Sometimes the value of >> total allocated space

Re: [PATCH] Btrfs-progs: add feature to get mininum size for resizing a fs/device

2015-06-29 Thread David Sterba
On Wed, Jun 17, 2015 at 12:44:55PM +0100, fdman...@kernel.org wrote: > Currently there is not way for a user to know what is the minimum size a > device of a btrfs filesystem can be resized to. Sometimes the value of > total allocated space (sum of all allocated chunks/device extents), which > can

Linux 4.1.0: BTRFS: error (device sdb) in btrfs_run_delayed_refs:2821: errno=-17 Object already exists

2015-06-29 Thread Martin Tippmann
Hi, is this a know issue and is there a already a patch out there for this? This is also happening with Linux 4.0.4. I can reproduce this expect it's happening under load... Mount options: /dev/sdb /media/storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0 # uname -r 4.1.0-040100-generic # bt

Linux 4.1.0: BTRFS: error (device sdb) in btrfs_run_delayed_refs:2821: errno=-17 Object already exists

2015-06-29 Thread Martin Tippmann
Hi, is this a know issue and is there a already a patch out there for this? This is also happening with Linux 4.0.4. I can reproduce this expect it's happening under load... Mount options: /dev/sdb /media/storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0 # uname -r 4.1.0-040100-generic # bt

Re: btrfs replace seems to corrupt the file system

2015-06-29 Thread Mordechay Kaganer
B.H. Regarding the main issue, the drive that was "recovered" using Noah's trick (mount -o degraded then btrfs replace cancel) appears to be clean. At least, it passes scrub without any errors. It even contains all changes that were made during the replace was ongoing. Also i've run MD's consisten

Re: btrfs replace seems to corrupt the file system

2015-06-29 Thread Mike Fleetwood
On 29 June 2015 at 09:08, Duncan <1i5t5.dun...@cox.net> wrote: > Meanwhile, unlike many filesystems, btrfs uses the UUID as part of the > metadata, so changing the UUID isn't as simple as rewriting a superblock; > the metadata must be rewritten to the new UUID. There's actually a tool > now availa

Re: [PATCH] check: check so offset is not bigger then the leaf

2015-06-29 Thread Trollkarlen Marklund
> On 18 Jun 2015, at 19:44, David Sterba wrote: > > On Thu, Jun 18, 2015 at 01:59:13AM +0200, Robert Marklund wrote: >> This could crash before because of dangerous dangling >> offset of pointer. > > That's right, this can happen. There are more btrfs_item_ptr that would > be good to validate t

Re: WARNING: at /fs/btrfs/extent-tree.c:4029 btrfs_free_reserved_data_space

2015-06-29 Thread Lutz Vieweg
On 06/29/2015 11:35 AM, David Weber wrote: we are testing Btrfs as a kvm storage system with "defaults,space_cache,nodatacow" mount options and regular snapshots. That sounds brave - even with "nodatacow" it appeared to me that using btrfs with often partially overwritten files like VM images r

Re: Scrub errors but no errors ?!?

2015-06-29 Thread Duncan
Swâmi Petaramesh posted on Mon, 29 Jun 2015 10:42:14 +0200 as excerpted: > Using kernel 3.19 on Ubuntu 15.04 > > I have a BTRFS Raid-1 FS made from 2 Luks-encrypted devices, each one > built atop a bcache device. > > The machine runs smoothly from this setup. > > When I scrub said FS, it ends b

WARNING: at /fs/btrfs/extent-tree.c:4029 btrfs_free_reserved_data_space

2015-06-29 Thread David Weber
Hi, we are testing Btrfs as a kvm storage system with "defaults,space_cache,nodatacow" mount options and regular snapshots. While testing we sometimes get these warnings: WARNING: CPU: 5 PID: 11335 at /home/kernel/COD/linux/fs/btrfs/extent- tree.c:4029 btrfs_free_reserved_data_space+0x102/0x110

Re: [PATCH 2/2] [btrfs] btrfs_rename(): don't ignore btrfs_end_transaction() return

2015-06-29 Thread Filipe David Manana
On Sun, Jun 28, 2015 at 10:47 PM, Davide C. C. Italiano wrote: > From: Davide Italiano > > btrfs_end_transaction() can return an error -- this happens, e.g. > if it tries to commit and the transaction was aborted in the meanhwile. > Swallowing the error is wrong, so explicitly return it. > > Sign

Re: [PATCH 1/2] [btrfs] btrfs_rename: abort transaction in case of error.

2015-06-29 Thread Filipe David Manana
On Sun, Jun 28, 2015 at 10:47 PM, Davide C. C. Italiano wrote: > From: Davide Italiano > > btrfs_insert_inode_ref() may fail and we want to make sure > the transaction is aborted before calling btrfs_end_transaction(), > as it already happens everywhere else in this function in case > of error. >

Re: [RFC PATCH V11 02/21] Btrfs: subpagesize-blocksize: Fix whole page write.

2015-06-29 Thread Chandan Rajendra
On Friday 26 Jun 2015 17:50:54 Liu Bo wrote: > On Mon, Jun 01, 2015 at 08:52:37PM +0530, Chandan Rajendra wrote: > > For the subpagesize-blocksize scenario, a page can contain multiple > > blocks. In such cases, this patch handles writing data to files. > > > > Also, When setting EXTENT_DELALLOC,

Scrub errors but no errors ?!?

2015-06-29 Thread Swâmi Petaramesh
Hi, Using kernel 3.19 on Ubuntu 15.04 I have a BTRFS Raid-1 FS made from 2 Luks-encrypted devices, each one built atop a bcache device. The machine runs smoothly from this setup. When I scrub said FS, it ends by spitting a message on console stating that "Scrub found errors but corrected them

Re: btrfs replace seems to corrupt the file system

2015-06-29 Thread Duncan
Mordechay Kaganer posted on Mon, 29 Jun 2015 08:02:01 +0300 as excerpted: > On Sun, Jun 28, 2015 at 10:32 PM, Chris Murphy > wrote: >> On Sun, Jun 28, 2015 at 1:20 PM, Mordechay Kaganer >> wrote: >> >> Use of dd can cause corruption of the original. >> > But doing a block-level copy and taking c

Re: why does df spin up disks?

2015-06-29 Thread Holger Hoffstätte
On 06/29/15 03:43, Russell Coker wrote: > When I have a mounted filesystem why doesn't the kernel store the amount of > free space? Why does it need to spin up a disk that had been spun down? Most likely because the inode has been evicted due to memory pressure. I can df my mostly-idle backup d