Re: [PATCH] btrfs: list usage cleanup

2018-10-01 Thread David Sterba
On Thu, Sep 27, 2018 at 11:47:04AM -0700, Omar Sandoval wrote: > On Wed, Sep 26, 2018 at 04:35:45PM +0800, zhong jiang wrote: > > Trival cleanup, list_move_tail will implement the same function that > > list_del() + list_add_tail() will do. hence just replace them. > > > > Signed-off-by: zhong jia

[PATCH] verify hardening agaist duplicate fsid

2018-10-01 Thread Anand Jain
Its not that impossible to imagine that a device OR a btrfs image is been copied just by using the dd or the cp command. Which in case both the copies of the btrfs will have the same fsid. If on the system with automount enabled, the copied FS gets scanned. We have a known bug in btrfs, that we le

[PATCH] fstests: btrfs verify hardening agaist duplicate fsid

2018-10-01 Thread Anand Jain
We have a known bug in btrfs, that we let the device path be changed after the device has been mounted. So using this loop hole the new copied device would appears as if its mounted immediately after its been copied. So this test case reproduces this issue. For example: Initially.. /dev/mmcblk0p4

[PATCH RFC] btrfs: harden agaist duplicate fsid

2018-10-01 Thread Anand Jain
Its not that impossible to imagine that a device OR a btrfs image is been copied just by using the dd or the cp command. Which in case both the copies of the btrfs will have the same fsid. If on the system with automount enabled, the copied FS gets scanned. We have a known bug in btrfs, that we le

Re: [PLEASE Ignore] [PATCH] verify hardening agaist duplicate fsid

2018-10-01 Thread Anand Jain
Please ignore this patch. Instead help review/integrate [1] which is the same patch with subject and change log updated. [1] [PATCH] fstests: btrfs verify hardening agaist duplicate fsid Thanks, Anand

Re: [PATCH RFC] btrfs: harden agaist duplicate fsid

2018-10-01 Thread Austin S. Hemmelgarn
On 2018-10-01 04:56, Anand Jain wrote: Its not that impossible to imagine that a device OR a btrfs image is been copied just by using the dd or the cp command. Which in case both the copies of the btrfs will have the same fsid. If on the system with automount enabled, the copied FS gets scanned.

Re: [PATCH RFC] btrfs: harden agaist duplicate fsid

2018-10-01 Thread Anand Jain
On 10/01/2018 07:17 PM, Austin S. Hemmelgarn wrote: On 2018-10-01 04:56, Anand Jain wrote: Its not that impossible to imagine that a device OR a btrfs image is been copied just by using the dd or the cp command. Which in case both the copies of the btrfs will have the same fsid. If on the sys

cross-fs copy support

2018-10-01 Thread Joshi
I was wondering about the cross-fs copy through copy_file_range. It seems current implement has below check, that disables such copy. 1577 /* this could be relaxed once a method supports cross-fs copies */ 1578 if (inode_in->i_sb != inode_out->i_sb) 1579 return -EXD

[PATCH 01/10] btrfs-progs: Add support for freespace tree in btrfs_read_fs_root

2018-10-01 Thread Nikolay Borisov
For completeness sake add code to btrfs_read_fs_root so that it can handle the freespace tree. Signed-off-by: Nikolay Borisov --- disk-io.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/disk-io.c b/disk-io.c index 2e6d56a36af9..14f0fd5c2f0c 100644 --- a/disk-io.c +++ b/disk-io.c @@ -668

[PATCH 02/10] btrfs-progs: Add extent buffer bitmap manipulation infrastructure

2018-10-01 Thread Nikolay Borisov
Those functions are in preparation for adding the freespace tree repair code since it needs to be able to deal with bitmap based fsts. This patch adds extent_buffer_bitmap_set and extent_buffer_bitmap_clear functions. Since in userspace we don't have to deal with page mappings their implementation

[PATCH 03/10] btrfs-progs: Replace homegrown bitops related functions with kernel counterparts

2018-10-01 Thread Nikolay Borisov
Replace existing find_*_bit functions with kernel equivalent. This reduces duplication, simplifies the code (we really have one worker function _find_next_bit) and is quite likely faster. No functional changes. Signed-off-by: Nikolay Borisov --- kernel-lib/bitops.h | 142 +---

[PATCH 04/10] btrfs-progs: Implement find_*_bit_le operations

2018-10-01 Thread Nikolay Borisov
This commit introduces explicit little endian bit operations. The only difference with the existing bitops implementation is that bswap(32|64) is called when the _le versions are invoked on a big-endian machine. This is in preparation for adding free space tree conversion support. Signed-off-by: N

[PATCH 07/10] btrfs-progs: Add freespace tree as compat_ro supported feature

2018-10-01 Thread Nikolay Borisov
The RO_FREE_SPACE_TREE(_VALID) flags are required in order to be able to open an FST filesystem in repair mode. Add them to BTRFS_FEATURE_COMPAT_RO_SUPP. Signed-off-by: Nikolay Borisov --- ctree.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ctree.h b/ctree.h index a6d6

[PATCH 08/10] btrfs-progs: check: Add support for freespace tree fixing

2018-10-01 Thread Nikolay Borisov
Now that all the prerequisite code for proper support of free space tree repair is in, it's time to wire it in. This is achieved by first hooking the freespace tree to the __free_extent/alloc_reserved_tree_block functions. And then introducing a wrapper function to contains the existing check_space

[PATCH 10/10] btrfs-progs: check: Fix wrong error message in case of corrupted bitmap

2018-10-01 Thread Nikolay Borisov
Similarly to the fix in e444c7bfa65f ("btrfs-progs: check: Fix wrong error message in case of corrupted extent") this commits addresses the same problem but for corrupted bitmap objects. Signed-off-by: Nikolay Borisov --- free-space-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) d

[PATCH 09/10] btrfs-progs: tests: Test for FST corruption detection/repair

2018-10-01 Thread Nikolay Borisov
Simple test case which preps a filesystem, then corrupts the FST and finally repairs it. Tests both extent based and bitmap based FSTs. Signed-off-by: Nikolay Borisov --- tests/fsck-tests/036-freespacetree-repair/test.sh | 76 +++ 1 file changed, 76 insertions(+) create mode

[PATCH 05/10] btrfs-progs: Pull free space tree related code from kernel

2018-10-01 Thread Nikolay Borisov
To help implement free space tree checker in user space some kernel function are necessary, namely iterating/deleting/adding freespace items, some internal search functions. Functions to populate a block group based on the extent tree. The code is largely copy/paste from the kernel with locking eli

[PATCH 06/10] btrfs-progs: Hook FST code in extent (de)alloc

2018-10-01 Thread Nikolay Borisov
For now this doesn't change the functionality since FST code is not yet enabled via the compat bits. But this will be needed when it's enabled so that the FST is correctly modified during repair operations that allocate/deallocate extents. Signed-off-by: Nikolay Borisov --- extent-tree.c | 11 +

[PATCH 00/10] Freespace tree repair support v2

2018-10-01 Thread Nikolay Borisov
Hello, Here is the v2 of the freespace tree repair support patches. Version 1 can be found at [0]. For background on the series please refer to the initial cover letter. This time round a number of changes have been incorporated based on feedback from Omar. Namely: 1. Added 2 new patches, pa

Re: cross-fs copy support

2018-10-01 Thread Qu Wenruo
On 2018/10/1 下午10:32, Joshi wrote: > I was wondering about the cross-fs copy through copy_file_range. The term "cross-fs" looks pretty confusing. If you mean "cross-subvolume", then it should work without problem in btrfs. If you mean reflink across two different file systems (not matter the s

Re: cross-fs copy support

2018-10-01 Thread Joshi
Sorry if my post was not clear enough. I picked the term "cross-fs" from the implementation of vfs_copy_file_range. Below snippet, inside vfs_copy_file_range - 1578 /* this could be relaxed once a method supports cross-fs copies */ 1579 if (inode_in->i_sb != inode_out->i_sb) 1580

Re: cross-fs copy support

2018-10-01 Thread Eric Sandeen
On 10/1/18 9:48 AM, Qu Wenruo wrote: > > > On 2018/10/1 下午10:32, Joshi wrote: >> I was wondering about the cross-fs copy through copy_file_range. > > The term "cross-fs" looks pretty confusing. > > If you mean "cross-subvolume", then it should work without problem in btrfs. > > If you mean ref

Re: [PATCH 08/42] btrfs: dump block_rsv whe dumping space info

2018-10-01 Thread David Sterba
On Fri, Sep 28, 2018 at 07:17:47AM -0400, Josef Bacik wrote: > For enospc_debug having the block rsvs is super helpful to see if we've > done something wrong. > > Signed-off-by: Josef Bacik > Reviewed-by: Omar Sandoval Reviewed-by: David Sterba

Re: [PATCH 10/42] btrfs: protect space cache inode alloc with nofs

2018-10-01 Thread David Sterba
On Fri, Sep 28, 2018 at 07:17:49AM -0400, Josef Bacik wrote: > If we're allocating a new space cache inode it's likely going to be > under a transaction handle, so we need to use memalloc_nofs_save() in > order to avoid deadlocks, and more importantly lockdep messages that > make xfstests fail. >

Re: [PATCH 36/42] btrfs: wait on caching when putting the bg cache

2018-10-01 Thread David Sterba
On Fri, Sep 28, 2018 at 07:18:15AM -0400, Josef Bacik wrote: > While testing my backport I noticed there was a panic if I ran > generic/416 generic/417 generic/418 all in a row. This just happened to > uncover a race where we had outstanding IO after we destroy all of our > workqueues, and then we

Re: cross-fs copy support

2018-10-01 Thread Andreas Dilger
On Oct 1, 2018, at 9:49 AM, Eric Sandeen wrote: > > > On 10/1/18 9:48 AM, Qu Wenruo wrote: >> >> >> On 2018/10/1 下午10:32, Joshi wrote: >>> I was wondering about the cross-fs copy through copy_file_range. >> >> The term "cross-fs" looks pretty confusing. >> >> If you mean "cross-subvolume

Re: [patch] file dedupe (and maybe clone) data corruption (was Re: [PATCH] generic: test for deduplication between different files)

2018-10-01 Thread Andreas Dilger
On Sep 20, 2018, at 10:40 PM, Zygo Blaxell wrote: > > On Fri, Sep 21, 2018 at 12:59:31PM +1000, Dave Chinner wrote: >> On Wed, Sep 19, 2018 at 12:12:03AM -0400, Zygo Blaxell wrote: > [...] >> With no DMAPI in the future, people with custom HSM-like interfaces >> based on dmapi are starting to tu

Re: btrfs send receive ERROR: chown failed: No such file or directory

2018-10-01 Thread Leonard Lausen
Hello, does anyone have an idea about below issue? It is a severe issue as it renders btrfs send / receive dysfunctional and it is not clear if there may be a data corruption issue hiding in the current send / receive code. Thank you. Best regards Leonard Leonard Lausen writes: > Hello! > > I