[PATCH 09/10] libbtrfsutil: bump version to 1.1.0

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval With the previous few fixes and features, we should bump the minor version. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h index

[PATCH 10/10] libbtrfsutil: document API in README

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval btrfsutil.h and the Python docstrings are thorough, but I've gotten a couple of requests for a high-level overview of the available interfaces and example usages. Add them to README.md. Signed-off-by: Omar Sandoval --- libbtrfsutil/README.md | 422

[PATCH 07/10] libbtrfsutil: relax the privileges of subvolume_info()

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval Attempt to use the BTRFS_IOC_GET_SUBVOL_INFO ioctl (added in kernel 4.18) for subvolume_info() if not root. Also, rename get_subvolume_info_root() -> get_subvolume_info_privileged() for consistency with further changes. This is based on a patch from Misono Tomohiro.

[PATCH 08/10] libbtrfsutil: relax the privileges of subvolume iterator

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval We can use the new BTRFS_IOC_GET_SUBVOL_ROOTREF and BTRFS_IOC_INO_LOOKUP_USER ioctls to allow non-root users to list subvolumes. This is based on a patch from Misono Tomohiro but takes a different approach (mainly, this approach is more similar to the existing tree search

[PATCH 06/10] libbtrfsutil: allow tests to create multiple Btrfs instances

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval Some upcoming tests will need to create a second Btrfs filesystem, so add support for this to the test helpers. Signed-off-by: Omar Sandoval --- libbtrfsutil/python/tests/__init__.py | 35 +-- 1 file changed, 22 insertions(+), 13 deletions(-) diff

[PATCH 03/10] libbtrfsutil: document qgroup_inherit parameter in Python bindings

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval This has been supported since day one, but it wasn't documented. Signed-off-by: Omar Sandoval --- libbtrfsutil/python/module.c | 13 + 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c

[PATCH 05/10] libbtrfsutil: add test helpers for dropping privileges

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval These will be used for testing some upcoming changes which allow unprivileged operations. Signed-off-by: Omar Sandoval --- libbtrfsutil/python/tests/__init__.py | 31 ++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git

[PATCH 01/10] libbtrfsutil: use top=0 as default for SubvolumeIterator()

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval Right now, we're defaulting to top=5 (i.e, all subvolumes). The documented default is top=0 (i.e, only beneath the given path). This is the expected behavior. Fix it and make the test cases cover it. Reported-by: Jonathan Lemon Signed-off-by: Omar Sandoval ---

[PATCH 00/10] btrfs-progs: my libbtrfsutil patch queue

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval Hi, This series contains my backlog of libbtrfsutil changes which I've been collecting over the past few weeks. Patches 1-4 are fixes. Patches 5-6 add functionality to the unit tests which is needed for patches 7-8. Patches 7-8 add support for the unprivileged ioctls added

[PATCH 02/10] libbtrfsutil: change async parameters to async_ in Python bindings

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval async became a keyword in Python 3.7, so, e.g., create_subvolume('foo', async=True) is now a syntax error. Fix it with the Python convention of adding a trailing underscore to the keyword (async -> async_). This is what several other Python libraries did to handle this.

[PATCH 04/10] libbtrfsutil: use SubvolumeIterator as context manager in tests

2018-11-13 Thread Omar Sandoval
From: Omar Sandoval We're leaking file descriptors, which makes it impossible to clean up the temporary mount point created by the test. Signed-off-by: Omar Sandoval --- libbtrfsutil/python/tests/test_subvolume.py | 51 - 1 file changed, 30 insertions(+), 21 deletions(-)

Re: [PATCH] btrfs: Fix suspicious RCU usage warning in device_list_add

2018-11-13 Thread Nikolay Borisov
On 14.11.18 г. 9:24 ч., Lu Fengqi wrote: > = > WARNING: suspicious RCU usage > 4.20.0-rc2+ #23 Tainted: G O > - > fs/btrfs/volumes.c:886 suspicious rcu_dereference_check() usage! > > Use btrfs_info_in_rcu instead of pr_info for

[PATCH] btrfs: Fix suspicious RCU usage warning in device_list_add

2018-11-13 Thread Lu Fengqi
= WARNING: suspicious RCU usage 4.20.0-rc2+ #23 Tainted: G O - fs/btrfs/volumes.c:886 suspicious rcu_dereference_check() usage! Use btrfs_info_in_rcu instead of pr_info for the required lock/unlock of RCU string. Fixes:

[PATCH] btrfs: qgroup: Move reserved data account from btrfs_delayed_ref_head to btrfs_qgroup_extent_record

2018-11-13 Thread Qu Wenruo
[BUG] Btrfs/139 will fail with a pretty high possibility if the testing machine (VM) only has 2G ram. Resulting the final write success while it should fail due to EDQUOT, and the result fs will has quota exceeding the limit by 16K. The simplified reproducer will be: (needs a 2G ram VM)

[PATCH 4/9 v2.1] btrfs: fix UAF due to race between replace start and cancel

2018-11-13 Thread Anand Jain
replace cancel thread can race with the replace start thread and if fs_info::scrubs_running is not yet set the btrfs_scrub_cancel() will fail to stop the scrub thread, so the scrub thread continue with the scrub for replace which then shall try to write to the target device and which is already

Re: [PATCH v15.1 00/13] Btrfs In-band De-duplication

2018-11-13 Thread Lu Fengqi
On Tue, Nov 13, 2018 at 02:45:45PM +0100, David Sterba wrote: >On Tue, Nov 06, 2018 at 02:41:09PM +0800, Lu Fengqi wrote: >> This patchset can be fetched from github: >> https://github.com/littleroad/linux.git dedupe_latest >> >> Now the new base is v4.20-rc1. > >Before anybody spends more time

Re: [PATCH 4/9] btrfs: fix UAF due to race between replace start and cancel

2018-11-13 Thread Anand Jain
On 11/14/2018 01:24 AM, David Sterba wrote: On Sun, Nov 11, 2018 at 10:22:19PM +0800, Anand Jain wrote: replace cancel thread can race with the replace start thread and if fs_info::scrubs_running is not yet set the btrfs_scrub_cancel() will fail to stop the scrub thread, so the scrub thread

Re: [PATCH V7] Btrfs: enhance raid1/10 balance heuristic

2018-11-13 Thread Anand Jain
I am ok with the least used path approach here for the IO routing that's probably most reasonable in generic configurations. It can be default read mirror policy as well. But as I mentioned. Not all configurations would agree to the heuristic approach here. For example: To make use of the SAN

[PATCH V8] Btrfs: enhance raid1/10 balance heuristic

2018-11-13 Thread Timofey Titovets
From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue length to underline devices By default try use pid % num_mirrors guessing, but: - If one of

Re: [PATCH v2 0/6] btrfs: qgroup: Delay subtree scan to reduce overhead

2018-11-13 Thread Qu Wenruo
On 2018/11/14 上午1:58, Filipe Manana wrote: > On Tue, Nov 13, 2018 at 5:08 PM David Sterba wrote: >> >> On Mon, Nov 12, 2018 at 10:33:33PM +0100, David Sterba wrote: >>> On Thu, Nov 08, 2018 at 01:49:12PM +0800, Qu Wenruo wrote: This patchset can be fetched from github:

Re: [PATCH V7] Btrfs: enhance raid1/10 balance heuristic

2018-11-13 Thread Goffredo Baroncelli
On 12/11/2018 12.58, Timofey Titovets wrote: > From: Timofey Titovets > > Currently btrfs raid1/10 balancer bаlance requests to mirrors, > based on pid % num of mirrors. [...] > v6 -> v7: > - Fixes based on Nikolay Borisov review: > * Assume num == 2 > * Remove "for" loop based

Re: [PATCH v2 0/6] btrfs: qgroup: Delay subtree scan to reduce overhead

2018-11-13 Thread Filipe Manana
On Tue, Nov 13, 2018 at 5:08 PM David Sterba wrote: > > On Mon, Nov 12, 2018 at 10:33:33PM +0100, David Sterba wrote: > > On Thu, Nov 08, 2018 at 01:49:12PM +0800, Qu Wenruo wrote: > > > This patchset can be fetched from github: > > >

Re: [PATCH 0/9 v2] fix replace-start and replace-cancel racing

2018-11-13 Thread David Sterba
On Sun, Nov 11, 2018 at 10:22:15PM +0800, Anand Jain wrote: > v1->v2: > 2/9: Drop writeback required > 3/9: Drop writeback required > 7/9: Use the condition within the WARN_ON() > 6/9: Use the condition within the ASSERT() > > Replace-start and replace-cancel threads can race to create a

Re: [PATCH 4/9] btrfs: fix UAF due to race between replace start and cancel

2018-11-13 Thread David Sterba
On Sun, Nov 11, 2018 at 10:22:19PM +0800, Anand Jain wrote: > replace cancel thread can race with the replace start thread and if > fs_info::scrubs_running is not yet set the btrfs_scrub_cancel() will fail > to stop the scrub thread, so the scrub thread continue with the scrub for > replace which

Re: [PATCH v2 0/6] btrfs: qgroup: Delay subtree scan to reduce overhead

2018-11-13 Thread David Sterba
On Mon, Nov 12, 2018 at 10:33:33PM +0100, David Sterba wrote: > On Thu, Nov 08, 2018 at 01:49:12PM +0800, Qu Wenruo wrote: > > This patchset can be fetched from github: > > https://github.com/adam900710/linux/tree/qgroup_delayed_subtree_rebased > > > > Which is based on v4.20-rc1. > > Thanks,

Re: [PATCH 0/6] Chunk allocator DUP fix and cleanups

2018-11-13 Thread Hans van Kranenburg
On 11/13/18 4:03 PM, David Sterba wrote: > On Thu, Oct 11, 2018 at 07:40:22PM +, Hans van Kranenburg wrote: >> On 10/11/2018 05:13 PM, David Sterba wrote: >>> On Thu, Oct 04, 2018 at 11:24:37PM +0200, Hans van Kranenburg wrote: This patch set contains an additional fix for a newly exposed

Re: [PATCH RFC] btrfs: harden agaist duplicate fsid

2018-11-13 Thread Anand Jain
On 11/13/2018 11:31 PM, David Sterba wrote: On Mon, Oct 01, 2018 at 09:31:04PM +0800, Anand Jain wrote: +    /* + * we are going to replace the device path, make sure its the + * same device if the device mounted + */ +    if (device->bdev) { +   

Re: [PATCH RFC RESEND] btrfs: harden agaist duplicate fsid

2018-11-13 Thread Anand Jain
On 11/13/2018 11:21 PM, David Sterba wrote: On Mon, Oct 15, 2018 at 10:45:17AM +0800, Anand Jain wrote: (Thanks for the comments on requiring to warn_on if we fail the device change.) (This fixes an ugly bug, I appreciate if you have any further comments). Its not that impossible to imagine

Re: [PATCH RFC] btrfs: harden agaist duplicate fsid

2018-11-13 Thread Austin S. Hemmelgarn
On 11/13/2018 10:31 AM, David Sterba wrote: On Mon, Oct 01, 2018 at 09:31:04PM +0800, Anand Jain wrote: +    /* + * we are going to replace the device path, make sure its the + * same device if the device mounted + */ +    if (device->bdev) { +    struct

Re: [PATCH RFC] btrfs: harden agaist duplicate fsid

2018-11-13 Thread David Sterba
On Mon, Oct 01, 2018 at 09:31:04PM +0800, Anand Jain wrote: > >> +    /* > >> + * we are going to replace the device path, make sure its the > >> + * same device if the device mounted > >> + */ > >> +    if (device->bdev) { > >> +    struct block_device

Re: [PATCH RFC RESEND] btrfs: harden agaist duplicate fsid

2018-11-13 Thread David Sterba
On Mon, Oct 15, 2018 at 10:45:17AM +0800, Anand Jain wrote: > (Thanks for the comments on requiring to warn_on if we fail the device > change.) > (This fixes an ugly bug, I appreciate if you have any further comments). > > Its not that impossible to imagine that a device OR a btrfs image is >

Re: [PATCH 0/6] Chunk allocator DUP fix and cleanups

2018-11-13 Thread David Sterba
On Thu, Oct 11, 2018 at 07:40:22PM +, Hans van Kranenburg wrote: > On 10/11/2018 05:13 PM, David Sterba wrote: > > On Thu, Oct 04, 2018 at 11:24:37PM +0200, Hans van Kranenburg wrote: > >> This patch set contains an additional fix for a newly exposed bug after > >> the previous attempt to fix

Re: [PATCH] Btrfs: do not set log for full commit when creating non-data block groups

2018-11-13 Thread Filipe Manana
On Tue, Nov 13, 2018 at 2:31 PM David Sterba wrote: > > On Thu, Nov 08, 2018 at 02:48:29PM +, Filipe Manana wrote: > > On Thu, Nov 8, 2018 at 2:37 PM Filipe Manana wrote: > > > > > > On Thu, Nov 8, 2018 at 2:35 PM Qu Wenruo wrote: > > > > > > > > > > > > > > > > On 2018/11/8 下午9:17,

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-11-13 Thread David Sterba
On Mon, Nov 05, 2018 at 04:36:34PM +, Filipe Manana wrote: > On Mon, Nov 5, 2018 at 4:34 PM David Sterba wrote: > > On Mon, Nov 05, 2018 at 04:30:35PM +, Filipe Manana wrote: > > > On Mon, Nov 5, 2018 at 4:29 PM David Sterba wrote: > > > > On Wed, Oct 24, 2018 at 01:48:40PM +0100, Filipe

Re: [PATCH] Btrfs: do not set log for full commit when creating non-data block groups

2018-11-13 Thread David Sterba
On Thu, Nov 08, 2018 at 02:48:29PM +, Filipe Manana wrote: > On Thu, Nov 8, 2018 at 2:37 PM Filipe Manana wrote: > > > > On Thu, Nov 8, 2018 at 2:35 PM Qu Wenruo wrote: > > > > > > > > > > > > On 2018/11/8 下午9:17, fdman...@kernel.org wrote: > > > > From: Filipe Manana > > > > > > > > When

Re: [PATCH v15.1 00/13] Btrfs In-band De-duplication

2018-11-13 Thread David Sterba
On Tue, Nov 06, 2018 at 02:41:09PM +0800, Lu Fengqi wrote: > This patchset can be fetched from github: > https://github.com/littleroad/linux.git dedupe_latest > > Now the new base is v4.20-rc1. Before anybody spends more time with this patchset: this is a big feature and quite intrusive to

Re: [PATCH v2] btrfs: add zstd compression level support

2018-11-13 Thread Timofey Titovets
вт, 13 нояб. 2018 г. в 04:52, Nick Terrell : > > > > > On Nov 12, 2018, at 4:33 PM, David Sterba wrote: > > > > On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrell wrote: > >> From: Jennifer Liu > >> > >> Adds zstd compression level support to btrfs. Zstd requires > >> different amounts of

Re: [PATCH] Btrfs: fix rare chances for data loss when doing a fast fsync

2018-11-13 Thread David Sterba
On Mon, Nov 12, 2018 at 10:23:58AM +, fdman...@kernel.org wrote: > From: Filipe Manana > > After the simplification of the fast fsync patch done recently by commit > b5e6c3e170b7 ("btrfs: always wait on ordered extents at fsync time") and > commit e7175a692765 ("btrfs: remove the wait

Re: [PATCH 1/3] bitops: Fix big endian compilation

2018-11-13 Thread David Sterba
On Mon, Nov 05, 2018 at 11:06:41AM -0800, Rosen Penev wrote: > Replaced bswap with _ variants. While it's a glibc extension, all of the > popular libc implementations (glibc, uClibc, musl, BIONIC) seem to support > it. > > Added static inline to two functions to match little endian variants. This

Re: [RFC] BTRFS_DEV_REPLACE_ITEM_STATE_* doesn't match with on disk

2018-11-13 Thread Anand Jain
David, Gentle ping. Thanks, Anand On 11/12/2018 03:50 PM, Nikolay Borisov wrote: On 12.11.18 г. 6:58 ч., Anand Jain wrote: The dev_replace_state defines are miss matched between the BTRFS_IOCTL_DEV_REPLACE_STATE_* and BTRFS_DEV_REPLACE_ITEM_STATE_* [1]. [1] -

Re: [PATCH] btrfs: fix computation of max fs size for multiple device fs tests

2018-11-13 Thread Anand Jain
On 11/06/2018 11:34 PM, fdman...@kernel.org wrote: From: Filipe Manana We were sorting numerical values with the 'sort' tool without telling it that we are sorting numbers, giving us unexpected ordering. So just pass the '-n' option to the 'sort' tool. Example: $ echo -e "11\n9\n20" |

Re: [PATCH] btrfs: qgroups: Prepare to deprecate BTRFS_QGROUP_LIMIT_RSV_RFER/EXCL flags

2018-11-13 Thread Qu Wenruo
On 2018/11/13 下午5:48, Nikolay Borisov wrote: > > > On 13.11.18 г. 8:09 ч., Qu Wenruo wrote: >> These two flags are only used to set btrfs_qgroup::rsv_rfer/excl number, >> but then are never used. >> >> Nor these flags are really used by btrfs-progs, so it's pretty safe just >> to deprecate

Re: [PATCH 1/2] btrfs-progs: ctree.h: Deprecate unused limit flags

2018-11-13 Thread Nikolay Borisov
On 13.11.18 г. 8:09 ч., Qu Wenruo wrote: > BTRFS_QGROUP_LIMIT_RSV_RFER/EXCL are never used by btrfs-progs. > And kernel only do the basic assignment but don't really make use of > them. > > So deprecate these two flags in btrfs-progs. > Also, add some comment for all the used flags. > >

Re: [PATCH] btrfs: qgroup: Remove duplicated trace points for qgroup_rsv_add/release()

2018-11-13 Thread Nikolay Borisov
On 13.11.18 г. 9:05 ч., Qu Wenruo wrote: > Inside qgroup_rsv_add/release(), we have trace events > trace_qgroup_update_reserve() to catch reserved space update. > > However we still have two manual trace_qgroup_update_reserve() calls > just outside these functions. > > Remove these duplicated

Re: [PATCH] btrfs: qgroups: Prepare to deprecate BTRFS_QGROUP_LIMIT_RSV_RFER/EXCL flags

2018-11-13 Thread Nikolay Borisov
On 13.11.18 г. 8:09 ч., Qu Wenruo wrote: > These two flags are only used to set btrfs_qgroup::rsv_rfer/excl number, > but then are never used. > > Nor these flags are really used by btrfs-progs, so it's pretty safe just > to deprecate them in next kernel release. > > This patch will mark