[PATCH] btrfs: qgroup: Fix tree dead lock caused by backref walk during snapshot dropping

2018-12-10 Thread Qu Wenruo
[BUG] System may lockup during snapshot dropping with quota enabled. The backtrace would be: btrfs-cleaner D0 4062 2 0x8000 Call Trace: schedule+0x32/0x90 btrfs_tree_read_lock+0x93/0x130 [btrfs] find_parent_nodes+0x29b/0x1170 [btrfs] btrfs_find_all_roots_safe+0xa8/

Re: [PATCH] btrfs: extent-tree: cleanup one-shot usage of @blocksize in do_walk_down

2018-12-10 Thread Johannes Thumshirn
Looks good, Reviewed-by: Johannes Thumshirn -- Johannes ThumshirnSUSE Labs Filesystems jthumsh...@suse.de+49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG N

Re: [RFC PATCH] btrfs: Remove __extent_readpages

2018-12-10 Thread Nikolay Borisov
On 5.12.18 г. 18:58 ч., Josef Bacik wrote: > On Mon, Dec 03, 2018 at 12:25:32PM +0200, Nikolay Borisov wrote: >> When extent_readpages is called from the generic readahead code it first >> builds a batch of 16 pages (which might or might not be consecutive, >> depending on whether add_to_page_ca

Re: [PATCH v2] Btrfs: use generic_remap_file_range_prep() for cloning and deduplication

2018-12-10 Thread Nikolay Borisov
On 7.12.18 г. 17:25 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > Since cloning and deduplication are no longer Btrfs specific operations, we > now have generic code to handle parameter validation, compare file ranges > used for deduplication, clear capabilities when cloning, etc. T

Re: [PATCH 3/8] btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_tree_ref()

2018-12-10 Thread Nikolay Borisov
On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: > btrfs_add_delayed_tree_ref() has a longer and longer parameter list, and > some caller like btrfs_inc_extent_ref() are using @owner as level for > delayed tree ref. > > Instead of making the parameter list longer and longer, use btrfs_ref to > refactor

Re: [PATCH 4/8] btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_data_ref()

2018-12-10 Thread Nikolay Borisov
On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: > Just like btrfs_add_delayed_tree_ref(), use btrfs_ref to refactor > btrfs_add_delayed_data_ref(). > > Signed-off-by: Qu Wenruo > --- > fs/btrfs/delayed-ref.c | 19 +-- > fs/btrfs/delayed-ref.h | 8 +++- > fs/btrfs/extent-tree.c |

Re: [PATCH 6/8] btrfs: extent-tree: Use btrfs_ref to refactor add_pinned_bytes()

2018-12-10 Thread Nikolay Borisov
On 6.12.18 г. 8:59 ч., Qu Wenruo wrote: > Since add_pinned_bytes() only needs to know if the extent is metadata > and if it's a chunk tree extent, btrfs_ref is a perfect match for it, as > we don't need various owner/level trick to determine extent type. > > Signed-off-by: Qu Wenruo Reviewed-

Re: [PATCH 1/8] btrfs: delayed-ref: Introduce better documented delayed ref structures

2018-12-10 Thread Nikolay Borisov
On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: > Current delayed ref interface has several problems: > - Longer and longer parameter lists > bytenr > num_bytes > parent > ref_root > owner > offset > for_reloc << Only qgroup code cares. > > - Different interpretation for the same paramete

Re: [PATCH 7/8] btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()

2018-12-10 Thread Nikolay Borisov
On 6.12.18 г. 8:59 ч., Qu Wenruo wrote: > Now we don't need to play the dirty game of reusing @owner for tree block > level. > > Signed-off-by: Qu Wenruo > --- > fs/btrfs/ctree.h | 6 ++--- > fs/btrfs/extent-tree.c | 58 ++ > fs/btrfs/file.c

Re: [PATCH v2 0/6] btrfs: qgroup: Delay subtree scan to reduce overhead

2018-12-10 Thread Filipe Manana
On Sat, Dec 8, 2018 at 12:51 AM Qu Wenruo wrote: > > > > On 2018/12/8 上午8:47, David Sterba wrote: > > On Fri, Dec 07, 2018 at 06:51:21AM +0800, Qu Wenruo wrote: > >> > >> > >> On 2018/12/7 上午3:35, David Sterba wrote: > >>> On Mon, Nov 12, 2018 at 10:33:33PM +0100, David Sterba wrote: > On Thu

Re: [PATCH 1/8] btrfs: delayed-ref: Introduce better documented delayed ref structures

2018-12-10 Thread Qu Wenruo
On 2018/12/10 下午5:48, Nikolay Borisov wrote: > > > On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: >> Current delayed ref interface has several problems: >> - Longer and longer parameter lists >> bytenr >> num_bytes >> parent >> ref_root >> owner >> offset >> for_reloc << Only qgroup cod

Re: [PATCH v2 0/6] btrfs: qgroup: Delay subtree scan to reduce overhead

2018-12-10 Thread Qu Wenruo
On 2018/12/10 下午6:45, Filipe Manana wrote: > On Sat, Dec 8, 2018 at 12:51 AM Qu Wenruo wrote: >> >> >> >> On 2018/12/8 上午8:47, David Sterba wrote: >>> On Fri, Dec 07, 2018 at 06:51:21AM +0800, Qu Wenruo wrote: On 2018/12/7 上午3:35, David Sterba wrote: > On Mon, Nov 12, 2018 at

Re: [PATCH 1/8] btrfs: delayed-ref: Introduce better documented delayed ref structures

2018-12-10 Thread Nikolay Borisov
On 10.12.18 г. 13:20 ч., Qu Wenruo wrote: > > > On 2018/12/10 下午5:48, Nikolay Borisov wrote: >> >> >> On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: >>> Current delayed ref interface has several problems: >>> - Longer and longer parameter lists >>> bytenr >>> num_bytes >>> parent >>> ref_root

Re: [PATCH 1/8] btrfs: delayed-ref: Introduce better documented delayed ref structures

2018-12-10 Thread Qu Wenruo
On 2018/12/10 下午7:32, Nikolay Borisov wrote: > > > On 10.12.18 г. 13:20 ч., Qu Wenruo wrote: >> >> >> On 2018/12/10 下午5:48, Nikolay Borisov wrote: >>> >>> >>> On 6.12.18 г. 8:58 ч., Qu Wenruo wrote: Current delayed ref interface has several problems: - Longer and longer parameter lis

Kernel traces

2018-12-10 Thread Stephen R. van den Berg
For the (roughly) past 12 months, I've been experiencing the following (repeatable, nightly) kernel traces on btrfs filesystems which are under heavy (nightly backups) btrfs-receive traffic (over the network). The problems intensify when the amount of data per second increases. If the amount of da

Re: Kernel traces

2018-12-10 Thread Chris Murphy
I suggest reproducing the problem and issuing sysrq+w and then post the entire resulting output for a developer to evaluate. I find it's best to have the sysrq setup in a shell before hand so all you have to do is hit return, and of course you can test the before the problem happens. It dumps to dm

Re: [PATCH 1/8] btrfs: delayed-ref: Introduce better documented delayed ref structures

2018-12-10 Thread David Sterba
On Mon, Dec 10, 2018 at 07:37:57PM +0800, Qu Wenruo wrote: > >>> Also please use explicit bool type: > >>> > >>> bool :1 > >> > >> Is this valid? Haven't seen such usage in kernel code IIRC. > > > > git grep 'bool .*:1' | wc -l > > 417 > > grep -IR 'bool .*:1' fs/btrfs/ | wc -l > 0 > > So I

Re: [PATCH 7/8] btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()

2018-12-10 Thread David Sterba
On Mon, Dec 10, 2018 at 11:52:25AM +0200, Nikolay Borisov wrote: > On 6.12.18 г. 8:59 ч., Qu Wenruo wrote: > > Now we don't need to play the dirty game of reusing @owner for tree block > > level. > > > > Signed-off-by: Qu Wenruo > > --- > > fs/btrfs/ctree.h | 6 ++--- > > fs/btrfs/extent-

[PATCH] Btrfs: use nofs context when initializing security xattrs to avoid deadlock

2018-12-10 Thread fdmanana
From: Filipe Manana When initializing the security xattrs, we are holding a transaction handle therefore we need to use a GFP_NOFS context in order to avoid a deadlock with reclaim in case it's triggered. Fixes: 39a27ec1004e8 ("btrfs: use GFP_KERNEL for xattr and acl allocations") Signed-off-by:

[PATCH] Btrfs: send, fix race with transaction commits that create snapshots

2018-12-10 Thread fdmanana
From: Filipe Manana If we create a snapshot of a snapshot currently being used by a send operation, we can end up with send failing unexpectedly (returning -ENOENT error to user space for example). The following diagram shows how this happens. CPU 1

Re: [PATCH] Btrfs: scrub, move setup of nofs contexts higher in the stack

2018-12-10 Thread David Sterba
On Fri, Dec 07, 2018 at 01:23:32PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > Since scrub workers only do memory allocation with GFP_KERNEL when they > need to perform repair, we can move the recent setup of the nofs context > up to scrub_handle_errored_block() instead of setting

Re: [PATCH 7/8] btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()

2018-12-10 Thread Qu Wenruo
On 2018/12/11 上午1:28, David Sterba wrote: > On Mon, Dec 10, 2018 at 11:52:25AM +0200, Nikolay Borisov wrote: >> On 6.12.18 г. 8:59 ч., Qu Wenruo wrote: >>> Now we don't need to play the dirty game of reusing @owner for tree block >>> level. >>> >>> Signed-off-by: Qu Wenruo >>> --- >>> fs/btrfs

[PATCH] btrfs: qgroup: Don't trigger backref walk at delayed ref insert time

2018-12-10 Thread Qu Wenruo
[BUG] Since fb235dc06fac ("btrfs: qgroup: Move half of the qgroup accounting time out of commit trans"), kernel may lockup with quota enabled. There is one backref trace triggered by snapshot dropping along with write operation in the source subvolume. The example can be stably reproduced. btrf