[PATCH] btrfs-progs: Kill ASSERT() for damaged filesystem

2019-07-03 Thread WenRuo Qu
For damaged filesystem like 'bko-155621-bad-block-group-offset.raw' from fuzzed tests, there may be no valid METADATA blocks at all. Thus we could hit the following ASSERT(): extent-tree.c:2484: alloc_tree_block: Assertion `sinfo` failed, value 0 btrfs(+0x20ef8)[0x555adf5b2ef8] btrfs(+0x2107

[PATCH] btrfs: Ensure replaced device doesn't have pending chunk allocation

2019-07-03 Thread Nikolay Borisov
Recent FITRIM work, namely bbbf7243d62d ("btrfs: combine device update operations during transaction commit") combined the way certain operations are recoded in a transaction. As a result an ASSERT was added in dev_replace_finish to ensure the new code works correctly. Unfortunately I got reports t

[PATCH] btrfs: Ensure replaced device doesn't have pending chunk allocation

2019-07-03 Thread Nikolay Borisov
Recent FITRIM work, namely bbbf7243d62d ("btrfs: combine device update operations during transaction commit") combined the way certain operations are recoded in a transaction. As a result an ASSERT was added in dev_replace_finish to ensure the new code works correctly. Unfortunately I got reports t

[PATCH] btrfs: Ensure replaced device doesn't have pending chunk allocation

2019-07-03 Thread Nikolay Borisov
Recent FITRIM work, namely bbbf7243d62d ("btrfs: combine device update operations during transaction commit") combined the way certain operations are recoded in a transaction. As a result an ASSERT was added in dev_replace_finish to ensure the new code works correctly. Unfortunately I got reports t

[PATCH] btrfs: Ensure replaced device doesn't have pending chunk allocation

2019-07-03 Thread Nikolay Borisov
Recent FITRIM work, namely bbbf7243d62d ("btrfs: combine device update operations during transaction commit") combined the way certain operations are recoded in a transaction. As a result an ASSERT was added in dev_replace_finish to ensure the new code works correctly. Unfortunately I got reports t

[PATCH] btrfs: Ensure replaced device doesn't have pending chunk allocation

2019-07-03 Thread Nikolay Borisov
Recent FITRIM work, namely bbbf7243d62d ("btrfs: combine device update operations during transaction commit") combined the way certain operations are recoded in a transaction. As a result an ASSERT was added in dev_replace_finish to ensure the new code works correctly. Unfortunately I got reports t

Re: [PATCH] btrfs-progs: Kill ASSERT() for damaged filesystem

2019-07-03 Thread David Sterba
On Wed, Jul 03, 2019 at 07:24:42AM +, WenRuo Qu wrote: > For damaged filesystem like 'bko-155621-bad-block-group-offset.raw' from > fuzzed tests, there may be no valid METADATA blocks at all. > > Thus we could hit the following ASSERT(): > extent-tree.c:2484: alloc_tree_block: Assertion `sin

[PATCH] btrfs: Refactor btrfs_calc_avail_data_space

2019-07-03 Thread Nikolay Borisov
Simplify the code by removing variables that don't bring any real value as well as simplifying the checks when buidling the candidate list of devices. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/super.c | 30 ++ 1 file changed, 10 insertions(+),

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread David Sterba
On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: > From: Anand Jain > > The cli 'btrfs inspect dump-tree ' will scan for the partner devices > if any by default. > > So as of now you can not inspect each mirrored device independently. > > This patch adds noscan option, which when use

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread David Sterba
On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: > From: Anand Jain > > The cli 'btrfs inspect dump-tree ' will scan for the partner devices > if any by default. > > So as of now you can not inspect each mirrored device independently. > > This patch adds noscan option, which when use

Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index

2019-07-03 Thread David Sterba
On Fri, Jun 28, 2019 at 10:26:11AM +0800, Anand Jain wrote: > At the time mkfs.btrfs the device id and stripe index gets reversed as > shown in [1]. This patch helps to keep them in order at the time of > mkfs.btrfs. And makes it easier to debug. > > Before: > Stripe 0 is on devid 2; Stipe 1 is on

Need help with a lockdep splat, possibly perf related?

2019-07-03 Thread Josef Bacik
Hello, I've been seeing a variation of the following splat recently and I have no earthly idea what it's trying to tell me. I either get this one, or I get one that tells me the same thing except it's complaining about &cpuctx_mutex instead of sb_pagefaults. There is no place we take the reloc_m

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread David Sterba
On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: > From: Anand Jain > > The cli 'btrfs inspect dump-tree ' will scan for the partner devices > if any by default. > > So as of now you can not inspect each mirrored device independently. > > This patch adds noscan option, which when use

Re: Need help with a lockdep splat, possibly perf related?

2019-07-03 Thread Peter Zijlstra
On Wed, Jul 03, 2019 at 09:54:06AM -0400, Josef Bacik wrote: > Hello, > > I've been seeing a variation of the following splat recently and I have no > earthly idea what it's trying to tell me. That you have a lock cycle; aka. deadlock, obviously :-) > I either get this one, or I get one > that t

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread Anand Jain
> On 4 Jul 2019, at 12:09 AM, David Sterba wrote: > > On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: >> From: Anand Jain >> >> The cli 'btrfs inspect dump-tree ' will scan for the partner devices >> if any by default. >> >> So as of now you can not inspect each mirrored device

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread David Sterba
On Thu, Jul 04, 2019 at 06:16:54AM +0800, Anand Jain wrote: > > > > On 4 Jul 2019, at 12:09 AM, David Sterba wrote: > > > > On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: > >> From: Anand Jain > >> > >> The cli 'btrfs inspect dump-tree ' will scan for the partner devices > >> if

Re: [PATCH v7 RESEND Rebased] btrfs-progs: dump-tree: add noscan option

2019-07-03 Thread Anand Jain
> On 4 Jul 2019, at 7:39 AM, David Sterba wrote: > > On Thu, Jul 04, 2019 at 06:16:54AM +0800, Anand Jain wrote: >> >> >>> On 4 Jul 2019, at 12:09 AM, David Sterba wrote: >>> >>> On Wed, Jun 26, 2019 at 01:30:17AM -0700, Anand Jain wrote: From: Anand Jain The cli 'btrfs in

Re: [PATCH v2 00/14] btrfs-progs: image: Enhance and bug fixes

2019-07-03 Thread Anand Jain
On 2/7/19 6:07 PM, WenRuo Qu wrote: This patchset is based on v5.1.1 tag. With this update, the patchset has the following features: - various small fixes and enhancements for btrfs-image * Fix an indent misalign * Fix an access-beyond-boundary bug * Fix a confusing error message due to

Re: [PATCH v2 00/14] btrfs-progs: image: Enhance and bug fixes

2019-07-03 Thread Qu Wenruo
On 2019/7/4 上午10:13, Anand Jain wrote: > On 2/7/19 6:07 PM, WenRuo Qu wrote: >> This patchset is based on v5.1.1 tag. >> >> With this update, the patchset has the following features: >> - various small fixes and enhancements for btrfs-image >>    * Fix an indent misalign >>    * Fix an access-bey

[PATCH v2.1 00/10] btrfs-progs: image: Enhancement with new data dump feature

2019-07-03 Thread Qu Wenruo
This updated patchset is rebased on devel branch, whose HEAD is: commit 73289664917ad7c73f3f3393593e93a118228ad8 (david/devel) Author: David Sterba Date: Thu Jul 4 01:42:15 2019 +0200 btrfs-progs: kerncompat: define __always_inline conditionally This updated patchset includes the following

[PATCH v2.1 03/10] btrfs-progs: image: Don't waste memory when we're just extracting super block

2019-07-03 Thread Qu Wenruo
There is no need to allocate 2 * max_pending_size (which can be 256M) if we're just extracting super block. We only need to prepare BTRFS_SUPER_INFO_SIZE as buffer size. Signed-off-by: Qu Wenruo --- image/main.c | 13 + 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/i

[PATCH v2.1 01/10] btrfs-progs: image: Output error message for chunk tree build error

2019-07-03 Thread Qu Wenruo
Signed-off-by: Qu Wenruo --- image/main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/image/main.c b/image/main.c index 28ef1609b5ff..fb407f33f858 100644 --- a/image/main.c +++ b/image/main.c @@ -2406,8 +2406,10 @@ static int restore_metadump(const char *input, FILE *o

[PATCH v2.1 02/10] btrfs-progs: image: Fix error output to show correct return value

2019-07-03 Thread Qu Wenruo
We can easily get confusing error message like: ERROR: restore failed: Success This is caused by wrong "%m" usage, as we normally use ret to indicate error, without populating errno. This patch will fix it by output the return value directly as normally we have extra error message to show more

[PATCH v2.1 06/10] btrfs-progs: image: Rework how we search chunk tree blocks

2019-07-03 Thread Qu Wenruo
Before this patch, we were using a very inefficient way to search chunks: We iterate through all clusters to find the chunk root tree block first, then re-iterate all clusters again to find every child tree blocks. Every time we need to iterate all clusters just to find a chunk tree block. This i

[PATCH v2.1 09/10] btrfs-progs: image: Reduce memory requirement for decompression

2019-07-03 Thread Qu Wenruo
With recent change to enlarge max_pending_size to 256M for data dump, the decompress code requires quite a lot of memory space. (256M * 4). The main reason behind it is, we're using wrapped uncompress() function call, which needs the buffer to be large enough to contain the decompressed data. Thi

[PATCH v2.1 07/10] btrfs-progs: image: Introduce framework for more dump versions

2019-07-03 Thread Qu Wenruo
The original dump format only contains a @magic member to verify the format, this means if we want to introduce new on-disk format or change certain size limit, we can only introduce new magic as kind of version. This patch will introduce the framework to allow multiple magic to co-exist for furth

[PATCH v2.1 04/10] btrfs-progs: image: Allow restore to record system chunk ranges for later usage

2019-07-03 Thread Qu Wenruo
Currently we are doing a pretty slow search for system chunks before restoring real data. The current behavior is to search all clusters for chunk tree root first, then search all clusters again and again for every chunk tree block. This causes recursive calls and pretty slow start up, the only go

[PATCH v2.1 08/10] btrfs-progs: image: Introduce -d option to dump data

2019-07-03 Thread Qu Wenruo
This new data dump feature will dump the whole image, not long the existing tree blocks but also all its data extents(*). This feature will rely on the new dump format (_DUmP_v1), as it needs extra large extent size limit, and older btrfs-image dump can't handle such large item/cluster size. Sinc

[PATCH v2.1 05/10] btrfs-progs: image: Introduce helper to determine if a tree block is in the range of system chunks

2019-07-03 Thread Qu Wenruo
Introduce a new helper function, is_in_sys_chunks(), to determine if an item is in the range of system chunks. Since btrfs-image will merge adjacent same type extents into one item, this function is designed to return true for any bytes in system chunk range. Signed-off-by: Qu Wenruo --- image/

[PATCH v2.1 10/10] btrfs-progs: image: Reduce memory usage for chunk tree search

2019-07-03 Thread Qu Wenruo
Just like original restore_worker, search_for_chunk_blocks() will also use a lot of memory for restoring large uncompressed extent or compressed extent with data dump. Reduce the memory usage by: - Use fixed buffer size for uncompressed extent Now we will use fixed 512K as buffer size for readin

Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle

2019-07-03 Thread Qu Wenruo
On 2019/6/5 上午1:43, Josef Bacik wrote: > On Tue, Jun 04, 2019 at 08:31:23AM +0800, Qu Wenruo wrote: >> >> >> On 2019/6/4 上午1:36, Josef Bacik wrote: >>> On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote: On 2019/2/13 上午12:03, David Sterba wrote: > On Thu, Jan 24, 2019 at