Re: [PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-04 Thread Qu Wenruo
On 2018年07月04日 15:08, Nikolay Borisov wrote: > > > On 3.07.2018 12:10, Qu Wenruo wrote: >> If a crafted btrfs has missing block group items, it could cause >> unexpected behavior and breaks our expectation on 1:1 >> chunk<->block group mapping. >> &g

Re: A list of bugs in btrfs found by fuzzing

2018-07-03 Thread Qu Wenruo
On 2018年07月04日 07:26, Xu, Wen wrote: > I see, then does BTRFS have a dev branch for merging into mainline?… We have, although these fixes are not fully merged yet. https://github.com/kdave/btrfs-devel/tree/misc-next Thanks, Qu > > Thanks, > Wen > >> On Jul 3, 2018,

Re: So, does btrfs check lowmem take days? weeks?

2018-07-03 Thread Qu Wenruo
On 2018年07月04日 06:00, Marc MERLIN wrote: > On Tue, Jul 03, 2018 at 03:46:59PM -0600, Chris Murphy wrote: >> On Tue, Jul 3, 2018 at 2:50 AM, Qu Wenruo wrote: >>> >>> >>> There must be something wrong, however due to the size of the fs, and >>>

Re: A list of bugs in btrfs found by fuzzing

2018-07-03 Thread Qu Wenruo
On 2018年07月04日 00:27, Xu, Wen wrote: > So this is the dev branch of btrfs?… Just my branch of all the related fixes. Thanks, Qu > > Thanks, > -Wen > >> On Jul 3, 2018, at 9:36 AM, Qu Wenruo wrote: >> >> >> >> On 2018年07月03日 20:37, Xu, Wen wrot

[PATCH] btrfs-progs: Don't BUG_ON() if we failed to load one device or one chunk

2018-07-03 Thread Qu Wenruo
Don't panic for btrfs_read_chunk_tree() if one device or chunk is corrupted. Caller can already handle it pretty well. Link: https://bugzilla.kernel.org/show_bug.cgi?id=199839 Signed-off-by: Qu Wenruo --- volumes.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git

Re: A list of bugs in btrfs found by fuzzing

2018-07-03 Thread Qu Wenruo
cefully with error message showing what's going wrong. You could also test using this branch: https://github.com/adam900710/linux/tree/tree_checker_enhance Thanks, Qu > >> On Jul 3, 2018, at 4:32 AM, Qu Wenruo wrote: >> >> >> >> On 2018年07月01日 08:06, Xu, Wen wro

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-03 Thread Qu Wenruo
On 2018年07月03日 17:55, Paul Jones wrote: >> -Original Message- >> From: linux-btrfs-ow...@vger.kernel.org > ow...@vger.kernel.org> On Behalf Of Marc MERLIN >> Sent: Tuesday, 3 July 2018 2:16 PM >> To: Qu Wenruo >> Cc: Su Yue ; linux-btrfs@vger.ke

[PATCH 4/5] btrfs: Check each block group has corresponding chunk at mount time

2018-07-03 Thread Qu Wenruo
time. Link: https://bugzilla.kernel.org/show_bug.cgi?id=199837 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 55 -- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 3d9fe58

[PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-03 Thread Qu Wenruo
ck to ensure each chunk has its corresponding block group. Link: https://bugzilla.kernel.org/show_bug.cgi?id=199847 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 52 +- 1 file changed, 51 insertions(+), 1 deletion(-) diff --gi

[PATCH 2/5] btrfs: tree-checker: Detect invalid empty essential tree

2018-07-03 Thread Qu Wenruo
Signed-off-by: Qu Wenruo --- fs/btrfs/tree-checker.c | 12 1 file changed, 12 insertions(+) diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c index 1cd735b099df..3a307efab46b 100644 --- a/fs/btrfs/tree-checker.c +++ b/fs/btrfs/tree-checker.c @@ -497,8 +497,20 @@ static int

[PATCH 3/5] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-03 Thread Qu Wenruo
n't call set_reloc_control() thus fs_info->reloc_ctl is still NULL. Link: https://bugzilla.kernel.org/show_bug.cgi?id=199833 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/relocation.c | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/fs

[PATCH 1/5] btrfs: tree-checker: Verify block_group_item

2018-07-03 Thread Qu Wenruo
?id=199849 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/tree-checker.c | 101 fs/btrfs/volumes.c | 2 +- fs/btrfs/volumes.h | 2 + 3 files changed, 104 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs

[PATCH 0/5] Enhancement for block group/chunk verification

2018-07-03 Thread Qu Wenruo
letely in memory using rb_tree, so it shouldn't add too much overhead. Qu Wenruo (5): btrfs: tree-checker: Verify block_group_item btrfs: tree-checker: Detect invalid empty essential tree btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized btrfs: Check each block

Re: So, does btrfs check lowmem take days? weeks?

2018-07-03 Thread Qu Wenruo
On 2018年07月03日 12:22, Marc MERLIN wrote: > On Mon, Jul 02, 2018 at 06:31:43PM -0600, Chris Murphy wrote: >> So the idea behind journaled file systems is that journal replay >> enabled mount time "repair" that's faster than an fsck. Already Btrfs >> use cases with big, but not huge, file systems

Re: [PATCH v2 1/2] btrfs: Check each block group has corresponding chunk at mount time

2018-07-03 Thread Qu Wenruo
On 2018年07月03日 16:33, Nikolay Borisov wrote: > > > On 3.07.2018 11:08, Qu Wenruo wrote: >> Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199837, if a >> crafted btrfs with incorrect chunk<->block group mapping, it could leads >> to a lot of un

Re: A list of bugs in btrfs found by fuzzing

2018-07-03 Thread Qu Wenruo
On 2018年07月01日 08:06, Xu, Wen wrote: > Dear BTRFS developers, > > I would like to know if these issues are fixed or handled? Thanks. > > -Wen All these images should be addressed by the following patches now: https://patchwork.kernel.org/patch/10503415/

Re: ac0b4145d662 ("btrfs: scrub: Don't use inode pages for device replace") breaking btrfs/100

2018-07-03 Thread Qu Wenruo
On 2018年07月03日 15:58, Nikolay Borisov wrote: > Hello Qu, > > The commit from $SUBJECT breaks btrfs/100. Before that commit this test > takes around 25 seconds and it succeeds, whereas with this patch applied > the test is twice as fast but is broken after just a couple of iterations. > The

[PATCH v2 2/2] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-03 Thread Qu Wenruo
will do extra check to ensure each chunk has its corresponding block group. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Newly added --- fs/btrfs/extent-tree.c | 52 +- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/fs/btrfs

[PATCH v2 1/2] btrfs: Check each block group has corresponding chunk at mount time

2018-07-03 Thread Qu Wenruo
, len, flags) chunk for each block group at mount time. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Add better error message for each mismatch case. Rename function name, to co-operate with later patch. Add flags mismatch check. --- fs/btrfs

[PATCH] btrfs: tree-checker: Detect invalid empty essential tree

2018-07-03 Thread Qu Wenruo
-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/tree-checker.c | 12 1 file changed, 12 insertions(+) diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c index 1cd735b099df..3a307efab46b 100644 --- a/fs/btrfs/tree-checker.c +++ b/fs/btrfs/tree-checker.c @@ -497,8

[PATCH] btrfs: tree-checker: Detect invalid empty essential tree

2018-07-03 Thread Qu Wenruo
-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/tree-checker.c | 12 1 file changed, 12 insertions(+) diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c index 1cd735b099df..3a307efab46b 100644 --- a/fs/btrfs/tree-checker.c +++ b/fs/btrfs/tree-checker.c @@ -497,8

[PATCH v3] btrfs: tree-checker: Verify block_group_item

2018-07-03 Thread Qu Wenruo
the crafted image. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Format change for if() code block, added one new line for each block. Use "hweight64() > 1" to detect profile error, replacing the bloated !(flags & MASK == 0 || hweight64() == 1) condi

Re: [PATCH v2] btrfs: Add chunk type check in read a chunk

2018-07-03 Thread Qu Wenruo
-by: Xu Wen > Signed-off-by: Gu Jinxiang Reviewed-by: Qu Wenruo Thanks, Qu > --- > changelog: > v2: > deal with comment by Qu, add precise check for chunk type. > > fs/btrfs/volumes.c | 27 +++ > 1 file changed, 27 insertions(+) > > d

Re: [PATCH] btrfs: Add chunk type check in read a chunk

2018-07-02 Thread Qu Wenruo
On 2018年07月03日 12:27, Gu Jinxiang wrote: > Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199839, > which has a invalid chunk, not return error opportunlly. > > Add chunk type check in btrfs_check_chunk_valid, > to make error be returned in advance. > > Signed-off-by: Gu Jinxiang >

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 23:18, Marc MERLIN wrote: > Hi Qu, > > I'll split this part into a new thread: > >> 2) Don't keep unrelated snapshots in one btrfs. >>I totally understand that maintain different btrfs would hugely add >>maintenance pressure, but as explains, all snapshots share one >>

Re: [PATCH v2 1/2] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-07-02 Thread Qu Wenruo
On 2018年07月03日 06:12, David Sterba wrote: > On Wed, Jun 20, 2018 at 08:38:38AM +0800, Qu Wenruo wrote: >> In function handle_global_options(), we reset @optind to 1. >> However according to man page of getopt(3) NOTES section, if we need to >> rescan options later, @optind

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 22:48, David Sterba wrote: > On Mon, Jul 02, 2018 at 04:34:14PM +0800, Qu Wenruo wrote: >> As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, >> a crafted image with invalid block group items could make free space cache >> code to cause panic.

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 22:05, Marc MERLIN wrote: > On Mon, Jul 02, 2018 at 02:22:20PM +0800, Su Yue wrote: >>> Ok, that's 29MB, so it doesn't fit on pastebin: >>> http://marc.merlins.org/tmp/dshelf2_inspect.txt >>> >> Sorry Marc. After offline communication with Qu, both >> of us think the filesystem

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 16:37, Nikolay Borisov wrote: > > > On 2.07.2018 11:34, Qu Wenruo wrote: >> +if (hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) > 1) { >> +block_group_err(fs_info, leaf, slot, >> +"invalid profile flags, have 0x%l

[PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
) Chunk objectid 3) Type Exactly 1 bit set for type and no more than 1 bit set for profile 4) Used space No more than block group size. This should allow btrfs to detect and refuse to mount the crafted image. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Format change

[PATCH] btrfs: Do extra chunk block group mapping check at mount

2018-07-02 Thread Qu Wenruo
tected early and cause no harm. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 38 ++ 1 file changed, 38 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 3d9fe58c0080..7a11dc76cd6d 100644 --- a/fs/btrfs/

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 16:01, Nikolay Borisov wrote: > > > On 2.07.2018 10:53, Qu Wenruo wrote: >> >> >> On 2018年07月02日 15:31, Nikolay Borisov wrote: >>> >>> >>> On 2.07.2018 09:25, Qu Wenruo wrote: >>>> Reported in https://bugz

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 15:31, Nikolay Borisov wrote: > > > On 2.07.2018 09:25, Qu Wenruo wrote: >> Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an >> invalid tree reloc tree can cause kernel NULL pointer dereference when >> btrfs does s

Re: [PATCH] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 15:28, Nikolay Borisov wrote: > > > On 2.07.2018 08:53, Qu Wenruo wrote: >> As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, >> a crafted image with invalid block group items could make free space cache >> code to cause panic.

[PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
rol after all reloc roots are verified. So when we hit out: tag, we haven't call set_reloc_control() thus fs_info->reloc_ctl is still NULL. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/relocation.c | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-)

[PATCH] btrfs: tree-checker: Verify block_group_item

2018-07-01 Thread Qu Wenruo
) Chunk objectid 3) Type Exactly 1 bit set for type and no more than 1 bit set for profile 4) Used space No more than block group size. This should allow btrfs to detect and refuse to mount the crafted image. Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- fs/btrfs/tree-checker.c | 88

Re: btrfs check --readonly crash

2018-07-01 Thread Qu Wenruo
dling in `btrfs check` that picks up > this situation instead of crashing? Patch submitted already: https://patchwork.kernel.org/patch/10498479/ Thanks, Qu > > On Sat, Jun 30, 2018 at 10:07 PM, Qu Wenruo wrote: >> >> >> On 2018年07月01日 09:59, Lewis Diamond wrote: >&

Re: A list of bugs in btrfs found by fuzzing

2018-07-01 Thread Qu Wenruo
On 2018年07月01日 08:06, Xu, Wen wrote: > Dear BTRFS developers, > > I would like to know if these issues are fixed or handled? Thanks. Some bugs looks interesting and may be resolved by more checks in tree-checker. I'll take a look at them. Thanks for the detailed and comprehensive report, Qu

[PATCH] btrfs-progs: free-space-cache: Don't panic when free space cache is corrupted

2018-06-30 Thread Qu Wenruo
In btrfs_add_free_space(), if the free space to be added is already here, we trigger ASSERT() which is just another BUG_ON(). Let's remove such BUG_ON() at all. Reported-by: Lewis Diamond Signed-off-by: Qu Wenruo --- free-space-cache.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions

Re: btrfs check --readonly crash

2018-06-30 Thread Qu Wenruo
On 2018年07月01日 09:59, Lewis Diamond wrote: > Hi, > I've been told to report this issue to this mailing list. > > sudo btrfs check --readonly /dev/sdc > Checking filesystem on /dev/sdc > UUID: 2630aec8-8399-4bd8-9397-8c04953a35d5 > checking extents > checking free space cache > there is no free

Re: So, does btrfs check lowmem take days? weeks?

2018-06-30 Thread Qu Wenruo
On 2018年06月30日 10:44, Marc MERLIN wrote: > Well, there goes that. After about 18H: > ERROR: extent[156909494272, 55320576] referencer count mismatch (root: 21872, > owner: 374857, offset: 235175936) wanted: 1, have: 1452 > backref.c:466: __add_missing_keys: Assertion `ref->root_id` failed,

Re: btrfs suddenly think's it's raid6

2018-06-30 Thread Qu Wenruo
On 2018年06月30日 06:11, marble wrote: > Hey, > Thanks for the quick reply :-) > >> Is there anything like unexpected power loss happens? > Power loss may have happened. >> And would you provide the following data for debugging? >> >> # btrfs ins dump-super -fFa /dev/mapper/black > I attached it.

Re: [PATCH 3/4] btrfs: Rename EXTENT_BUFFER_DUMMY to EXTENT_BUFFER_PRIVATE

2018-06-29 Thread Qu Wenruo
On 2018年06月29日 20:46, David Sterba wrote: > On Wed, Jun 27, 2018 at 04:38:24PM +0300, Nikolay Borisov wrote: >> EXTENT_BUFFER_DUMMY is an awful name for this flag. Buffers which have >> this flag set are not in any way dummy. Rather, they are private in >> the sense that are not linked to the

Re: btrfs suddenly think's it's raid6

2018-06-29 Thread Qu Wenruo
On 2018年06月29日 19:04, marble wrote: > Hello, > I have an external HDD. The HDD contains no partition. > I use the whole HDD as a LUKS container. Inside that LUKS is a btrfs. > It's used to store some media files. > The HDD was hooked up to a Raspberry Pi running up-to-date Arch Linux > to play

Re: So, does btrfs check lowmem take days? weeks?

2018-06-29 Thread Qu Wenruo
On 2018年06月29日 14:59, Marc MERLIN wrote: > On Fri, Jun 29, 2018 at 02:29:10PM +0800, Qu Wenruo wrote: >>> If --repair doesn't work, check is useless to me sadly. >> >> Not exactly. >> Although it's time consuming, I have manually patched several users fs, >

Re: So, does btrfs check lowmem take days? weeks?

2018-06-29 Thread Qu Wenruo
On 2018年06月29日 14:06, Marc MERLIN wrote: > On Fri, Jun 29, 2018 at 01:48:17PM +0800, Qu Wenruo wrote: >> Just normal btrfs check, and post the output. >> If normal check eats up all your memory, btrfs check --mode=lowmem. > > Does check without --repair eat les

Re: So, does btrfs check lowmem take days? weeks?

2018-06-28 Thread Qu Wenruo
On 2018年06月29日 13:28, Marc MERLIN wrote: > On Fri, Jun 29, 2018 at 01:07:20PM +0800, Qu Wenruo wrote: >>> lowmem repair seems to be going still, but it's been days and -p seems >>> to do absolutely nothing. >> >> I'm a afraid you hit a bug in lowmem repai

Re: So, does btrfs check lowmem take days? weeks?

2018-06-28 Thread Qu Wenruo
On 2018年06月29日 12:27, Marc MERLIN wrote: > Regular btrfs check --repair has a nice progress option. It wasn't > perfect, but it showed something. > > But then it also takes all your memory quicker than the linux kernel can > defend itself and reliably completely kills my 32GB server quicker

Re: call trace: WARNING: at /build/linux-uwVqDp/linux-4.16.16/fs/btrfs/ctree.h:1565 btrfs_update_device

2018-06-28 Thread Qu Wenruo
On 2018年06月29日 09:05, Christoph Anton Mitterer wrote: > Hey Qu and Nikolay. > > > On Thu, 2018-06-28 at 22:58 +0800, Qu Wenruo wrote: >> Nothing special. Btrfs-progs will handle it pretty well. > Since this a remote system where the ISP provides only a rescue image >

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 22:36, Adam Borowski wrote: > On Thu, Jun 28, 2018 at 03:04:43PM +0800, Qu Wenruo wrote: >> There is a reporter considering btrfs raid1 has a major design flaw >> which can't handle nodatasum files. >> >> Despite his incorrect expectation, btrfs i

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-28 Thread Qu Wenruo
On 2018年06月29日 01:10, Andrei Borzenkov wrote: > 28.06.2018 12:15, Qu Wenruo пишет: >> >> >> On 2018年06月28日 16:16, Andrei Borzenkov wrote: >>> On Thu, Jun 28, 2018 at 8:39 AM, Qu Wenruo wrote: >>>> >>>> >>>> On 2018年06月28日 11:14,

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 16:04, Nikolay Borisov wrote: > > > On 28.06.2018 10:04, Qu Wenruo wrote: >> There is a reporter considering btrfs raid1 has a major design flaw >> which can't handle nodatasum files. >> >> Despite his incorrect expectation, btrfs indeed do

Re: call trace: WARNING: at /build/linux-uwVqDp/linux-4.16.16/fs/btrfs/ctree.h:1565 btrfs_update_device

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 22:17, Christoph Anton Mitterer wrote: > On Thu, 2018-06-28 at 22:09 +0800, Qu Wenruo wrote: >>> [ 72.168662] WARNING: CPU: 0 PID: 242 at /build/linux- >>> uwVqDp/linux-4.16.16/fs/btrfs/ctree.h:1565 >>> btrfs_update_device+0x1b2/0x1c0It >

Re: call trace: WARNING: at /build/linux-uwVqDp/linux-4.16.16/fs/btrfs/ctree.h:1565 btrfs_update_device

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 21:19, Christoph Anton Mitterer wrote: > Hey. > > On a 4.16.16 kernel with a RAID 1 btrfs I got the following messages > since today. > > Data seems still to be readable (correctly)... and there are no other > errors (like SATA errors) in the kernel log. > > Any idea what these

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 21:26, Anand Jain wrote: > > > On 06/28/2018 04:02 PM, Qu Wenruo wrote: >> Also CC Anand Jain, since he is also working on various device related >> work, and an expert on this. >> >> Especially I'm not pretty sure if such enhanc

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 19:12, Austin S. Hemmelgarn wrote: > On 2018-06-28 05:15, Qu Wenruo wrote: >> >> >> On 2018年06月28日 16:16, Andrei Borzenkov wrote: >>> On Thu, Jun 28, 2018 at 8:39 AM, Qu Wenruo >>> wrote: >>>> >>>> >>>>

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 16:16, Andrei Borzenkov wrote: > On Thu, Jun 28, 2018 at 8:39 AM, Qu Wenruo wrote: >> >> >> On 2018年06月28日 11:14, r...@georgianit.com wrote: >>> >>> >>> On Wed, Jun 27, 2018, at 10:55 PM, Qu Wenruo wrote: >>> >>

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
Also CC Anand Jain, since he is also working on various device related work, and an expert on this. Especially I'm not pretty sure if such enhancement is already on his agenda or there are already some unmerged patch for this. Thanks, Qu On 2018年06月28日 15:04, Qu Wenruo wrote

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 15:17, Su Yue wrote: > > > On 06/28/2018 03:04 PM, Qu Wenruo wrote: >> There is a reporter considering btrfs raid1 has a major design flaw >> which can't handle nodatasum files. >> >> Despite his incorrect expectation, btrfs indeed doesn't han

Re: [PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
On 2018年06月28日 15:06, Nikolay Borisov wrote: > > > On 28.06.2018 10:04, Qu Wenruo wrote: >> There is a reporter considering btrfs raid1 has a major design flaw >> which can't handle nodatasum files. >> >> Despite his incorrect expectation, btrfs indeed do

Re: Enabling quota may not correctly rescan on 4.17

2018-06-28 Thread Qu Wenruo
On 2018年06月27日 16:25, Misono Tomohiro wrote: > On 2018/06/27 17:10, Qu Wenruo wrote: >> >> >> On 2018年06月26日 14:00, Misono Tomohiro wrote: >>> Hello Nikolay, >>> >>> I noticed that commit 5d23515be669 ("btrfs: Move qgroup rescan >>

[PATCH RFC] btrfs: Do extra device generation check at mount time

2018-06-28 Thread Qu Wenruo
using "device" and "degraded" mount option (if possible), then replace the offending device to manually "rebuild" the fs. Signed-off-by: Qu Wenruo --- I totally understand that, generation based solution can't handle split-brain case (where 2 RAID1 devices get mount

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-27 Thread Qu Wenruo
On 2018年06月28日 11:14, r...@georgianit.com wrote: > > > On Wed, Jun 27, 2018, at 10:55 PM, Qu Wenruo wrote: > >> >> Please get yourself clear of what other raid1 is doing. > > A drive failure, where the drive is still there when the computer reboots, is &

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-27 Thread Qu Wenruo
On 2018年06月28日 10:10, Remi Gauvin wrote: > On 2018-06-27 09:58 PM, Qu Wenruo wrote: >> >> >> On 2018年06月28日 09:42, Remi Gauvin wrote: >>> There seems to be a major design flaw with BTRFS that needs to be better >>> documented, to avoid massive data los

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-06-27 Thread Qu Wenruo
On 2018年06月28日 09:42, Remi Gauvin wrote: > There seems to be a major design flaw with BTRFS that needs to be better > documented, to avoid massive data loss. > > Tested with Raid 1 on Ubuntu Kernel 4.15 > > The use case being tested was a Virtualbox VDI file created with > NODATACOW attribute,

Re: [PATCH] Btrfs: fix mount failure when qgroup rescan is in progress

2018-06-27 Thread Qu Wenruo
UP_STATUS_FLAG_ON > are set. My fault. > > A test case for fstests follows up soon. > > Fixes: 9593bf49675e ("btrfs: qgroup: show more meaningful qgroup_rescan_init > error message") > Signed-off-by: Filipe Manana Reviewed-by: Qu Wenruo Thanks, Qu > ---

[PATCH v1] btrfs: quota: Set rescan progress to (u64)-1 if we hit last leaf

2018-06-27 Thread Qu Wenruo
fs: qgroup: Finish rescan when hit the last leaf of extent tree") Signed-off-by: Qu Wenruo --- changelog: v2: Commit message update, as the bug only changes the resulting quota status item without impacting the behavior. --- fs/btrfs/qgroup.c | 4 +++- 1 file changed, 3 insertions(+), 1

Re: Enabling quota may not correctly rescan on 4.17

2018-06-27 Thread Qu Wenruo
On 2018年06月27日 16:57, Qu Wenruo wrote: > > > On 2018年06月27日 16:47, Nikolay Borisov wrote: >> >> >> On 27.06.2018 11:38, Qu Wenruo wrote: >>> >>> >>> On 2018年06月27日 16:34, Qu Wenruo wrote: >>>> >>>> >>>&g

Re: Enabling quota may not correctly rescan on 4.17

2018-06-27 Thread Qu Wenruo
On 2018年06月27日 16:47, Nikolay Borisov wrote: > > > On 27.06.2018 11:38, Qu Wenruo wrote: >> >> >> On 2018年06月27日 16:34, Qu Wenruo wrote: >>> >>> >>> On 2018年06月27日 16:25, Misono Tomohiro wrote: >>>> On 2018/06/27 17:10, Qu Wen

[PATCH] btrfs: quota: Reset rescan progress if we hit last leaf

2018-06-27 Thread Qu Wenruo
e whole quota accounting. Reported-by: Misono Tomohiro Fixes: ff3d27a048d9 ("btrfs: qgroup: Finish rescan when hit the last leaf of extent tree") Signed-off-by: Qu Wenruo --- fs/btrfs/qgroup.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/qgroup.c b/

Re: Enabling quota may not correctly rescan on 4.17

2018-06-27 Thread Qu Wenruo
On 2018年06月27日 16:34, Qu Wenruo wrote: > > > On 2018年06月27日 16:25, Misono Tomohiro wrote: >> On 2018/06/27 17:10, Qu Wenruo wrote: >>> >>> >>> On 2018年06月26日 14:00, Misono Tomohiro wrote: >>>> Hello Nikolay, >>>> &

Re: Enabling quota may not correctly rescan on 4.17

2018-06-27 Thread Qu Wenruo
On 2018年06月27日 16:25, Misono Tomohiro wrote: > On 2018/06/27 17:10, Qu Wenruo wrote: >> >> >> On 2018年06月26日 14:00, Misono Tomohiro wrote: >>> Hello Nikolay, >>> >>> I noticed that commit 5d23515be669 ("btrfs: Move qgroup rescan >>

Re: Enabling quota may not correctly rescan on 4.17

2018-06-27 Thread Qu Wenruo
On 2018年06月26日 14:00, Misono Tomohiro wrote: > Hello Nikolay, > > I noticed that commit 5d23515be669 ("btrfs: Move qgroup rescan > on quota enable to btrfs_quota_enable") in 4.17 sometimes causes > to fail correctly rescanning quota when quota is enabled. > > Simple reproducer: > > $

Re: [PATCH] btrfs: qgroups: Move transaction managed inside btrfs_quota_enable

2018-06-27 Thread Qu Wenruo
On 2018年06月26日 16:46, Misono Tomohiro wrote: > On 2018/06/26 16:09, Nikolay Borisov wrote: >> Commit 5d23515be669 ("btrfs: Move qgroup rescan on quota enable to >> btrfs_quota_enable") not only resulted in an easier to follow code but >> it also introduced a subtle bug. It changed the timing

Re: Bad superblock when mounting rw, ro mount works

2018-06-22 Thread Qu Wenruo
back 0x5582efb12930 > backpointer mismatch on [49230998155264 16384] > adding new data backref on 49230998155264 root 5 owner 669291 offset > 3905650688 found 1 Looks it's processing. But it's better to keep all the output and verify the same section of bytenr doesn't re-appear too many

Re: [PATCH v4] btrfs: Don't remove block group still has pinned down bytes

2018-06-22 Thread Qu Wenruo
On 2018年06月22日 17:52, David Sterba wrote: > On Fri, Jun 22, 2018 at 12:35:00PM +0800, Qu Wenruo wrote: >> [BUG] >> Under certain KVM load and LTP tests, we are possible to hit the >> following calltrace if quota is enabled: >> -- >> BTRFS critical (dev

Re: [PATCH] btrfs: treat -ERANGE as an error case in btrfs_get_acl()

2018-06-22 Thread Qu Wenruo
On 2018年06月22日 15:42, cgxu519 wrote: > On 06/22/2018 01:59 PM, Qu Wenruo wrote: >> >> On 2018年06月22日 10:58, Chengguang Xu wrote: >>> Currently, when encoutering -ERANGE in btrfs_get_acl(), >>> just set acl to NULL so that we cannot get proper >>&g

Re: [PATCH] btrfs: treat -ERANGE as an error case in btrfs_get_acl()

2018-06-22 Thread Qu Wenruo
On 2018年06月22日 10:58, Chengguang Xu wrote: > Currently, when encoutering -ERANGE in btrfs_get_acl(), > just set acl to NULL so that we cannot get proper > acl information but the operation looks successful. > > This patch treats -ERANGE as an error case and meanwhile > print real errno before

[PATCH v4] btrfs: Don't remove block group still has pinned down bytes

2018-06-21 Thread Qu Wenruo
blocks at update_block_group(), and such empty block group with pinned bytes won't go through update_block_group() again, such block group won't be removed, until it get new extent allocated and removed. Signed-off-by: Qu Wenruo --- changelog: v2: Commit message update, to better indicate how pinne

Re: [PATCH v3] btrfs: Don't remove block group still has pinned down bytes

2018-06-21 Thread Qu Wenruo
On 2018年06月22日 00:34, Filipe Manana wrote: > On Wed, Jun 20, 2018 at 12:03 PM, Qu Wenruo wrote: >> >> >> On 2018年06月20日 17:33, Filipe Manana wrote: >>> On Wed, Jun 20, 2018 at 10:22 AM, Qu Wenruo wrote: >>>> >>>> >>>> On 2018年

Re: (about compiler optimization) [PATCH 00/34] fs_info cleanup of extent-tree.c

2018-06-21 Thread Qu Wenruo
On 2018年06月20日 20:48, Nikolay Borisov wrote: > Hello, > > This series aims at removing all the redundant btrfs_fs_info args being > passed to functions in extent-tree.c. Each patch removes the arg from a > one function hence it should be fairly easy to review each one of those > patches.

Re: [PATCH] btrfs: check-integrity: Fix NULL pointer dereference for degraded mount

2018-06-21 Thread Qu Wenruo
On 2018年06月21日 21:58, David Sterba wrote: > On Wed, Jun 20, 2018 at 03:38:58PM +0800, Qu Wenruo wrote: >> Commit f8f84b2dfda5 ("btrfs: index check-integrity state hash by a dev_t") >> changed how btrfsic how we index device state hash. >> >> Now we need

Re: [PATCH 00/34] fs_info cleanup of extent-tree.c

2018-06-20 Thread Qu Wenruo
sra 543 --543 > lookup_extent_data_ref.isra 649 --649 > insert_extent_data_ref.isra 729 --729 > Total: Before=91208, After=90929, chg -0.31% > > Overall this series is a win both in code density as well as sta

Re: [PATCH 34/34] btrfs: Remove fs_info from convert_extent_item_v0

2018-06-20 Thread Qu Wenruo
On 2018年06月20日 20:49, Nikolay Borisov wrote: > It can be referenced from trans since the function is always called > within a transaction > > Signed-off-by: Nikolay Borisov > --- > fs/btrfs/extent-tree.c | 10 -- > 1 file changed, 4 insertions(+), 6 deletions(-) > > diff --git

Re: [PATCH v3] btrfs: Don't remove block group still has pinned down bytes

2018-06-20 Thread Qu Wenruo
On 2018年06月20日 17:33, Filipe Manana wrote: > On Wed, Jun 20, 2018 at 10:22 AM, Qu Wenruo wrote: >> >> >> On 2018年06月20日 17:13, Filipe Manana wrote: >>> On Fri, Jun 15, 2018 at 2:35 AM, Qu Wenruo wrote: >>>> [BUG] >>>> Under

Re: [PATCH v3] btrfs: Don't remove block group still has pinned down bytes

2018-06-20 Thread Qu Wenruo
On 2018年06月20日 17:13, Filipe Manana wrote: > On Fri, Jun 15, 2018 at 2:35 AM, Qu Wenruo wrote: >> [BUG] >> Under certain KVM load and LTP tests, we are possible to hit the >> following calltrace if quota is enabled: >> -- >> BTRFS critical (device vda2):

[PATCH] btrfs: check-integrity: Fix NULL pointer dereference for degraded mount

2018-06-20 Thread Qu Wenruo
from btrfsic_map_block(), so it won't cause harm. Fixes: f8f84b2dfda5 ("btrfs: index check-integrity state hash by a dev_t") Signed-off-by: Qu Wenruo --- Please note there are still quite some problem about check-integrity, including: 1) Warning for degraded mount 2) Meaningless empty lin

Re: About commit f8f84b2dfda5 ("btrfs: index check-integrity state hash by a dev_t")

2018-06-20 Thread Qu Wenruo
On 2018年06月20日 14:36, Qu Wenruo wrote: > Hi Liu and David, > > I see mainline kernel merged commit f8f84b2dfda5 ("btrfs: index > check-integrity state hash by a dev_t"), and you reviewed it but I can't > find the mail in mail list, nor it's signed by David. > &g

About commit f8f84b2dfda5 ("btrfs: index check-integrity state hash by a dev_t")

2018-06-20 Thread Qu Wenruo
Hi Liu and David, I see mainline kernel merged commit f8f84b2dfda5 ("btrfs: index check-integrity state hash by a dev_t"), and you reviewed it but I can't find the mail in mail list, nor it's signed by David. Is this btrfs related commit merged without review from btrfs community? (Well, at

Re: [PATCH v3] btrfs: Don't remove block group still has pinned down bytes

2018-06-19 Thread Qu Wenruo
On 2018年06月20日 13:25, Nikolay Borisov wrote: > > > On 15.06.2018 04:35, Qu Wenruo wrote: >> [BUG] >> Under certain KVM load and LTP tests, we are possible to hit the >> following calltrace if quota is enabled: >> -- >> BTRFS critical (device vda2): u

Re: [PATCH v3] btrfs: Don't remove block group still has pinned down bytes

2018-06-19 Thread Qu Wenruo
Gentle ping. Since it's causing LTP tests failure, I'd like to backport it asap. Thanks, Qu On 2018年06月15日 09:35, Qu Wenruo wrote: > [BUG] > Under certain KVM load and LTP tests, we are possible to hit the > following calltrace if quota is enabled: > -- > BTRFS critic

[PATCH v2 1/2] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-06-19 Thread Qu Wenruo
trfs-progs: rework option parser to use getopt for global options") Signed-off-by: Qu Wenruo --- changelog: v2: Instead of resetting @optind at handle_global_options(), reset @optind before each later getopt() call. Since there are cases uses @optind and expects it to be starti

[PATCH 2/2] btrfs-progs: misc-tests: Fix 029 test cases for sudo test environment

2018-06-19 Thread Qu Wenruo
Test misc/029 only works if the test case is executed as root, while for sudo usage, it doesn't work as initial mkdir and final cleanup doesn't use $SUDO_HELPER. Add "run_check $SUDO_HELPER" for such cases to allow it works under sudo usage. Signed-off-by: Qu Wenruo --- tests/misc

Re: About more loose parameter sequence requirement

2018-06-19 Thread Qu Wenruo
On 2018年06月20日 07:18, Qu Wenruo wrote: > > > On 2018年06月19日 22:47, David Sterba wrote: >> On Mon, Jun 18, 2018 at 09:05:59PM +0800, Qu Wenruo wrote: >>>> New code needs to be tested, documented and maintained, that's the cost >>>> I find too high for some

Re: About more loose parameter sequence requirement

2018-06-19 Thread Qu Wenruo
On 2018年06月19日 22:47, David Sterba wrote: > On Mon, Jun 18, 2018 at 09:05:59PM +0800, Qu Wenruo wrote: >>> New code needs to be tested, documented and maintained, that's the cost >>> I find too high for something that's convenience for people who are used >&

Re: [PATCH 2/3] btrfs: Document __btrfs_inc_extent_ref

2018-06-18 Thread Qu Wenruo
On 2018年06月18日 19:59, Nikolay Borisov wrote: > Here is a doc-only patch which tires to deobfuscate the terra-incognita > that arguments for delayed refs are. > > Signed-off-by: Nikolay Borisov Reviewed-by: Qu Wenruo > --- > fs/btrfs/

Re: [PATCH 1/3] btrfs: Remove fs_info argument from __btrfs_inc_extent_ref

2018-06-18 Thread Qu Wenruo
On 2018年06月18日 19:59, Nikolay Borisov wrote: > This function already takes a transaction which holds a reference to > the fs_info struct. Use that reference and remove the extra arg. No > functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Qu Wenruo Thanks,

[PATCH] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-06-18 Thread Qu Wenruo
trfs-progs: rework option parser to use getopt for global options") Signed-off-by: Qu Wenruo --- btrfs.c | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/btrfs.c b/btrfs.c index 2d39f2ced3e8..ebc4a507165d 100644 --- a/btrfs.c +++ b/btrfs.c @@ -169,7 +169,7 @@ s

Re: About more loose parameter sequence requirement

2018-06-18 Thread Qu Wenruo
On 2018年06月19日 12:34, Qu Wenruo wrote: > > > On 2018年06月18日 20:02, David Sterba wrote: >> On Mon, Jun 18, 2018 at 07:40:44PM +0800, Qu Wenruo wrote: >>> >>> >>> On 2018年06月18日 19:34, David Sterba wrote: >>>> On Thu, Jun 14, 2018 at

Re: About more loose parameter sequence requirement

2018-06-18 Thread Qu Wenruo
On 2018年06月18日 20:02, David Sterba wrote: > On Mon, Jun 18, 2018 at 07:40:44PM +0800, Qu Wenruo wrote: >> >> >> On 2018年06月18日 19:34, David Sterba wrote: >>> On Thu, Jun 14, 2018 at 03:17:45PM +0800, Qu Wenruo wrote: >>>> I understand that btrfs-progs

<    3   4   5   6   7   8   9   10   11   12   >