Re: [PATCH 2/2] btrfs: drop devices declare in btrfs_init_new_device()

2018-07-02 Thread Nikolay Borisov
On 3.07.2018 08:14, Anand Jain wrote: > There is only one usage of the declared devices variable, instead > use its value directly. > > Signed-off-by: Anand Jain Reviewed-by: Nikolay Borisov > --- > fs/btrfs/volumes.c | 5 + > 1 file changed, 1 insertion(+), 4 deletions(-) > > diff

Re: [PATCH 2/2] btrfs: fix missing superblock update in the device delete commit transaction

2018-07-02 Thread Nikolay Borisov
On 3.07.2018 08:12, Anand Jain wrote: > When a device is deleted, the btrfs_super_block::number_devices is > reduced by 1, but we do that after the commit transaction, so this > change did not made it to the disk and waits for the next commit > transaction whenever it happens. > > This can be

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
On Tue, Jul 03, 2018 at 04:26:37AM +, Paul Jones wrote: > I don't have any experience with this, but since it's the internet let me > tell you how I'd do it anyway  That's the spirit :) > raid5 > dm-crypt > lvm (using thin provisioning + cache) > btrfs > > The cache mode on lvm requires

Re: [PATCH] btrfs: Add chunk type check in read a chunk

2018-07-02 Thread Qu Wenruo
On 2018年07月03日 12:27, Gu Jinxiang wrote: > Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199839, > which has a invalid chunk, not return error opportunlly. > > Add chunk type check in btrfs_check_chunk_valid, > to make error be returned in advance. > > Signed-off-by: Gu Jinxiang >

[PATCH 2/2] btrfs: drop devices declare in btrfs_init_new_device()

2018-07-02 Thread Anand Jain
There is only one usage of the declared devices variable, instead use its value directly. Signed-off-by: Anand Jain --- fs/btrfs/volumes.c | 5 + 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index f7fa0ea26e9c..124bd8728c37 100644 ---

[PATCH 1/2] btrfs: declare fs_devices in btrfs_init_new_device()

2018-07-02 Thread Anand Jain
There are many instances of the %fs_info->fs_devices pointer de-reference, so declare a %fs_devices pointer instead. Signed-off-by: Anand Jain --- fs/btrfs/volumes.c | 42 +- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git

[PATCH RESEND 1/2] btrfs: fix parent in memory total_devices after seed delete

2018-07-02 Thread Anand Jain
In case of deleting the seed device the %cur_devices (seed) and the %fs_devices (parent) are different. Now, as the parent fs_devices::total_devices also maintains the total number of devices including the seed device, so decrement its in-memory value for the successful seed delete. We are already

[PATCH 2/2] btrfs: fix missing superblock update in the device delete commit transaction

2018-07-02 Thread Anand Jain
When a device is deleted, the btrfs_super_block::number_devices is reduced by 1, but we do that after the commit transaction, so this change did not made it to the disk and waits for the next commit transaction whenever it happens. This can be easily demonstrated using the following test case

[PATCH] btrfs: Add chunk type check in read a chunk

2018-07-02 Thread Gu Jinxiang
Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199839, which has a invalid chunk, not return error opportunlly. Add chunk type check in btrfs_check_chunk_valid, to make error be returned in advance. Signed-off-by: Gu Jinxiang --- fs/btrfs/volumes.c | 6 ++ 1 file changed, 6

RE: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Paul Jones
> -Original Message- > From: Marc MERLIN > Sent: Tuesday, 3 July 2018 2:07 PM > To: Paul Jones > Cc: linux-btrfs@vger.kernel.org > Subject: Re: how to best segment a big block device in resizeable btrfs > filesystems? > > On Tue, Jul 03, 2018 at 12:51:30AM +, Paul Jones wrote: > >

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Andrei Borzenkov
02.07.2018 21:35, Austin S. Hemmelgarn пишет: > them (trimming blocks on BTRFS gets rid of old root trees, so it's a bit > dangerous to do it while writes are happening). Could you please elaborate? Do you mean btrfs can trim data before new writes are actually committed to disk? -- To

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Andrei Borzenkov
03.07.2018 04:37, Qu Wenruo пишет: > > BTW, IMHO the bcache is not really helping for backup system, which is > more write oriented. > There is new writecache target which may help in this case. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Marc MERLIN
On Mon, Jul 02, 2018 at 06:31:43PM -0600, Chris Murphy wrote: > So the idea behind journaled file systems is that journal replay > enabled mount time "repair" that's faster than an fsck. Already Btrfs > use cases with big, but not huge, file systems makes btrfs check a > problem. Either running

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
On Tue, Jul 03, 2018 at 09:37:47AM +0800, Qu Wenruo wrote: > > If I do this, I would have > > software raid 5 < dmcrypt < bcache < lvm < btrfs > > That's a lot of layers, and that's also starting to make me nervous :) > > If you could keep the number of snapshots to minimal (less than 10) for >

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
On Tue, Jul 03, 2018 at 12:51:30AM +, Paul Jones wrote: > You could combine bcache and lvm if you are happy to use dm-cache instead > (which lvm uses). > I use it myself (but without thin provisioning) and it works well. Interesting point. So, I used to use lvm and then lvm2 many years ago

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 23:18, Marc MERLIN wrote: > Hi Qu, > > I'll split this part into a new thread: > >> 2) Don't keep unrelated snapshots in one btrfs. >>I totally understand that maintain different btrfs would hugely add >>maintenance pressure, but as explains, all snapshots share one >>

Re: [PATCH v2 1/2] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-07-02 Thread Qu Wenruo
On 2018年07月03日 06:12, David Sterba wrote: > On Wed, Jun 20, 2018 at 08:38:38AM +0800, Qu Wenruo wrote: >> In function handle_global_options(), we reset @optind to 1. >> However according to man page of getopt(3) NOTES section, if we need to >> rescan options later, @optind should be reset to 0

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 22:48, David Sterba wrote: > On Mon, Jul 02, 2018 at 04:34:14PM +0800, Qu Wenruo wrote: >> As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, >> a crafted image with invalid block group items could make free space cache >> code to cause panic. >> >> We could early

RE: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Paul Jones
> -Original Message- > From: linux-btrfs-ow...@vger.kernel.org ow...@vger.kernel.org> On Behalf Of Marc MERLIN > Sent: Tuesday, 3 July 2018 1:19 AM > To: Qu Wenruo > Cc: Su Yue ; linux-btrfs@vger.kernel.org > Subject: Re: how to best segment a big block device in resizeable btrfs >

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Chris Murphy
On Mon, Jul 2, 2018 at 8:42 AM, Qu Wenruo wrote: > > > On 2018年07月02日 22:05, Marc MERLIN wrote: >> On Mon, Jul 02, 2018 at 02:22:20PM +0800, Su Yue wrote: Ok, that's 29MB, so it doesn't fit on pastebin: http://marc.merlins.org/tmp/dshelf2_inspect.txt >>> Sorry Marc. After offline

[PATCH] Btrfs: make sure there is always room for generation number

2018-07-02 Thread Zhihui Zhang
io_ctl_set_generation() assumes that the generation number shares the same page with inline CRCs. Let's make sure this is always true. Signed-off-by: Zhihui Zhang --- fs/btrfs/free-space-cache.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/free-space-cache.c

Re: [PATCH 0/6] btrfs-progs: Fixes inline ram_bytes related bugs

2018-07-02 Thread David Sterba
On Thu, Jun 07, 2018 at 10:37:16PM -0600, Steve Leung wrote: > On 06/06/2018 01:27 AM, Qu Wenruo wrote: > > The patchset can be fetched from github (*): > > https://github.com/adam900710/btrfs-progs/tree/inline_ram_bytes > > > > It's based on David's devel branch, whose HEAD is: > > commit

Re: [PATCH 0/6] btrfs-progs: Fixes inline ram_bytes related bugs

2018-07-02 Thread David Sterba
On Wed, Jun 06, 2018 at 03:27:11PM +0800, Qu Wenruo wrote: > The patchset can be fetched from github (*): > https://github.com/adam900710/btrfs-progs/tree/inline_ram_bytes > > It's based on David's devel branch, whose HEAD is: > commit 0d1c5812e28e286648781c7b35b542311cc01aa4 (david/devel) >

Re: [PATCH v2 1/2] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-07-02 Thread David Sterba
On Wed, Jun 20, 2018 at 08:38:38AM +0800, Qu Wenruo wrote: > In function handle_global_options(), we reset @optind to 1. > However according to man page of getopt(3) NOTES section, if we need to > rescan options later, @optind should be reset to 0 to initialize the > internal variables correctly.

Re: [PATCH] btrfs-progs: free-space-cache: Don't panic when free space cache is corrupted

2018-07-02 Thread David Sterba
On Sun, Jul 01, 2018 at 10:45:26AM +0800, Qu Wenruo wrote: > In btrfs_add_free_space(), if the free space to be added is already > here, we trigger ASSERT() which is just another BUG_ON(). > > Let's remove such BUG_ON() at all. > > Reported-by: Lewis Diamond > Signed-off-by: Qu Wenruo

Re: [PATCH] btrfs-progs: check: Fix wrong root parameter of btrfs_next_leaf call

2018-07-02 Thread David Sterba
On Mon, Jun 18, 2018 at 02:10:39PM +0300, Nikolay Borisov wrote: > The first thing that check_chunks_and_extents does is to iterate all > the root items in the root tree and link them to either the "normal_list" > or "dropping_trees" list. If a leaf has to be crossed during this > operation

Re: [PATCH] btrfs-progs: Check factor out root parsing from check_chunks_and_extents

2018-07-02 Thread David Sterba
On Mon, Jun 18, 2018 at 02:10:58PM +0300, Nikolay Borisov wrote: > check_chunks_and_extents does quite a number of distinct things. The > first of those is going through all root items in the root tree and > classify every root depending on whether they have a dropping operation > in progress or

Re: [PATCH v2 1/2] btrfs-progs: Fix wrong optind re-initialization to allow mixed option and non-option

2018-07-02 Thread David Sterba
On Wed, Jun 20, 2018 at 08:38:38AM +0800, Qu Wenruo wrote: > In function handle_global_options(), we reset @optind to 1. > However according to man page of getopt(3) NOTES section, if we need to > rescan options later, @optind should be reset to 0 to initialize the > internal variables correctly.

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
On Mon, Jul 02, 2018 at 02:35:19PM -0400, Austin S. Hemmelgarn wrote: > >I kind of linked the thin provisioning idea because it's hands off, > >which is appealing. Any reason against it? > No, not currently, except that it adds a whole lot more stuff between > BTRFS and whatever layer is below

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-07-02 13:34, Marc MERLIN wrote: On Mon, Jul 02, 2018 at 12:59:02PM -0400, Austin S. Hemmelgarn wrote: Am I supposed to put LVM thin volumes underneath so that I can share the same single 10TB raid5? Actually, because of the online resize ability in BTRFS, you don't technically _need_

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Marc MERLIN
On Mon, Jul 02, 2018 at 10:33:09PM +0500, Roman Mamedov wrote: > On Mon, 2 Jul 2018 08:19:03 -0700 > Marc MERLIN wrote: > > > I actually have fewer snapshots than this per filesystem, but I backup > > more than 10 filesystems. > > If I used as many snapshots as you recommend, that would already

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
On Mon, Jul 02, 2018 at 12:59:02PM -0400, Austin S. Hemmelgarn wrote: > > Am I supposed to put LVM thin volumes underneath so that I can share > > the same single 10TB raid5? > > Actually, because of the online resize ability in BTRFS, you don't > technically _need_ to use thin provisioning here.

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Roman Mamedov
On Mon, 2 Jul 2018 08:19:03 -0700 Marc MERLIN wrote: > I actually have fewer snapshots than this per filesystem, but I backup > more than 10 filesystems. > If I used as many snapshots as you recommend, that would already be 230 > snapshots for 10 filesystems :) (...once again me with my rsync

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-07-02 11:19, Marc MERLIN wrote: Hi Qu, thanks for the detailled and honest answer. A few comments inline. On Mon, Jul 02, 2018 at 10:42:40PM +0800, Qu Wenruo wrote: For full, it depends. (but for most real world case, it's still flawed) We have small and crafted images as test cases,

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-07-02 11:18, Marc MERLIN wrote: Hi Qu, I'll split this part into a new thread: 2) Don't keep unrelated snapshots in one btrfs. I totally understand that maintain different btrfs would hugely add maintenance pressure, but as explains, all snapshots share one fragile extent

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread David Sterba
On Mon, Jul 02, 2018 at 02:25:38PM +0800, Qu Wenruo wrote: > Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an > invalid tree reloc tree can cause kernel NULL pointer dereference when > btrfs does some cleanup for reloc roots. > > It turns out that fs_info->reloc_ctl can be

Re: [PATCH] btrfs: qgroups: Move transaction managed inside btrfs_quota_enable

2018-07-02 Thread David Sterba
On Mon, Jul 02, 2018 at 02:00:34PM +0300, Nikolay Borisov wrote: > Commit 5d23515be669 ("btrfs: Move qgroup rescan on quota enable to > btrfs_quota_enable") not only resulted in an easier to follow code but > it also introduced a subtle bug. It changed the timing when the initial > transaction

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Marc MERLIN
Hi Qu, thanks for the detailled and honest answer. A few comments inline. On Mon, Jul 02, 2018 at 10:42:40PM +0800, Qu Wenruo wrote: > For full, it depends. (but for most real world case, it's still flawed) > We have small and crafted images as test cases, which btrfs check can > repair without

Re: how to best segment a big block device in resizeable btrfs filesystems?

2018-07-02 Thread Marc MERLIN
Hi Qu, I'll split this part into a new thread: > 2) Don't keep unrelated snapshots in one btrfs. >I totally understand that maintain different btrfs would hugely add >maintenance pressure, but as explains, all snapshots share one >fragile extent tree. Yes, I understand that this is

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread David Sterba
On Mon, Jul 02, 2018 at 04:34:14PM +0800, Qu Wenruo wrote: > As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, > a crafted image with invalid block group items could make free space cache > code to cause panic. > > We could early detect such invalid block group item by checking:

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 22:05, Marc MERLIN wrote: > On Mon, Jul 02, 2018 at 02:22:20PM +0800, Su Yue wrote: >>> Ok, that's 29MB, so it doesn't fit on pastebin: >>> http://marc.merlins.org/tmp/dshelf2_inspect.txt >>> >> Sorry Marc. After offline communication with Qu, both >> of us think the filesystem

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Marc MERLIN
On Mon, Jul 02, 2018 at 02:22:20PM +0800, Su Yue wrote: > > Ok, that's 29MB, so it doesn't fit on pastebin: > > http://marc.merlins.org/tmp/dshelf2_inspect.txt > > > Sorry Marc. After offline communication with Qu, both > of us think the filesystem is hard to repair. > The filesystem is too large

Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread Su Yue
> Sent: Monday, July 02, 2018 at 5:43 PM > From: "Nikolay Borisov" > To: "Su Yue" , linux-btrfs@vger.kernel.org > Subject: Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair > > > > On 2.07.2018 12:28, Su Yue wrote: > > Since lowmem repair is dangerous, it should remind

Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread Su Yue
--- Su > Sent: Monday, July 02, 2018 at 7:09 PM > From: "Nikolay Borisov" > To: "David Disseldorp" > Cc: "Su Yue" , linux-btrfs@vger.kernel.org > Subject: Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair > > > > On 2.07.2018 14:07, David Disseldorp wrote: > > On Mon, 2

Re: [PATCH 4/4] btrfs: Remove unnecessary locking code in qgroup_rescan_leaf

2018-07-02 Thread Nikolay Borisov
On 27.06.2018 16:38, Nikolay Borisov wrote: > In qgroup_rescan_leaf a copy is made of the target leaf by calling > btrfs_clone_extent_buffer. The latter allocates a new buffer and > attaches a new set of pages and copies the content of the source > buffer. The new scratch buffer is only used to

Re: Major design flaw with BTRFS Raid, temporary device drop will corrupt nodatacow files

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-06-30 02:33, Duncan wrote: Austin S. Hemmelgarn posted on Fri, 29 Jun 2018 14:31:04 -0400 as excerpted: On 2018-06-29 13:58, james harvey wrote: On Fri, Jun 29, 2018 at 1:09 PM, Austin S. Hemmelgarn wrote: On 2018-06-29 11:15, james harvey wrote: On Thu, Jun 28, 2018 at 6:27 PM,

Re: unsolvable technical issues?

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-06-30 01:32, Andrei Borzenkov wrote: 30.06.2018 06:22, Duncan пишет: Austin S. Hemmelgarn posted on Mon, 25 Jun 2018 07:26:41 -0400 as excerpted: On 2018-06-24 16:22, Goffredo Baroncelli wrote: On 06/23/2018 07:11 AM, Duncan wrote: waxhead posted on Fri, 22 Jun 2018 01:13:31 +0200

Re: unsolvable technical issues?

2018-07-02 Thread Austin S. Hemmelgarn
On 2018-06-29 23:22, Duncan wrote: Austin S. Hemmelgarn posted on Mon, 25 Jun 2018 07:26:41 -0400 as excerpted: On 2018-06-24 16:22, Goffredo Baroncelli wrote: On 06/23/2018 07:11 AM, Duncan wrote: waxhead posted on Fri, 22 Jun 2018 01:13:31 +0200 as excerpted: According to this:

Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread David Disseldorp
On Mon, 2 Jul 2018 12:43:20 +0300, Nikolay Borisov wrote: > On 2.07.2018 12:28, Su Yue wrote: > > Since lowmem repair is dangerous, it should remind user more obviously. > > The patchset add 10 seconds delay like btrfs balance and add am option > > '--force-repair-lowmem' to skip the delay. >

Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 14:07, David Disseldorp wrote: > On Mon, 2 Jul 2018 12:43:20 +0300, Nikolay Borisov wrote: > >> On 2.07.2018 12:28, Su Yue wrote: >>> Since lowmem repair is dangerous, it should remind user more obviously. >>> The patchset add 10 seconds delay like btrfs balance and add am

Re: [PATCH] btrfs: qgroups: Move transaction managed inside btrfs_quota_enable

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 14:00, Nikolay Borisov wrote: > Commit 5d23515be669 ("btrfs: Move qgroup rescan on quota enable to > btrfs_quota_enable") not only resulted in an easier to follow code but > it also introduced a subtle bug. It changed the timing when the initial > transaction rescan was happening

[PATCH] btrfs: qgroups: Move transaction managed inside btrfs_quota_enable

2018-07-02 Thread Nikolay Borisov
Commit 5d23515be669 ("btrfs: Move qgroup rescan on quota enable to btrfs_quota_enable") not only resulted in an easier to follow code but it also introduced a subtle bug. It changed the timing when the initial transaction rescan was happening - before the commit it would happen after transaction

Re: [PATCH 1/4] btrfs: Refactor loop in btrfs_release_extent_buffer_page

2018-07-02 Thread Nikolay Borisov
On 29.06.2018 15:35, David Sterba wrote: > On Wed, Jun 27, 2018 at 04:38:22PM +0300, Nikolay Borisov wrote: >> The purpose of the function is to free all the pages comprising an >> extent buffer. This can be achieved with a simple for loop rather than >> the slitghly more involved 'do {} while'

Re: [PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 12:28, Su Yue wrote: > Since lowmem repair is dangerous, it should remind user more obviously. > The patchset add 10 seconds delay like btrfs balance and add am option > '--force-repair-lowmem' to skip the delay. IMO this is the wrong way to approach a dangerous option. If it's

[PATCH RFC 0/3] btrfs-progs: lowmem: delay before lowmem repair

2018-07-02 Thread Su Yue
Since lowmem repair is dangerous, it should remind user more obviously. The patchset add 10 seconds delay like btrfs balance and add am option '--force-repair-lowmem' to skip the delay. --- I don't whether it's a good idea to add delay and the option only for lowmem repair is acceptable, so make

[PATCH RFC 1/3] btrfs-progs: lowmem: delay before lowmem repair starts

2018-07-02 Thread Su Yue
Since lowmem mode repair is so dangerous, delay 10 seconds before start. Signed-off-by: Su Yue --- check/main.c | 28 ++-- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/check/main.c b/check/main.c index 3190b5d4f293..b9997460162f 100644 ---

[PATCH RFC 3/3] btrfs-progs: tests: append '--force-repair-lowmem' if lowmem repair is enabled

2018-07-02 Thread Su Yue
Since commit 62785de7c107 ("btrfs-progs: lowmem: force to start without delay with option '--force-repair-lowmem'") delays repair in lowmem mode. For tests which repair in lowmem mode, append '--force-repair-lowmem' to force them to start without delay. Signed-off-by: Su Yue ---

[PATCH RFC 2/3] btrfs-progs: lowmem: force to start without delay with option '--force-repair-lowmem'

2018-07-02 Thread Su Yue
Add an option '--force-repair-lowmem' to start check without any delay. Signed-off-by: Su Yue --- check/main.c | 17 +++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/check/main.c b/check/main.c index b9997460162f..3c9dc242f3db 100644 --- a/check/main.c +++

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 11:47, Qu Wenruo wrote: > > > On 2018年07月02日 16:37, Nikolay Borisov wrote: >> >> >> On 2.07.2018 11:34, Qu Wenruo wrote: >>> + if (hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) > 1) { >>> + block_group_err(fs_info, leaf, slot, >>> +"invalid profile flags, have

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 16:37, Nikolay Borisov wrote: > > > On 2.07.2018 11:34, Qu Wenruo wrote: >> +if (hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) > 1) { >> +block_group_err(fs_info, leaf, slot, >> +"invalid profile flags, have 0x%llx (%lu bits set) expect no more than 1 >>

Re: [PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 11:34, Qu Wenruo wrote: > + if (hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) > 1) { > + block_group_err(fs_info, leaf, slot, > +"invalid profile flags, have 0x%llx (%lu bits set) expect no more than 1 bit > set", > + flags &

Re: [PATCH] btrfs: Do extra chunk block group mapping check at mount

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 11:29, Qu Wenruo wrote: > Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199837, if a > crafted btrfs with incorrect chunk<->block group mapping, it could leads > to a lot of unexpected behavior. > > Although the crafted image can be catched by block group item checker >

[PATCH v2] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, a crafted image with invalid block group items could make free space cache code to cause panic. We could early detect such invalid block group item by checking: 1) Size (key) We have a up limit on block group item (10G) 2)

[PATCH] btrfs: Do extra chunk block group mapping check at mount

2018-07-02 Thread Qu Wenruo
Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199837, if a crafted btrfs with incorrect chunk<->block group mapping, it could leads to a lot of unexpected behavior. Although the crafted image can be catched by block group item checker added in "[PATCH] btrfs: tree-checker: Verify

Re: Enabling quota may not correctly rescan on 4.17

2018-07-02 Thread Misono Tomohiro
> Misono, > > Can you please try the attached patch? > I tried and it works (on 4.18.0-rc3). Committing transaction before starting rescan worker is what btrfs_qgroup_resan() does, so it looks fine. (though I'm not sure why you don't see the problem in your machine.) Reviewed-by: Misono

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 16:01, Nikolay Borisov wrote: > > > On 2.07.2018 10:53, Qu Wenruo wrote: >> >> >> On 2018年07月02日 15:31, Nikolay Borisov wrote: >>> >>> >>> On 2.07.2018 09:25, Qu Wenruo wrote: Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an invalid tree

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 11:01, Nikolay Borisov wrote: > > > On 2.07.2018 10:53, Qu Wenruo wrote: >> >> >> On 2018年07月02日 15:31, Nikolay Borisov wrote: >>> >>> >>> On 2.07.2018 09:25, Qu Wenruo wrote: Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an invalid tree reloc

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 10:53, Qu Wenruo wrote: > > > On 2018年07月02日 15:31, Nikolay Borisov wrote: >> >> >> On 2.07.2018 09:25, Qu Wenruo wrote: >>> Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an >>> invalid tree reloc tree can cause kernel NULL pointer dereference when >>>

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 15:31, Nikolay Borisov wrote: > > > On 2.07.2018 09:25, Qu Wenruo wrote: >> Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an >> invalid tree reloc tree can cause kernel NULL pointer dereference when >> btrfs does some cleanup for reloc roots. >> >> It

Re: [PATCH] btrfs: use correct compare function of dirty_metadata_bytes

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 10:44, Ethan Lien wrote: > We use customized, nodesize batch value to update dirty_metadata_bytes. > We should also use batch version of compare function or we will easily > goto fast path and get false result from percpu_counter_compare(). > > Signed-off-by: Ethan Lien

[PATCH] btrfs: use correct compare function of dirty_metadata_bytes

2018-07-02 Thread Ethan Lien
We use customized, nodesize batch value to update dirty_metadata_bytes. We should also use batch version of compare function or we will easily goto fast path and get false result from percpu_counter_compare(). Signed-off-by: Ethan Lien --- fs/btrfs/disk-io.c | 10 ++ 1 file changed, 6

Re: [PATCH] btrfs-progs: free-space-cache: Don't panic when free space cache is corrupted

2018-07-02 Thread Nikolay Borisov
On 1.07.2018 05:45, Qu Wenruo wrote: > In btrfs_add_free_space(), if the free space to be added is already > here, we trigger ASSERT() which is just another BUG_ON(). > > Let's remove such BUG_ON() at all. > > Reported-by: Lewis Diamond > Signed-off-by: Qu Wenruo Reviewed-by: Nikolay

Re: [PATCH] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Qu Wenruo
On 2018年07月02日 15:28, Nikolay Borisov wrote: > > > On 2.07.2018 08:53, Qu Wenruo wrote: >> As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, >> a crafted image with invalid block group items could make free space cache >> code to cause panic. >> >> We could early detect such

Re: [PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 09:25, Qu Wenruo wrote: > Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an > invalid tree reloc tree can cause kernel NULL pointer dereference when > btrfs does some cleanup for reloc roots. > > It turns out that fs_info->reloc_ctl can be NULL in >

Re: [PATCH] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread Nikolay Borisov
On 2.07.2018 08:53, Qu Wenruo wrote: > As reported in https://bugzilla.kernel.org/show_bug.cgi?id=199849, > a crafted image with invalid block group items could make free space cache > code to cause panic. > > We could early detect such invalid block group item by checking: > 1) Size (key) >

[PATCH] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized

2018-07-02 Thread Qu Wenruo
Reported in https://bugzilla.kernel.org/show_bug.cgi?id=199833, where an invalid tree reloc tree can cause kernel NULL pointer dereference when btrfs does some cleanup for reloc roots. It turns out that fs_info->reloc_ctl can be NULL in btrfs_recover_relocation() as we allocate relocation control

Re: [PATCH] btrfs: tree-checker: Verify block_group_item

2018-07-02 Thread kbuild test robot
/linux/commits/Qu-Wenruo/btrfs-tree-checker-Verify-block_group_item/20180702-135502 base: https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git next config: i386-randconfig-x016-201826 (attached as .config) compiler: gcc-7 (Debian 7.3.0-16) 7.3.0 reproduce: # save the attached

Re: So, does btrfs check lowmem take days? weeks?

2018-07-02 Thread Su Yue
On 07/02/2018 11:22 AM, Marc MERLIN wrote: On Mon, Jul 02, 2018 at 10:02:33AM +0800, Su Yue wrote: Could you try follow dumps? They shouldn't cost much time. #btrfs inspect dump-tree -t 21872 | grep -C 50 "374857 EXTENT_DATA " #btrfs inspect dump-tree -t 22911 | grep -C 50 "374857