Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 15:20, Nikolay Borisov wrote: > > > On 13.02.2018 03:13, Qu Wenruo wrote: >> There are reports in mail list, even with latest mainline kernel, btrfs >> can't survive a power loss. >> >> Unlike journal based filesystem, btrfs doesn't use journal for such >> work. (log tree is an o

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 11:00, Anand Jain wrote: > Fixes the endianness bug in the fs_info::super_copy by using its > btrfs_set_super...() function to set values in the SB, as these > functions manage the endianness compatibility nicely. > > Signed-off-by: Anand Jain Also went through all btrfs_super_b

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 11:53, Anand Jain wrote: > For forensic investigations of issues, we would want to know > if and when the user cancels the replace. > > Signed-off-by: Anand Jain > --- > v1->v2: use btrfs_dev_name() instead of rcu_str_deref() > as btrfs_dev_name() also provides "missing" s

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

2018-02-13 Thread Nikolay Borisov
On 13.02.2018 11:05, Qu Wenruo wrote: > > > On 2018年02月13日 11:53, Anand Jain wrote: >> For forensic investigations of issues, we would want to know >> if and when the user cancels the replace. >> >> Signed-off-by: Anand Jain > --- >> v1->v2: use btrfs_dev_name() instead of rcu_str_deref() >>

Re: [PATCH] btrfs: verify max_inline mount parameter

2018-02-13 Thread Anand Jain
On 02/13/2018 01:13 AM, David Sterba wrote: On Mon, Feb 12, 2018 at 11:35:46PM +0800, Anand Jain wrote: We aren't verifying the parameter passed to the max_inline mount option, so we won't report and fail the mount if a junk value is specified for example, -o max_inline=abc. This patch convert

[PATCH v2 2/2] btrfs: verify max_inline mount parameter

2018-02-13 Thread Anand Jain
We aren't verifying the parameter passed to the max_inline mount option, so we won't report and fail the mount if a junk value is specified for example, -o max_inline=abc. This patch converts the max_inline option to %d and checks if it's a number >= 0. Signed-off-by: Anand Jain --- v1->v2: use m

[PATCH v2 1/2] btrfs: declare max_inline as u32

2018-02-13 Thread Anand Jain
As of now btrfs_fs_info::max_line is u64, which can't be larger than btrfs_fs_info::sectorsize which is defined as u32, so make btrfs_fs_info::max_line u32, Signed-off-by: Anand Jain --- v1->v2: Born in v2. fs/btrfs/ctree.h | 2 +- fs/btrfs/super.c | 8 2 files changed, 5 insertions(+),

[PATCH] btrfs: manage metadata_ratio mount option as %u

2018-02-13 Thread Anand Jain
As -o metadata_ratio mount option is unsinged so manage it as %u for token verifications, instead of %s. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index bf629c8a6a47..b

[PATCH] btrfs: fix bare unsigned declarations

2018-02-13 Thread Anand Jain
Kernel style prefers "unsigned int " over "unsigned " and "signed int " over "signed ". Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 80d1d4d12f9d..c6de1a5281ca 100644 --- a/fs/btrf

[PATCH] btrfs: add a comment to mark the deprecated mount option

2018-02-13 Thread Anand Jain
-o alloc_start and -o subvolrootid are deprecated mount options, comment them in the tokens list. And leave them as it is. No functional changes. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/sup

[PATCH] btrfs: manage thread_pool mount option as %u

2018-02-13 Thread Anand Jain
-o thread_pool is alway unsigned. Manage it that way all around. Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 2 +- fs/btrfs/disk-io.c | 4 ++-- fs/btrfs/super.c | 13 ++--- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h inde

[PATCH] btrfs: manage check_int_print_mask mount option as %u

2018-02-13 Thread Anand Jain
As -o check_int_print_mask mount option is unsinged so manage it as %u for token verifications, instead of %d. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 16 +--- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index b925a649a0

[PATCH] btrfs: manage commit mount option as %u

2018-02-13 Thread Anand Jain
As -o commit mount option is unsinged so manage it as %u for token verifications, instead of %d. Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 2 +- fs/btrfs/super.c | 26 ++ 2 files changed, 11 insertions(+), 17 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/

[PATCH] btrfs: manage subvolid mount option as %u

2018-02-13 Thread Anand Jain
As -o subvolid mount option is an u64 manage it as %u for token verifications, instead of %s. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 23 ++- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index 8112619cac95..bf629

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 17:08, Nikolay Borisov wrote: > > > On 13.02.2018 11:05, Qu Wenruo wrote: >> >> >> On 2018年02月13日 11:53, Anand Jain wrote: >>> For forensic investigations of issues, we would want to know >>> if and when the user cancels the replace. >>> >>> Signed-off-by: Anand Jain > --- >>> v1

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread John Ettedgui
On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: > > > No more. > > The dump is already good enough for me to dig for some time. > > We don't usually get such large extent tree dump from a real world use case. > > It would help us in several ways, from determine how fragmented a block > group is

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

2018-02-13 Thread Anand Jain
On 02/13/2018 05:01 PM, Qu Wenruo wrote: On 2018年02月13日 11:00, Anand Jain wrote: Fixes the endianness bug in the fs_info::super_copy by using its btrfs_set_super...() function to set values in the SB, as these functions manage the endianness compatibility nicely. Signed-off-by: Anand Jain

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 18:27, Anand Jain wrote: > > > On 02/13/2018 05:01 PM, Qu Wenruo wrote: >> >> >> On 2018年02月13日 11:00, Anand Jain wrote: >>> Fixes the endianness bug in the fs_info::super_copy by using its >>> btrfs_set_super...() function to set values in the SB, as these >>> functions manage t

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 18:21, John Ettedgui wrote: > On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: >> >> >> No more. >> >> The dump is already good enough for me to dig for some time. >> >> We don't usually get such large extent tree dump from a real world use case. >> >> It would help us in several

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 09:13, Qu Wenruo wrote: > There are reports in mail list, even with latest mainline kernel, btrfs > can't survive a power loss. > > Unlike journal based filesystem, btrfs doesn't use journal for such > work. (log tree is an optimization for fsync, not to keep fs healthy) > In btrf

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread John Ettedgui
On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: > > > On 2018年02月13日 18:21, John Ettedgui wrote: >> Hello there, >> >> have you found anything good since then? > > Unfortunately, not really much to speed it up. Oh :/ > > This reminds me of the old (and crazy) idea to skip block group build > for

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 19:25, John Ettedgui wrote: > On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >> >> >> On 2018年02月13日 18:21, John Ettedgui wrote: >>> Hello there, >>> >>> have you found anything good since then? >> >> Unfortunately, not really much to speed it up. > Oh :/ >> >> This reminds me

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread John Ettedgui
On Tue, Feb 13, 2018 at 3:40 AM, Qu Wenruo wrote: > > > On 2018年02月13日 19:25, John Ettedgui wrote: >> On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >>> >>> >>> >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. >>> >>> For new fs filled

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Holger Hoffstätte
On 02/13/18 12:40, Qu Wenruo wrote: >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. I have no idea why btrfs' mount even needs to touch all block groups to get going (which seems to be the root of the problem), but here's a not so crazy id

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 20:06, John Ettedgui wrote: >>> That's fairly straightforward to do, though it should be quite slow so >>> I'd hope not to have to do that too often. >> >> Then it could be tried on the most frequently updated files then. > > That's an interesting idea. > More than 3/4 of the

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread John Ettedgui
On Tue, Feb 13, 2018 at 4:46 AM, Qu Wenruo wrote: > > > On 2018年02月13日 20:06, John Ettedgui wrote: > That's fairly straightforward to do, though it should be quite slow so I'd hope not to have to do that too often. >>> >>> Then it could be tried on the most frequently updated files t

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Qu Wenruo
On 2018年02月13日 20:26, Holger Hoffstätte wrote: > On 02/13/18 12:40, Qu Wenruo wrote: The problem is not about how much space it takes, but how many extents are here in the filesystem. > > I have no idea why btrfs' mount even needs to touch all block groups to > get going (which seems t

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

2018-02-13 Thread Nikolay Borisov
On 13.02.2018 13:07, Qu Wenruo wrote: > > > On 2018年02月13日 09:13, Qu Wenruo wrote: >> There are reports in mail list, even with latest mainline kernel, btrfs >> can't survive a power loss. >> >> Unlike journal based filesystem, btrfs doesn't use journal for such >> work. (log tree is an optimiz

btrfs-progs: compile with musl libc

2018-02-13 Thread Dan Robertson
Ensure that limits.h is included in mkfs/main.c as PATH_MAX is used. The lack of this currently causes btrfs-progs to fail to compile when the systems libc is musl. From 7859e0d01778e844ae0fcbefe55581277ce7cab3 Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Tue, 13 Feb 2018 13:04:29 + Subj

Re: btrfs-cleaner / snapshot performance analysis

2018-02-13 Thread E V
On Mon, Feb 12, 2018 at 10:37 AM, Ellis H. Wilson III wrote: > On 02/11/2018 01:24 PM, Hans van Kranenburg wrote: >> >> Why not just use `btrfs fi du ` now and then and >> update your administration with the results? .. Instead of putting the >> burden of keeping track of all administration duri

[PATCH 2/2] btrfs: Refactor running of delayed inode items during transaction commit

2018-02-13 Thread Nikolay Borisov
Currently calls to btrfs_run_delayed_inode items have been scattered around the transaction commit code with no real design argument when they should be execute. We have one call, after transaction writers go to 0. Then we have the delayed items running as part of creating a snapshot (once per pe

[PATCH 1/2] btrfs: Remove extra run_delayed_items call

2018-02-13 Thread Nikolay Borisov
Before commit 581227d0d2b8 ("Btrfs: remove the time check in btrfs_commit_transaction()") there would be a loop in the transaction commit path that will go on until all existing writers ended their transaction handles. This loop would try and flush as much pending stuff as possible before going int

Re: [PATCH] btrfs: manage thread_pool mount option as %u

2018-02-13 Thread Nikolay Borisov
On 13.02.2018 11:50, Anand Jain wrote: > -o thread_pool is alway unsigned. Manage it that way all around. > > Signed-off-by: Anand Jain > --- > fs/btrfs/ctree.h | 2 +- > fs/btrfs/disk-io.c | 4 ++-- > fs/btrfs/super.c | 13 ++--- > 3 files changed, 9 insertions(+), 10 deletions(

Re: btrfs-progs: compile with musl libc

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 01:25:10PM +, Dan Robertson wrote: > Ensure that limits.h is included in mkfs/main.c as PATH_MAX is used. The lack > of this currently causes btrfs-progs to fail to compile when the systems libc > is musl. > From 7859e0d01778e844ae0fcbefe55581277ce7cab3 Mon Sep 17 00:00

Re: [PATCH] btrfs: manage thread_pool mount option as %u

2018-02-13 Thread Anand Jain
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index 02c7766e6849..8112619cac95 100644 --- a/fs/btrfs/super.c +++ b/fs/btrfs/super.c @@ -346,7 +346,7 @@ static const match_table_t tokens = { {Opt_barrier, "barrier"}, {Opt_max_inline, "max_inline=%u"}, {Opt_alloc_st

Re: btrfs-progs: compile with musl libc

2018-02-13 Thread Dan Robertson
> Thanks, this has been fixed in > https://github.com/kdave/btrfs-progs/commit/3aa1bbdd89ee9c9c48d260a6192fae08328f1b2f Ah, my bad. Didn't check the dev branch. Thanks for the quick response! Cheers, Dan signature.asc Description: Digital signature

Re: [PATCH] btrfs: manage thread_pool mount option as %u

2018-02-13 Thread Nikolay Borisov
On 13.02.2018 17:18, Anand Jain wrote: > > > > > > >>> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c >>> index 02c7766e6849..8112619cac95 100644 >>> --- a/fs/btrfs/super.c >>> +++ b/fs/btrfs/super.c >>> @@ -346,7 +346,7 @@ static const match_table_t tokens = { >>>   {Opt_barrier, "ba

Re: [PATCH] btrfs: manage thread_pool mount option as %u

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 05:34:56PM +0200, Nikolay Borisov wrote: > >>> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c > >>> index 02c7766e6849..8112619cac95 100644 > >>> --- a/fs/btrfs/super.c > >>> +++ b/fs/btrfs/super.c > >>> @@ -346,7 +346,7 @@ static const match_table_t tokens = { > >>>  

Re: [PATCH] btrfs: manage subvolid mount option as %u

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 05:50:43PM +0800, Anand Jain wrote: > As -o subvolid mount option is an u64 manage it as %u for > token verifications, instead of %s. > > Signed-off-by: Anand Jain > --- > fs/btrfs/super.c | 23 ++- > 1 file changed, 10 insertions(+), 13 deletions(-) >

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Holger Hoffstätte
On 02/13/18 13:54, Qu Wenruo wrote: > On 2018年02月13日 20:26, Holger Hoffstätte wrote: >> On 02/13/18 12:40, Qu Wenruo wrote: > The problem is not about how much space it takes, but how many extents > are here in the filesystem. >> >> I have no idea why btrfs' mount even needs to touch all bl

Re: [PATCH v2 2/2] btrfs: verify max_inline mount parameter

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 05:49:50PM +0800, Anand Jain wrote: > We aren't verifying the parameter passed to the max_inline mount option, > so we won't report and fail the mount if a junk value is specified for > example, -o max_inline=abc. > This patch converts the max_inline option to %d and checks

Re: [PATCH v2 1/2] btrfs: declare max_inline as u32

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 05:49:49PM +0800, Anand Jain wrote: > As of now btrfs_fs_info::max_line is u64, which can't be > larger than btrfs_fs_info::sectorsize which is defined as > u32, so make btrfs_fs_info::max_line u32, > > Signed-off-by: Anand Jain Reviewed-by: David Sterba -- To unsubscrib

Re: [PATCH] btrfs: extent_buffer_uptodate() make it static and inline

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 09:18:51AM +0200, Nikolay Borisov wrote: > > > On 13.02.2018 06:35, Anand Jain wrote: > > extent_buffer_uptodate() is a trivial wrapper around test_bit() > > and nothing else. So make it static and inline, save on code > > space and call indirection. > > > > Before: > >

Re: [PATCH] btrfs: fix bare unsigned declarations

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 05:50:48PM +0800, Anand Jain wrote: > Kernel style prefers "unsigned int " over "unsigned " > and "signed int " over "signed ". The changelog does not match the changes, you're switching to u32. I agree u32 looks more suitable and consistent with the other changes. Please u

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 11:53:43AM +0800, Anand Jain wrote: > For forensic investigations of issues, we would want to know > if and when the user cancels the replace. > > Signed-off-by: Anand Jain Reviewed-by: David Sterba I've updated the changelog a bit. -- To unsubscribe from this list: sen

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

2018-02-13 Thread David Sterba
On Tue, Feb 13, 2018 at 06:27:13PM +0800, Anand Jain wrote: > On 02/13/2018 05:01 PM, Qu Wenruo wrote: > > On 2018年02月13日 11:00, Anand Jain wrote: > >> Fixes the endianness bug in the fs_info::super_copy by using its > >> btrfs_set_super...() function to set values in the SB, as these > >> function

Re: [PATCH 1/2] btrfs: Remove invalid null checks from btrfs_cleanup_dirty_bgs

2018-02-13 Thread David Sterba
On Thu, Feb 08, 2018 at 10:57:11AM -0800, Liu Bo wrote: > On Thu, Feb 08, 2018 at 06:25:17PM +0200, Nikolay Borisov wrote: > > list_first_entry is essentially a wrapper over cotnainer_of. The latter > > can never return null even if it's working on inconsistent list since it > > will either crash o

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Qu Wenruo
On 2018年02月14日 00:24, Holger Hoffstätte wrote: > On 02/13/18 13:54, Qu Wenruo wrote: >> On 2018年02月13日 20:26, Holger Hoffstätte wrote: >>> On 02/13/18 12:40, Qu Wenruo wrote: >> The problem is not about how much space it takes, but how many extents >> are here in the filesystem. >>> >>> I

Re: btrfs-cleaner / snapshot performance analysis

2018-02-13 Thread Darrick J. Wong
On Sun, Feb 11, 2018 at 02:40:16PM +0800, Qu Wenruo wrote: > > > On 2018年02月10日 00:45, Ellis H. Wilson III wrote: > > Hi all, > > > > I am trying to better understand how the cleaner kthread (btrfs-cleaner) > > impacts foreground performance, specifically during snapshot deletion. > > My experie

[PATCH] btrfs-progs: Fix read beyond boundary bug in build_roots_info_cache()

2018-02-13 Thread Qu Wenruo
This bug is exposed by fsck-test with D=asan, hit by test case 020, with the following error report: = ==10740==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62161580 at pc 0x56051f0db6cd bp 0x7ffe170f3e20 sp 0x7ffe1

[PATCH] btrfs-progs: mkfs/rootdir: Fix memory leak in traverse_directory()

2018-02-13 Thread Qu Wenruo
The bug is exposed by mkfs test case 009, with D=asan. We are leaking memory of parent_dir_entry->path() which ,except the rootdir, is allocated by strdup(). Before fixing it, unifiy the allocation of parent_dir_entry() to heap allocation. Then fix it by adding "free(parent_dir_entry->path);" in