[PATCH v2] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Qu Wenruo
Provide an offline tool to disable quota. For kernel which skip_balance doesn't work, there is no way to disable quota on huge fs with balance, as quota will cause balance to hang for a long long time for each tree block switch. So add an offline rescue tool to disable quota. Reported-by: Dan

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Qu Wenruo
On 2018/8/22 上午10:33, Su Yue wrote: > > >> Sent: Sunday, August 12, 2018 at 9:33 AM >> From: "Qu Wenruo" >> To: linux-btrfs@vger.kernel.org, dan.meril...@gmail.com >> Subject: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline >> >> Provide an offline tool to disable quota. >>

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Duncan
Austin S. Hemmelgarn posted on Tue, 21 Aug 2018 13:01:00 -0400 as excerpted: > Otherwise, the only option for people who want it set is to patch the > kernel to get noatime as the default (instead of relatime). I would > look at pushing such a patch upstream myself actually, if it weren't for >

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Su Yue
> Sent: Sunday, August 12, 2018 at 9:33 AM > From: "Qu Wenruo" > To: linux-btrfs@vger.kernel.org, dan.meril...@gmail.com > Subject: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline > > Provide an offline tool to disable quota. > > For kernel which skip_balance doesn't work,

Re: [PATCH 0/2] btrfs-progs: task: add prefn() to task_info and simplify task_start()

2018-08-21 Thread Su Yue
> Sent: Wednesday, August 22, 2018 at 3:38 AM > From: "Nikolay Borisov" > To: damenly...@gmail.com, linux-btrfs@vger.kernel.org > Cc: "Su Yue" > Subject: Re: [PATCH 0/2] btrfs-progs: task: add prefn() to task_info and > simplify task_start() > > > > On 21.08.2018 21:23,

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Liu Bo
On Tue, Aug 21, 2018 at 02:48:47PM -0400, Chris Mason wrote: > On 21 Aug 2018, at 14:15, Liu Bo wrote: > > > On Tue, Aug 21, 2018 at 01:54:11PM -0400, Chris Mason wrote: > > > On 16 Aug 2018, at 17:07, Liu Bo wrote: > > > > > > > The lock contention on btree nodes (esp. root node) is apparently

Re: [PATCH 2/2] Btrfs: kill leave_spinning

2018-08-21 Thread Liu Bo
Please kindly ignore this patch as it did introduce soft lockup when btrfs_search_slot() returns a spinning path and its callers goes to sleep before releasing the path. thanks, liubo On Thu, Aug 16, 2018 at 2:07 PM, Liu Bo wrote: > As btrfs_clear_path_blocking() turns out to be a major source

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Liu Bo
On Tue, Aug 21, 2018 at 08:35:30PM +0200, Holger Hoffstätte wrote: > On 08/21/18 20:15, Liu Bo wrote: > > I just realize that patch 2 can result in softlockup as > > btrfs_search_slot() may return a path with all nodes being in spinning > > lock, and if the callers want to sleep, we're in trouble.

[PATCH v2] Btrfs: kill btrfs_clear_path_blocking

2018-08-21 Thread Liu Bo
Btrfs's btree locking has two modes, spinning mode and blocking mode, while searching btree, locking is always acquired in spinning mode and then converted to blocking mode if necessary, and in some hot paths we may switch the locking back to spinning mode by btrfs_clear_path_blocking(). When

Re: [PATCH 0/2] btrfs-progs: task: add prefn() to task_info and simplify task_start()

2018-08-21 Thread Nikolay Borisov
On 21.08.2018 21:23, damenly...@gmail.com wrote: > From: Su Yue > > This patchset is a prepare work for further process of check. > > patch[1] adds prefn() to task_info and call it before pthread_create(). > > patch[2] moves time and count initializations into the new init_ctx(), > assign

exporting uuid_t from uapi/linux/uuid.h

2018-08-21 Thread Nikolay Borisov
Hello, I'm doing some refactoring on btrfs and would like to convert some of the "naked" definition of UUID in on-disk object to uuid_t. Since those structs are defined in include/uapi/linux/btrfs_tree.h I naturally would need to expose the kernel uuid_t to the uapi header via

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Chris Mason
On 21 Aug 2018, at 14:15, Liu Bo wrote: On Tue, Aug 21, 2018 at 01:54:11PM -0400, Chris Mason wrote: On 16 Aug 2018, at 17:07, Liu Bo wrote: The lock contention on btree nodes (esp. root node) is apparently a bottleneck when there're multiple readers and writers concurrently trying to access

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Holger Hoffstätte
On 08/21/18 20:15, Liu Bo wrote: I just realize that patch 2 can result in softlockup as btrfs_search_slot() may return a path with all nodes being in spinning lock, and if the callers want to sleep, we're in trouble. I've removed patch 2 and am re-running the test (xfstests, fsmark and

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Liu Bo
On Tue, Aug 21, 2018 at 01:54:11PM -0400, Chris Mason wrote: > On 16 Aug 2018, at 17:07, Liu Bo wrote: > > > The lock contention on btree nodes (esp. root node) is apparently a > > bottleneck when there're multiple readers and writers concurrently > > trying to access them. Unfortunately this is

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Chris Mason
On 16 Aug 2018, at 17:07, Liu Bo wrote: The lock contention on btree nodes (esp. root node) is apparently a bottleneck when there're multiple readers and writers concurrently trying to access them. Unfortunately this is by design and it's not easy to fix it unless with some complex changes,

Re: [PATCH 1/2] Btrfs: kill btrfs_clear_path_blocking

2018-08-21 Thread Liu Bo
On Mon, Aug 20, 2018 at 10:31:49AM +0300, Nikolay Borisov wrote: > > > On 20.08.2018 09:07, Liu Bo wrote: > > On Fri, Aug 17, 2018 at 10:24:58AM +0300, Nikolay Borisov wrote: > >> > >> > >> On 17.08.2018 00:07, Liu Bo wrote: > >>> Btrfs's btree locking has two modes, spinning mode and blocking

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Austin S. Hemmelgarn
On 2018-08-21 12:05, David Sterba wrote: On Tue, Aug 21, 2018 at 10:10:04AM -0400, Austin S. Hemmelgarn wrote: On 2018-08-21 09:32, Janos Toth F. wrote: so pretty much everyone who wants to avoid the overhead from them can just use the `noatime` mount option. It would be great if someone

Re: Do btrfs compression option changes need to be atomic?

2018-08-21 Thread David Howells
David Sterba wrote: > The ordering of the bits set/cleared can prevent unwanted combinations, > like: when the COMPRESS bit is set, there's no NODATACOW anymore: > > - clear NODATACOW > - set compress_type > - set COMPRESS > > I haven't checked with the sources if this is so. The current code

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread David Sterba
On Tue, Aug 21, 2018 at 10:10:04AM -0400, Austin S. Hemmelgarn wrote: > On 2018-08-21 09:32, Janos Toth F. wrote: > so pretty much everyone who wants to avoid the overhead from them can > just > use the `noatime` mount option. > > > > It would be great if someone finally fixed

Re: [PATCH] generic: test for deduplication between different files

2018-08-21 Thread Filipe Manana
On Mon, Aug 20, 2018 at 12:11 AM, Dave Chinner wrote: > [cc linux-...@vger.kernel.org] > > On Fri, Aug 17, 2018 at 09:39:24AM +0100, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Test that deduplication of an entire file that has a size that is not >> aligned to the filesystem's block

Re: [patch] file dedupe (and maybe clone) data corruption (was Re: [PATCH] generic: test for deduplication between different files)

2018-08-21 Thread Filipe Manana
On Mon, Aug 20, 2018 at 2:09 AM, Dave Chinner wrote: > [cc linux-fsdevel now, too] > > On Mon, Aug 20, 2018 at 09:11:26AM +1000, Dave Chinner wrote: >> [cc linux-...@vger.kernel.org] >> >> On Fri, Aug 17, 2018 at 09:39:24AM +0100, fdman...@kernel.org wrote: >> > From: Filipe Manana >> > >> >

Re: Do btrfs compression option changes need to be atomic?

2018-08-21 Thread David Sterba
On Tue, Aug 21, 2018 at 03:34:06PM +0100, David Howells wrote: > David Sterba wrote: > > > Do you mean the compression type (btrfs_fs_info::compress_type) and the > > related bits in btrfs_fs_info::mount_opt ? There are more compression > > types but not used in the mount context. I assume

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread Qu Wenruo
On 2018/8/21 下午10:35, David Howells wrote: > David Sterba wrote: > >> - nothing: auto-detect non-rotating devices, enable SSD mount option in turn > > And it's right that this should only turn on SSD if *all* the devices are > non-rotating? If I didn't miss anything again, yes. It's

Re: Do btrfs compression option changes need to be atomic?

2018-08-21 Thread David Howells
David Sterba wrote: > Do you mean the compression type (btrfs_fs_info::compress_type) and the > related bits in btrfs_fs_info::mount_opt ? There are more compression > types but not used in the mount context. I assume you're interested > only in the mount-time settings, otherwise the defrag and

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread David Howells
David Sterba wrote: > - nothing: auto-detect non-rotating devices, enable SSD mount option in turn And it's right that this should only turn on SSD if *all* the devices are non-rotating? David

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread Qu Wenruo
On 2018/8/21 下午10:24, David Sterba wrote: > On Tue, Aug 21, 2018 at 02:43:35PM +0100, David Howells wrote: >> Qu Wenruo wrote: >> >>> But to be more clear, NOSSD shouldn't be a special case. >>> In fact currently NOSSD only affects whether we will output the message >>> "enabling ssd

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread David Sterba
On Tue, Aug 21, 2018 at 02:43:35PM +0100, David Howells wrote: > Qu Wenruo wrote: > > > But to be more clear, NOSSD shouldn't be a special case. > > In fact currently NOSSD only affects whether we will output the message > > "enabling ssd optimization", no real effect if I didn't miss anything.

Re: Do btrfs compression option changes need to be atomic?

2018-08-21 Thread David Sterba
On Tue, Aug 21, 2018 at 02:46:30PM +0100, David Howells wrote: > Should changes to the compression options on a btrfs mount be atomic, the > problem being that they're split across three variables? Do you mean the compression type (btrfs_fs_info::compress_type) and the related bits in

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread Austin S. Hemmelgarn
On 2018-08-21 09:43, David Howells wrote: Qu Wenruo wrote: But to be more clear, NOSSD shouldn't be a special case. In fact currently NOSSD only affects whether we will output the message "enabling ssd optimization", no real effect if I didn't miss anything. That's not quite true. In:

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Austin S. Hemmelgarn
On 2018-08-21 09:32, Janos Toth F. wrote: so pretty much everyone who wants to avoid the overhead from them can just use the `noatime` mount option. It would be great if someone finally fixed this old bug then: https://bugzilla.kernel.org/show_bug.cgi?id=61601 Until then, it seems practically

Re: Do btrfs compression option changes need to be atomic?

2018-08-21 Thread Chris Mason
On 21 Aug 2018, at 9:46, David Howells wrote: Should changes to the compression options on a btrfs mount be atomic, the problem being that they're split across three variables? Further to that, how much of an issue is it if the configuration is split out into its own struct that is accessed

Re: [PATCH 15/22] btrfs: run delayed iputs before committing

2018-08-21 Thread David Sterba
On Thu, Jul 19, 2018 at 10:49:59AM -0400, Josef Bacik wrote: > We want to have a complete picture of any delayed inode updates before > we make the decision to commit or not, so make sure we run delayed iputs > before making the decision to commit or not. This basically says "we need to do X, so

Do btrfs compression option changes need to be atomic?

2018-08-21 Thread David Howells
Should changes to the compression options on a btrfs mount be atomic, the problem being that they're split across three variables? Further to that, how much of an issue is it if the configuration is split out into its own struct that is accessed from struct btrfs_fs_info using RCU? David

Re: Are the btrfs mount options inconsistent?

2018-08-21 Thread David Howells
Qu Wenruo wrote: > But to be more clear, NOSSD shouldn't be a special case. > In fact currently NOSSD only affects whether we will output the message > "enabling ssd optimization", no real effect if I didn't miss anything. That's not quite true. In: if (!btrfs_test_opt(fs_info, NOSSD)

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Janos Toth F.
> >> so pretty much everyone who wants to avoid the overhead from them can just > >> use the `noatime` mount option. It would be great if someone finally fixed this old bug then: https://bugzilla.kernel.org/show_bug.cgi?id=61601 Until then, it seems practically impossible to use both noatime

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Austin S. Hemmelgarn
On 2018-08-21 08:06, Adam Borowski wrote: On Mon, Aug 20, 2018 at 08:16:16AM -0400, Austin S. Hemmelgarn wrote: Also, slightly OT, but atimes are not where the real benefit is here for most people. No sane software other than mutt uses atimes (and mutt's use of them is not sane, but that's a

Re: lazytime mount option—no support in Btrfs

2018-08-21 Thread Adam Borowski
On Mon, Aug 20, 2018 at 08:16:16AM -0400, Austin S. Hemmelgarn wrote: > Also, slightly OT, but atimes are not where the real benefit is here for > most people. No sane software other than mutt uses atimes (and mutt's use > of them is not sane, but that's a different argument) Right. There are

[PATCH 2/2] btrfs-progs: task: add init_ctx and clean up arguments of task_start()

2018-08-21 Thread damenly . su
From: Su Yue Add a function init_ctx() to initialize time and count of ctx while checking. Use it as prefn while calling task_init(). Arguments start_time and item_count of task_start() are useless, remove them. Signed-off-by: Su Yue --- check/main.c | 26 ++

[PATCH 1/2] btrfs-progs: task: add prefn() into struct task_info

2018-08-21 Thread damenly . su
From: Su Yue Previously, everytime task_start is called, arguments to be initialized (start_time and item_count) are passed manually. For scalability, add a member prefn() to struct task_info. prefn() will be called before pthread_create() in task_start(). Signed-off-by: Su Yue ---

[PATCH 0/2] btrfs-progs: task: add prefn() to task_info and simplify task_start()

2018-08-21 Thread damenly . su
From: Su Yue This patchset is a prepare work for further process of check. patch[1] adds prefn() to task_info and call it before pthread_create(). patch[2] moves time and count initializations into the new init_ctx(), assign it to prefun(). Remove arguments start_time and item_count. Su Yue

Re: [RFC PATCH v4 4/6] Btrfs: prevent ioctls from interfering with a swap file

2018-08-21 Thread Omar Sandoval
On Fri, May 25, 2018 at 06:10:43PM +0200, David Sterba wrote: > On Fri, May 25, 2018 at 09:00:58AM -0700, Omar Sandoval wrote: > > On Fri, May 25, 2018 at 04:50:55PM +0200, David Sterba wrote: > > > On Thu, May 24, 2018 at 02:41:28PM -0700, Omar Sandoval wrote: > > > > From: Omar Sandoval > > > >

[PATCH v2 3/4] btrfs: Refactor unclustered extent allocation into find_free_extent_unclustered()

2018-08-21 Thread Qu Wenruo
This patch will extract unclsutered extent allocation code into find_free_extent_unclustered(). And this helper function will use return value to indicate what to do next. This should make find_free_extent() a little easier to read. Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 112

[PATCH v2 0/4] btrfs: Refactor find_free_extent()

2018-08-21 Thread Qu Wenruo
Can be fetched from github: https://github.com/adam900710/linux/tree/refactor_find_free_extent Which is based on v4.18-rc5, with one extra cleanup commit b0f2261e077b ("btrfs: extent-tree: Remove dead alignment check"). extent-tree.c::find_free_extent() could be one of the most ill-structured

[PATCH v2 4/4] btrfs: Refactor find_free_extent() loops update into find_free_extent_update_loop()

2018-08-21 Thread Qu Wenruo
We have a complex loop design for find_free_extent(), that has different behavior for each loop, some even includes new chunk allocation. Instead of putting such a long code into find_free_extent() and makes it harder to read, just extract them into find_free_extent_update_loop(). With all the

[PATCH v2 1/4] btrfs: Introduce find_free_extent_ctrl structure for later rework

2018-08-21 Thread Qu Wenruo
Instead of tons of different local variables in find_free_extent(), extract them into find_free_extent_ctrl structure, and add better explanation for them. Some modification may looks redundant, but will later greatly simplify function parameter list during find_free_extent() refactor.

[PATCH v2 2/4] btrfs: Refactor clustered extent allocation into find_free_extent_clustered()

2018-08-21 Thread Qu Wenruo
We have two main methods to find free extents inside a block group: 1) clustered allocation 2) unclustered allocation This patch will extract the clustered allocation into find_free_extent_clustered() to make it a little easier to read. Instead of jumping between different labels in

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Qu Wenruo
On 2018/8/21 下午3:14, Misono Tomohiro wrote: > On 2018/08/12 10:33, Qu Wenruo wrote: >> Provide an offline tool to disable quota. >> >> For kernel which skip_balance doesn't work, there is no way to disable >> quota on huge fs with balance, as quota will cause balance to hang for a >> long long

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Misono Tomohiro
On 2018/08/12 10:33, Qu Wenruo wrote: > Provide an offline tool to disable quota. > > For kernel which skip_balance doesn't work, there is no way to disable > quota on huge fs with balance, as quota will cause balance to hang for a > long long time for each tree block switch. > > So add an

Re: [PATCH v2 00/20] btrfs-progs: Rework of "subvolume list/show" and relax the root privileges of them

2018-08-21 Thread Misono Tomohiro
On 2018/08/16 3:12, David Sterba wrote: > Hi, > > so I've merged the first bath to devel. There was quite some whitespace > damage and style issues, but I fixed that as I went through the pathes. Thanks for taking this. > > I've left out all patches that introduce new options, this is the hard