[PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

2017-04-21 Thread Sargun Dhillon
This patch allows processes with CAP_SYS_RESOURCE to exceed the qgroup limit. It's useful for administrative programs, such as log rotation, that may need to temporarily use more disk space in order to free up a greater amount of overall disk space without yielding more disk space to the rest of us

Experiences with metadata balance/convert

2017-04-21 Thread Hans van Kranenburg
This is a followup to my previous post "About free space fragmentation, metadata write amplification and (no)ssd", exploring how good or bad btrfs can handle filesystem that are larger than your average desktop computer. One of the things I'm looking at to do is to convert the metadata of a large

Re: Experiences with metadata balance/convert

2017-04-21 Thread Hans van Kranenburg
Doh, On 04/21/2017 12:26 PM, Hans van Kranenburg wrote: > [...] > > == Thinking out of the box == > > Technically, converting from DUP to single could also mean: > * Flipping one bit in the block group type flags to 0 for each block > group item > * Flipping one bit in the chunk type flags and r

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

2017-04-21 Thread Sargun Dhillon
The log rotation code that I have requires creating the new file before it calls FICLONERANGE. Since it tries to copy close to a newline and not the filesystem block boundary, it means that some of the clone isn't a lazy copy, and instead has to be cloned byte for byte. At a minimum I need a mechan

RE: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Reshetova, Elena
> Hi Elena, > > On Thu, Apr 20, 2017 at 04:10:16PM +, Reshetova, Elena wrote: > > > > > All the objections from DaveM on the amount of cycles spent on the > > > new refcount_t apply to the block layer fast path operations as well. > > > > Ok, could you please indicate the correct way to measur

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Christoph Hellwig
On Thu, Apr 20, 2017 at 11:33:19AM -0700, Eric Biggers wrote: > Like I suggested months ago, how about doing an efficient implementation of > refcount_t which doesn't use the bloated cmpxchg loop? Then there would be no > need for endless performance arguments. In fact, in PaX there are already >

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

2017-04-21 Thread Adam Borowski
On Fri, Apr 21, 2017 at 10:09:46AM +, Sargun Dhillon wrote: > This patch allows processes with CAP_SYS_RESOURCE to exceed the qgroup > limit. It's useful for administrative programs, such as log rotation, > that may need to temporarily use more disk space in order to free up > a greater amount

Re: Experiences with metadata balance/convert

2017-04-21 Thread Hans van Kranenburg
On 04/21/2017 12:31 PM, Hans van Kranenburg wrote: > Doh, > > On 04/21/2017 12:26 PM, Hans van Kranenburg wrote: >> [...] >> >> == Thinking out of the box == >> >> Technically, converting from DUP to single could also mean: >> * Flipping one bit in the block group type flags to 0 for each block >>

Re: Experiences with metadata balance/convert

2017-04-21 Thread Austin S. Hemmelgarn
On 2017-04-21 07:13, Hans van Kranenburg wrote: On 04/21/2017 12:31 PM, Hans van Kranenburg wrote: Doh, On 04/21/2017 12:26 PM, Hans van Kranenburg wrote: [...] == Thinking out of the box == Technically, converting from DUP to single could also mean: * Flipping one bit in the block group typ

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

2017-04-21 Thread Sargun Dhillon
What do you think about putting this behaviour behind a sysctl? Seems better than to start introducing a new mechanism of marking tasks? On Fri, Apr 21, 2017 at 6:05 AM, Adam Borowski wrote: > On Fri, Apr 21, 2017 at 10:09:46AM +, Sargun Dhillon wrote: >> This patch allows processes with CAP_

Prevent escaping btrfs quota

2017-04-21 Thread Christian Brauner
Hi guys, If a qgroup is created for a btrfs subvolume /some/path and limits are set and a new btrfs subvolume /some/path/bla is created it does not inherit the parent subvolume's /some/path qgroup and limits. The only way to achieve something similar is to create a common "parent" qgroup and assig

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Jens Axboe
On 04/21/2017 04:55 AM, Reshetova, Elena wrote: Please don't send any more conversions until those have been resolved. >> >> Like I suggested months ago, how about doing an efficient implementation of >> refcount_t which doesn't use the bloated cmpxchg loop? Then there would be >> no >> need

Re: Prevent escaping btrfs quota

2017-04-21 Thread Marat Khalili
Just some food for thought: there's already a tag that correctly assigns filesystem objects to users. It is called owner(ship). Instead of making qgroups repeat ownership logic, why not base qgroup assignments on ownership itself? (At least on per-subvolume basis.) -- With Best Regards, Marat

[PATCH] btrfs: fix local array declarations

2017-04-21 Thread fdmanana
From: Filipe Manana We were declaring local arrays using a notation that does not seem to be standard resulting in failures on some systems, like for example in a Debian Stretch installation with bash version 4.4.11(1)-release: $ ./check btrfs/003 btrfs/027 FSTYP -- btrfs PLATFORM -

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Jens Axboe
On 04/21/2017 09:22 AM, Peter Zijlstra wrote: > On Fri, Apr 21, 2017 at 08:03:13AM -0600, Jens Axboe wrote: >> You have it so easy - the code is completely standalone, building a >> small test framework around it and measuring performance in _user space_ >> is trivial. > > Something like this you

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Peter Zijlstra
On Fri, Apr 21, 2017 at 08:03:13AM -0600, Jens Axboe wrote: > You have it so easy - the code is completely standalone, building a > small test framework around it and measuring performance in _user space_ > is trivial. Something like this you mean: https://lkml.kernel.org/r/20170322165144.dtid

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Kees Cook
On Fri, Apr 21, 2017 at 3:55 AM, Reshetova, Elena wrote: > On Thu, Apr 20, 2017 at 11:33 AM, Eric Biggers wrote: >> Of course, having extra checks behind a debug option is fine. But they >> should >> not be part of the base feature; the base feature should just be mitigation >> of >> reference

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Eric Biggers
Hi Elena, On Fri, Apr 21, 2017 at 10:55:29AM +, Reshetova, Elena wrote: > > > > At the very least, what is there now could probably be made about twice as > > fast > > by removing the checks that don't actually help mitigate refcount overflow > > bugs, > > specifically all the checks in ref

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Kees Cook
On Fri, Apr 21, 2017 at 12:55 PM, Eric Biggers wrote: > Hi Elena, > > On Fri, Apr 21, 2017 at 10:55:29AM +, Reshetova, Elena wrote: >> > >> > At the very least, what is there now could probably be made about twice as >> > fast >> > by removing the checks that don't actually help mitigate refc

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread James Bottomley
On Fri, 2017-04-21 at 13:22 -0700, Kees Cook wrote: > On Fri, Apr 21, 2017 at 12:55 PM, Eric Biggers > wrote: > > > > Of course, having extra checks behind a debug option is fine. > > > > But they should not be part of the base feature; the base > > > > feature should just be mitigation of ref

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread Kees Cook
On Fri, Apr 21, 2017 at 2:27 PM, James Bottomley wrote: > On Fri, 2017-04-21 at 13:22 -0700, Kees Cook wrote: >> On Fri, Apr 21, 2017 at 12:55 PM, Eric Biggers >> wrote: >> > > > Of course, having extra checks behind a debug option is fine. >> > > > But they should not be part of the base featur

Re: [PATCH 0/5] v2: block subsystem refcounter conversions

2017-04-21 Thread James Bottomley
On Fri, 2017-04-21 at 14:30 -0700, Kees Cook wrote: > On Fri, Apr 21, 2017 at 2:27 PM, James Bottomley > wrote: > > On Fri, 2017-04-21 at 13:22 -0700, Kees Cook wrote: > > > On Fri, Apr 21, 2017 at 12:55 PM, Eric Biggers < > > > ebigge...@gmail.com> > > > wrote: > > > > > > Of course, having extra

[PATCH 0/2] btrfs: allow processes with exceed quota with override

2017-04-21 Thread Sargun Dhillon
This patchset makes it so that on a per-filesystem basis one can disable quota enforcement for users with cap_sys_resource. This patchset can likely later be extended to per-qgroup, or a per-volume basis. I'm thinking of extending the sysfs interface to list the qgroups and this same interface for

[PATCH 1/2] btrfs: add quota override attribute

2017-04-21 Thread Sargun Dhillon
This patch introduces the quota override flag to btrfs_fs_info, and a change to quota limit checking code to temporarily allow for quota to be overridden for processes with cap_sys_resource. It's useful for administrative programs, such as log rotation, that may need to temporarily use more disk s

[PATCH 2/2] btrfs: Add quota_override knob into sysfs

2017-04-21 Thread Sargun Dhillon
This patch adds the read-write attribute quota_override into sysfs. Any process which has cap_sys_resource can set this flag to on, and once it is set to true, processes with cap_sys_resource can exceed the quota. Signed-off-by: Sargun Dhillon --- fs/btrfs/sysfs.c | 36 ++