[PATCH][v2] btrfs: run delayed items before dropping the snapshot

2018-12-05 Thread Josef Bacik
From: Josef Bacik With my delayed refs patches in place we started seeing a large amount of aborts in __btrfs_free_extent BTRFS error (device sdb1): unable to find ref byte nr 91947008 parent 0 root 35964 owner 1 offset 0 Call Trace: ? btrfs_merge_delayed_refs+0xaf/0x340

Re: [RFC PATCH] btrfs: Remove __extent_readpages

2018-12-05 Thread Josef Bacik
some cases I'd suspect that we're not reflecting this higher fault rate in the performance counters properly. We should preserve the existing behavior, what hurts a little bit on a lightly loaded box is going to hurt a whole lot more on a heavily loaded box. Thanks, Josef

Re: [PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-12-04 Thread Josef Bacik
On Tue, Dec 04, 2018 at 01:46:58PM +0200, Nikolay Borisov wrote: > > > On 3.12.18 г. 18:06 ч., Josef Bacik wrote: > > The throttle path doesn't take cleaner_delayed_iput_mutex, which means > > we could think we're done flushing iputs in the data space reservation > &

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-12-04 Thread Josef Bacik
On Tue, Dec 04, 2018 at 11:21:14AM +0200, Nikolay Borisov wrote: > > > On 3.12.18 г. 18:06 ч., Josef Bacik wrote: > > The cleaner thread usually takes care of delayed iputs, with the > > exception of the btrfs_end_transaction_throttle path. The cleaner > > thread

[PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-12-03 Thread Josef Bacik
Manana Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 +++ fs/btrfs/disk-io.c | 3 +++ fs/btrfs/inode.c | 2 ++ 3 files changed, 8 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index c8ddbacb6748..dc56a4d940c3 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -769,6

[PATCH 1/3] btrfs: run delayed iputs before committing

2018-12-03 Thread Josef Bacik
. If there is and we freed enough we can then commit the transaction and potentially be able to make our reservation. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 9 + 1 file changed, 9 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[PATCH 0/3][V2] Delayed iput fixes

2018-12-03 Thread Josef Bacik
ing, and then cleanup and rework how we run delayed iputs to make it more straightforward to wait on them and make sure we're all done using them. Thanks, Josef

[PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-12-03 Thread Josef Bacik
the cleaner_delayed_iput_mutex whenever we flush the delayed iputs just replace it with an atomic counter and a waitqueue. This removes the short (or long depending on how big the inode is) window where we think there are no more pending iputs when there really are some. Signed-off-by: Josef Bacik --- fs/btrfs

[PATCH 8/8] btrfs: reserve extra space during evict()

2018-12-03 Thread Josef Bacik
We could generate a lot of delayed refs in evict but never have any left over space from our block rsv to make up for that fact. So reserve some extra space and give it to the transaction so it can be used to refill the delayed refs rsv every loop through the truncate path. Signed-off-by: Josef

[PATCH 1/8] btrfs: check if free bgs for commit

2018-12-03 Thread Josef Bacik
reservation. So instead of just returning ENOSPC, check if we have free block groups pending, and if so commit the transaction to allow us to use that free space. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 34

[PATCH 4/8] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-12-03 Thread Josef Bacik
tests in xfstests with my previous patch. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- fs/btrfs/extent-tree.c | 18 +- include/trace/events/btrfs.h | 1 + 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/fs

[PATCH 0/8][V2] Enospc cleanups and fixeS

2018-12-03 Thread Josef Bacik
we needed to add the ability to only ENOSPC tickets that were too big to satisfy, instead of failing all of the tickets. There's also a fix in here for one of the corner cases where we didn't quite have enough space reserved for the delayed refs we were generating during evict(). Thanks, Josef

[PATCH 2/8] btrfs: dump block_rsv whe dumping space info

2018-12-03 Thread Josef Bacik
For enospc_debug having the block rsvs is super helpful to see if we've done something wrong. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: David Sterba --- fs/btrfs/extent-tree.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/fs/btrfs/extent-tree.c

[PATCH 6/8] btrfs: loop in inode_rsv_refill

2018-12-03 Thread Josef Bacik
it re-calculate our new reservation size and try again. If our reservation size doesn't change between tries then we know we are actually out of space and can error out. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 58 +- 1 file changed

[PATCH 3/8] btrfs: don't use global rsv for chunk allocation

2018-12-03 Thread Josef Bacik
no longer require assuming the global reserve is used space in our calculations. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 9 - 1 file changed, 9 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 204b35434056..667b992d322d 100644 --- a/fs/bt

[PATCH 7/8] btrfs: be more explicit about allowed flush states

2018-12-03 Thread Josef Bacik
and to allocate chunks, everything else has the potential to deadlock. Future proof this by explicitly specifying the states that FLUSH_LIMIT is allowed to use. This will keep us from introducing bugs later on when adding new flush states. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 21

[PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-12-03 Thread Josef Bacik
of returning what reservation they did receive in hopes that it could satisfy reservations down the line. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 45 + 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs

[PATCH 08/10] btrfs: rework btrfs_check_space_for_delayed_refs

2018-12-03 Thread Josef Bacik
refs rsv. If our total size is beyond that amount then we know it's time to commit the transaction and stop any more delayed refs from being generated. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 48 ++-- fs

[PATCH 10/10] btrfs: fix truncate throttling

2018-12-03 Thread Josef Bacik
We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/inode.c

[PATCH 09/10] btrfs: don't run delayed refs in the end transaction logic

2018-12-03 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 38 -- 1 file changed, 38 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 2d8401bf8df9..01f39401619a 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -798,22 +798,12

[PATCH 02/10] btrfs: add cleanup_ref_head_accounting helper

2018-12-03 Thread Josef Bacik
From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we

[PATCH 00/10][V2] Delayed refs rsv

2018-12-03 Thread Josef Bacik
fs that could be generated. This is a niave approach and will probably evolve, but for now it works. With this patchset we've gone down to 2-8 failures per week. It's not perfect, there are some corner cases that still need to be addressed, but is significantly better than what we had. Thanks, Josef

[PATCH 06/10] btrfs: update may_commit_transaction to use the delayed refs rsv

2018-12-03 Thread Josef Bacik
have enough bytes to satisfy our reservation ticket then we are good to go, otherwise subtract out what space we would gain back by committing the transaction and compare that against the pinned space to make our decision. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 24

[PATCH 05/10] btrfs: introduce delayed_refs_rsv

2018-12-03 Thread Josef Bacik
From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted

[PATCH 04/10] btrfs: only track ref_heads in delayed_ref_updates

2018-12-03 Thread Josef Bacik
From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only

[PATCH 03/10] btrfs: cleanup extent_op handling

2018-12-03 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order

[PATCH 01/10] btrfs: add btrfs_delete_ref_head helper

2018-12-03 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent

[PATCH 07/10] btrfs: add new flushing states for the delayed refs rsv

2018-12-03 Thread Josef Bacik
delayed refs. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 10 ++ fs/btrfs/extent-tree.c | 14 ++ include/trace/events/btrfs.h | 2 ++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 52a87d446945

Re: [PATCH 1/2] btrfs: catch cow on deleting snapshots

2018-11-30 Thread Josef Bacik
On Fri, Nov 30, 2018 at 05:14:54PM +, Filipe Manana wrote: > On Fri, Nov 30, 2018 at 4:53 PM Josef Bacik wrote: > > > > From: Josef Bacik > > > > When debugging some weird extent reference bug I suspected that we were > > changing a snapshot while we were de

[PATCH 0/2] Fix aborts when dropping snapshots

2018-11-30 Thread Josef Bacik
was introduced where it could run once the snapshot was being deleted, which will result in all sorts of extent reference shenanigans. This was tricky to hit before, but with my iput changes it's become much easier to hit on our build boxes that are heavy users of snapshot creation/deletion. Thanks, Josef

[PATCH 1/2] btrfs: catch cow on deleting snapshots

2018-11-30 Thread Josef Bacik
From: Josef Bacik When debugging some weird extent reference bug I suspected that we were changing a snapshot while we were deleting it, which could explain my bug. This was indeed what was happening, and this patch helped me verify my theory. It is never correct to modify the snapshot once

[PATCH 2/2] btrfs: run delayed items before dropping the snapshot

2018-11-30 Thread Josef Bacik
From: Josef Bacik With my delayed refs patches in place we started seeing a large amount of aborts in __btrfs_free_extent BTRFS error (device sdb1): unable to find ref byte nr 91947008 parent 0 root 35964 owner 1 offset 0 Call Trace: ? btrfs_merge_delayed_refs+0xaf/0x340

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 07:59:42PM +, Chris Mason wrote: > On 27 Nov 2018, at 14:54, Josef Bacik wrote: > > > On Tue, Nov 27, 2018 at 10:26:15AM +0200, Nikolay Borisov wrote: > >> > >> > >> On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > >>&g

Re: [PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 10:29:57AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > > The throttle path doesn't take cleaner_delayed_iput_mutex, which means > > Which one is the throttle path? btrfs_end_transaction_throttle is only > cal

Re: [PATCH] btrfs: only run delayed refs if we're committing

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 07:43:39PM +, Filipe Manana wrote: > On Tue, Nov 27, 2018 at 7:22 PM Josef Bacik wrote: > > > > On Fri, Nov 23, 2018 at 04:59:32PM +, Filipe Manana wrote: > > > On Thu, Nov 22, 2018 at 12:35 AM Josef Bacik wrote: > > > > >

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 10:26:15AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > > The cleaner thread usually takes care of delayed iputs, with the > > exception of the btrfs_end_transaction_throttle path. The cleaner > > thread

Re: [PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-11-27 Thread Josef Bacik
On Mon, Nov 26, 2018 at 02:25:52PM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:03 ч., Josef Bacik wrote: > > With the introduction of the per-inode block_rsv it became possible to > > have really really large reservation requests made because of data >

Re: [PATCH] btrfs: only run delayed refs if we're committing

2018-11-27 Thread Josef Bacik
On Fri, Nov 23, 2018 at 04:59:32PM +, Filipe Manana wrote: > On Thu, Nov 22, 2018 at 12:35 AM Josef Bacik wrote: > > > > I noticed in a giant dbench run that we spent a lot of time on lock > > contention while running transaction commit. This is because dbench > >

Re: [PATCH 5/6] btrfs: introduce delayed_refs_rsv

2018-11-27 Thread Josef Bacik
On Mon, Nov 26, 2018 at 11:14:12AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 20:59 ч., Josef Bacik wrote: > > From: Josef Bacik > > > > Traditionally we've had voodoo in btrfs to account for the space that > > delayed refs may take up by having a g

Re: [PATCH 3/6] btrfs: cleanup extent_op handling

2018-11-27 Thread Josef Bacik
On Thu, Nov 22, 2018 at 12:09:34PM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 20:59 ч., Josef Bacik wrote: > > From: Josef Bacik > > > > The cleanup_extent_op function actually would run the extent_op if it > > needed running, which made the

Re: [PATCH] btrfs: Remove unnecessary code from __btrfs_rebalance

2018-11-26 Thread Josef Bacik
quot; > > Clearly, this piece of code has lost its original intent throughout > the years. It doesn't really bring any real practical benefits to the > relocation process. No functional changes. > > Signed-off-by: Nikolay Borisov > Suggested-by: Josef Bacik Reviewed-by: Josef Bacik Thanks, Josef

[PATCH 1/3] btrfs: run delayed iputs before committing

2018-11-21 Thread Josef Bacik
. If there is and we freed enough we can then commit the transaction and potentially be able to make our reservation. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 9 + 1 file changed, 9 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-21 Thread Josef Bacik
Manana Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 3da9ac463344..3c42d8887183 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -3264,6 +3264,7 @@ void btrfs_add_delayed_iput(struct inode *inode

[PATCH] btrfs: only run delayed refs if we're committing

2018-11-21 Thread Josef Bacik
. This leads to seconds of 0 throughput. Change this to only run the delayed refs if we're the ones committing the transaction. This makes the latency go away and we get no more lock contention. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/transaction.c | 24

[PATCH 0/3] Delayed iput fixes

2018-11-21 Thread Josef Bacik
wait for delayed iputs when deciding if we need to commit for enospc flushing, and then cleanup and rework how we run delayed iputs to make it more straightforward to wait on them and make sure we're all done using them. Thanks, Josef

[PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-11-21 Thread Josef Bacik
the cleaner_delayed_iput_mutex whenever we flush the delayed iputs just replace it with an atomic counter and a waitqueue. This removes the short (or long depending on how big the inode is) window where we think there are no more pending iputs when there really are some. Signed-off-by: Josef Bacik --- fs/btrfs

[PATCH 4/7] btrfs: call btrfs_create_pending_block_groups unconditionally

2018-11-21 Thread Josef Bacik
The first thing we do is loop through the list, this if (!list_empty()) btrfs_create_pending_block_groups(); thing is just wasted space. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 3 +-- fs/btrfs/transaction.c | 6 ++ 2 files changed, 3

[PATCH 6/7] btrfs: cleanup pending bgs on transaction abort

2018-11-21 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 826a15a07fce..3c1be9db897c 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -2269,6 +2269,10 @@ int btrfs_commit_transaction

[PATCH 5/7] btrfs: just delete pending bgs if we are aborted

2018-11-21 Thread Josef Bacik
We still need to do all of the accounting cleanup for pending block groups if we abort. So set the ret to trans->aborted so if we aborted the cleanup happens and everybody is happy. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 8 +++- 1 file chan

[PATCH 7/7] btrfs: wait on ordered extents on abort cleanup

2018-11-21 Thread Josef Bacik
transaction cleanup stuff. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 8 1 file changed, 8 insertions(+) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 8e7926c91e35..c5918ff8241b 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c

[PATCH 0/7] Abort cleanup fixes

2018-11-21 Thread Josef Bacik
the random leftovers and WARN_ON()'s when running whichever xfstest that was and things are much smoother now. Thanks, Josef

[PATCH 3/7] btrfs: handle delayed ref head accounting cleanup in abort

2018-11-21 Thread Josef Bacik
We weren't doing any of the accounting cleanup when we aborted transactions. Fix this by making cleanup_ref_head_accounting global and calling it from the abort code, this fixes the issue where our accounting was all wrong after the fs aborts. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h

[PATCH 2/7] btrfs: make btrfs_destroy_delayed_refs use btrfs_delete_ref_head

2018-11-21 Thread Josef Bacik
Instead of open coding this stuff use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 7 +-- 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index f062fb0487cd..7d02748cf3f6 100644

[PATCH 1/7] btrfs: make btrfs_destroy_delayed_refs use btrfs_delayed_ref_lock

2018-11-21 Thread Josef Bacik
We have this open coded in btrfs_destroy_delayed_refs, use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 11 ++- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index

[PATCH 3/8] btrfs: don't use global rsv for chunk allocation

2018-11-21 Thread Josef Bacik
readily. Instead use the actual used amount when determining if we need to allocate a chunk or not. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 9 - 1 file changed, 9 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 7a30fbc05e5e..a91b3183dcae

[PATCH 4/8] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-11-21 Thread Josef Bacik
tests in xfstests with my previous patch. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- fs/btrfs/extent-tree.c | 18 +- include/trace/events/btrfs.h | 1 + 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs

[PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-11-21 Thread Josef Bacik
of returning what reservation they did receive in hopes that it could satisfy reservations down the line. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 45 + 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs

[PATCH 7/8] btrfs: be more explicit about allowed flush states

2018-11-21 Thread Josef Bacik
for flushing with FLUSH_LIMIT and use that for our state machine. Then as we add new things that are safe we can just add them to this list. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 21 ++--- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/fs/btrfs

[PATCH 0/8] Enospc cleanups and fixes

2018-11-21 Thread Josef Bacik
for one of the corner cases where we didn't quite have enough space reserved for the delayed refs we were generating during evict(). Thanks, Josef

[PATCH 6/8] btrfs: loop in inode_rsv_refill

2018-11-21 Thread Josef Bacik
it re-calculate our new reservation size and try again. If our reservation size doesn't change between tries then we know we are actually out of space and can error out. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 56 -- 1 file changed

[PATCH 2/8] btrfs: dump block_rsv whe dumping space info

2018-11-21 Thread Josef Bacik
For enospc_debug having the block rsvs is super helpful to see if we've done something wrong. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: David Sterba --- fs/btrfs/extent-tree.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/fs/btrfs/extent-tree.c

[PATCH 8/8] btrfs: reserve extra space during evict()

2018-11-21 Thread Josef Bacik
We could generate a lot of delayed refs in evict but never have any left over space from our block rsv to make up for that fact. So reserve some extra space and give it to the transaction so it can be used to refill the delayed refs rsv every loop through the truncate path. Signed-off-by: Josef

[PATCH 1/8] btrfs: check if free bgs for commit

2018-11-21 Thread Josef Bacik
reservation. So instead of just returning ENOSPC, check if we have free block groups pending, and if so commit the transaction to allow us to use that free space. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 33 +++-- 1 file changed, 19

[PATCH 6/6] btrfs: fix truncate throttling

2018-11-21 Thread Josef Bacik
We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 79

[PATCH 5/6] btrfs: introduce delayed_refs_rsv

2018-11-21 Thread Josef Bacik
From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted

[PATCH 0/6] Delayed refs rsv

2018-11-21 Thread Josef Bacik
. With this patchset we've gone down to 2-8 failures per week. It's not perfect, there are some corner cases that still need to be addressed, but is significantly better than what we had. Thanks, Josef

[PATCH 4/6] btrfs: only track ref_heads in delayed_ref_updates

2018-11-21 Thread Josef Bacik
From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only

[PATCH 2/6] btrfs: add cleanup_ref_head_accounting helper

2018-11-21 Thread Josef Bacik
From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we

[PATCH 3/6] btrfs: cleanup extent_op handling

2018-11-21 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order

[PATCH 1/6] btrfs: add btrfs_delete_ref_head helper

2018-11-21 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent

Re: [PATCH v2] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-11-20 Thread Josef Bacik
> extent hang") Can we just remove the endio cleanup in __extent_writepage() and make this do the proper cleanup? I'm not sure if that is feasible or not, but seems like it would make the cleanup stuff less confusing and more straightforward. If not you can add Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: remove no longer used io_err from btrfs_log_ctx

2018-11-12 Thread Josef Bacik
e inode. We did this in > commit b5e6c3e170b7 ("btrfs: always wait on ordered extents at fsync > time") and commit a2120a473a80 ("btrfs: clean up the left over > logged_list usage") removed its last use. > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix rare chances for data loss when doing a fast fsync

2018-11-12 Thread Josef Bacik
; Fixes: b5e6c3e170b7 ("btrfs: always wait on ordered extents at fsync time") > CC: sta...@vger.kernel.org # 4.19+ > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [Regression bisected] btrfs: always wait on ordered extents at fsync time

2018-11-12 Thread Josef Bacik
tests I didn't see as harsh of a regression. I'll start working on making the regression suck less, thanks for running this down Mel, Josef

Re: [PATCH v15.1 04/13] btrfs: dedupe: Introduce function to remove hash from in-memory tree

2018-11-09 Thread Josef Bacik
pe_info) > + return 0; > + > + /* Don't allow disable status change in RO mount */ > + if (fs_info->sb->s_flags & MS_RDONLY) > + return -EROFS; > + > + /* > + * Wait for all unfinished writers and block further writers. > + * Then sync the whole fs so all current write will go through > + * dedupe, and all later write won't go through dedupe. > + */ > + block_all_writers(fs_info); > + ret = sync_filesystem(fs_info->sb); > + fs_info->dedupe_enabled = 0; > + fs_info->dedupe_info = NULL; > + unblock_all_writers(fs_info); This is awful, don't do this. Thanks, Josef

Re: [PATCH v15.1 01/13] btrfs: dedupe: Introduce dedupe framework and its header

2018-11-09 Thread Josef Bacik
_fs_info { > spinlock_t ref_verify_lock; > struct rb_root block_tree; > #endif > + > + /* > + * Inband de-duplication related structures > + */ > + unsigned long dedupe_enabled:1; Please use a BTRFS_FS_ flag for this. Thanks, Josef

Re: [PATCH] btrfs: Check for missing device before bio submission in btrfs_map_bio

2018-11-09 Thread Josef Bacik
count. No functional changes. > > Signed-off-by: Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: simpler and more efficient cleanup of a log tree's extent io tree

2018-11-09 Thread Josef Bacik
gt; Signed-off-by: Filipe Manana Sheesh Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix data corruption due to cloning of eof block

2018-11-07 Thread Josef Bacik
07d19dc9fbe9 ("vfs: avoid problematic remapping requests into > partial EOF block"). So this change is more geared towards stable kernels, > as it's unlikely the new VFS checks get removed intentionally. > > A test case for fstests follows soon, as well as an update to filter > existing tests that expect -EOPNOTSUPP to accept -EINVAL as well. > > CC: # 4.4+ > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 8/8] btrfs: Remove extent_io_ops::split_extent_hook callback

2018-11-01 Thread Josef Bacik
Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 7/8] btrfs: Remove extent_io_ops::merge_extent_hook callback

2018-11-01 Thread Josef Bacik
ve it a more descriptive name. No functional changes. > > Signed-off-by: Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 6/8] btrfs: Remove extent_io_ops::clear_bit_hook callback

2018-11-01 Thread Josef Bacik
isov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 5/8] btrfs: Remove extent_io_ops::set_bit_hook extent_io callback

2018-11-01 Thread Josef Bacik
ops weren't set > for the inode. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 4/8] btrfs: Remove extent_io_ops::check_extent_io_range callback

2018-11-01 Thread Josef Bacik
gt;private_data set i.e. relate to a data node and > not the btree one. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 3/8] btrfs: Remove extent_io_ops::writepage_end_io_hook

2018-11-01 Thread Josef Bacik
patch to remove the struct extent_state *state from the arg list as well. Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 2/8] btrfs: Remove extent_io_ops::writepage_start_hook

2018-11-01 Thread Josef Bacik
gt; callback definition, exports the callback function and calls it > directly at the only call site. Also give the function a more descriptive > name. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 1/8] btrfs: Remove extent_io_ops::fill_delalloc

2018-11-01 Thread Josef Bacik
ted > via the extent_io_ops structure. This patch removes the callback > definition, exports the function and calls it directly. No functional > changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] fstests: fix fssum to actually ignore file holes when supposed to

2018-10-31 Thread Josef Bacik
port holes and fallocate, we can just change > the test and pass the '-s' option to all fssum calls. > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix missing data checksums after a ranged fsync (msync)

2018-10-31 Thread Josef Bacik
t ranged fsync may collect them if needed. > Also, if we find a hole extent outside of the range still log it, just > to prevent having gaps between extent items after replaying the log, > otherwise fsck will complain when we are not using the NO_HOLES feature > (fstest btrfs/056 triggers such case). > > Fixes: e7175a692765 ("btrfs: remove the wait ordered logic in the > log_one_extent path") > CC: sta...@vger.kernel.org # 4.19+ > Signed-off-by: Filipe Manana Nice catch, Reviewed-by: Josef Bacik Josef

Re: [PATCH 0/3] fix pinned underflow in generic/475

2018-10-25 Thread Josef Bacik
rflow earlier > You can add Reviewed-by: Josef Bacik to the series, thanks, Josef

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
On Wed, Oct 24, 2018 at 01:48:40PM +0100, Filipe Manana wrote: > On Wed, Oct 24, 2018 at 1:40 PM Josef Bacik wrote: > > > > On Wed, Oct 24, 2018 at 12:53:59PM +0100, Filipe Manana wrote: > > > On Wed, Oct 24, 2018 at 12:37 PM Josef Bacik wrote: > > > > > &

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
On Wed, Oct 24, 2018 at 12:53:59PM +0100, Filipe Manana wrote: > On Wed, Oct 24, 2018 at 12:37 PM Josef Bacik wrote: > > > > On Wed, Oct 24, 2018 at 10:13:03AM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > When we

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
is_free_space_inode(inode)) path->search_commit_root = 1; in read_locked_inode? That would be cleaner. If we don't want to do that for the inode cache (I'm not sure if it's ok or not) we could just do if (root == fs_info->tree_root) instead. Thanks, Josef

Re: [PATCH v3] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-23 Thread Josef Bacik
On Mon, Oct 22, 2018 at 11:05:08PM +0100, Filipe Manana wrote: > On Mon, Oct 22, 2018 at 8:18 PM Josef Bacik wrote: > > > > On Mon, Oct 22, 2018 at 08:10:37PM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > When we

Re: [PATCH v3] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
ached block > groups have no free extents. > > Reported-by: Andrew Nelson > Link: > https://lore.kernel.org/linux-btrfs/captelenq9x5kowuq+fa7h1r3nsjg8vyith8+ifjurc_duhh...@mail.gmail.com/ > Fixes: 9d66e233c704 ("Btrfs: load free space cache if it exists") > Tested-by: Andrew Nelson > Signed-off-by: Filipe Manana Great, thanks, Reviewed-by: Josef Bacik Josef

Re: [PATCH v2] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
assed all the way down to find_free_extent properly, so we really should be able to just do if (root == fs_info->tree_root) and not add all this stuff. Not to mention this will race with anybody else doing stuff, so if another thread that isn't actually touching the tree_root it would skip caching a block group when it's completely ok for that thread to do it. Thanks, Josef

Re: [PATCH] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
t this a problem for anything that tries to allocate an extent while in the tree_root? Like we snapshot or make a subvolume or anything? We should just disallow if root == tree_root. But even then we only need to do this if we're using SPACE_CACHE, using the ye-olde caching or the free space tree are both ok. Let's just limit it to those cases. Thanks, Josef

Re: [PATCH] Btrfs: fix use-after-free when dumping free space

2018-10-22 Thread Josef Bacik
R0: 80050033 > [ 9520.416848] CR2: 0011 CR3: 000106b52000 CR4: > 06a0 > [ 9520.418477] DR0: DR1: DR2: > > [ 9520.418846] DR3: DR6: fffe0ff0 DR7: > 0400 > [ 9520.419204] Kernel panic - not syncing: Fatal exception > [ 9520.419666] Dumping ftrace buffer: > [ 9520.419930](ftrace buffer empty) > [ 9520.420168] Kernel Offset: disabled > [ 9520.420406] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > Fix this by acquiring the respective lock before iterating the rbtree. > > Reported-by: Nikolay Borisov > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

[PATCH 33/42] btrfs: fix insert_reserved error handling

2018-10-12 Thread Josef Bacik
are sure that both cases are properly cleaned up in case of a transaction abort. Reviewed-by: David Sterba Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 12 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/extent-tree.c b

[PATCH 31/42] btrfs: cleanup pending bgs on transaction abort

2018-10-12 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 46ca775a709e..9168efaca37e 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -2280,6 +2280,10 @@ int btrfs_commit_transaction

  1   2   3   4   5   6   7   8   9   10   >