Re: [PATCH v3 07/14] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-29 Thread Qu Wenruo
On 2017年09月30日 00:57, Goffredo Baroncelli wrote: On 09/28/2017 02:00 AM, Qu Wenruo wrote: On 2017年09月28日 00:20, David Sterba wrote: On Mon, Sep 25, 2017 at 07:15:30AM -0400, Austin S. Hemmelgarn wrote: On 2017-09-24 10:08, Goffredo Baroncelli wrote: On 09/24/2017 12:10 PM, Anand Jain wrot

Re: [PATCH 1/3] Btrfs: remove nr_async_bios

2017-09-29 Thread Liu Bo
On Wed, Sep 27, 2017 at 01:30:13PM +0200, David Sterba wrote: > On Thu, Sep 07, 2017 at 11:22:20AM -0600, Liu Bo wrote: > > This was intended to congest higher layers to not send bios, but as > > > > 1) the congested bit has been taken by writeback > > Can you please be more specific here? > Sur

Re: Can't remove device -> I/O error

2017-09-29 Thread DocMAX
>> Did you removed the disk before mounting (physically or doing echo 1 >/sys/block/xxx/device/delete)? Which steps you performed ? - removed drive physically - mounted degraded mode - btrfs dev del -> same i/o error -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" i

Re: Can't remove device -> I/O error

2017-09-29 Thread Goffredo Baroncelli
On 09/29/2017 11:09 PM, DocMAX wrote: > Thanks for the reply. > > I don't want to replace the drive. I want to remove. > > Also tried in degraded mode. I get the exact same error. Did you removed the disk before mounting (physically or doing echo 1 >/sys/block/xxx/device/delete)? Which steps yo

Re: Can't remove device -> I/O error

2017-09-29 Thread DocMAX
Thanks for the reply. I don't want to replace the drive. I want to remove. Also tried in degraded mode. I get the exact same error. I'm not sure but i think i formated the drive on Kernel 4.11. I am on Kernel 4.13 now. I have the bad feeling that i will never get rid of that small drive unl

Re: Can't remove device -> I/O error

2017-09-29 Thread Goffredo Baroncelli
On 09/29/2017 10:00 PM, Dirk Diggler wrote: > Hi, > > is there any chance to get my device removed? I simulated a device removing in KVM with echo 1 >/sys/block/sdj/device/delete then btrfs dev del 6 /mnt/ And I got success. But I am not sure if this is the right thing t

Re: Can't remove device -> I/O error

2017-09-29 Thread DocMAX
Kernel: Linux game 4.13.3-1-ARCH #1 SMP PREEMPT Thu Sep 21 20:33:16 CEST 2017 x86_64 GNU/Linux -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Can't remove device -> I/O error

2017-09-29 Thread Dirk Diggler
Hi, is there any chance to get my device removed? Scrub literally takes months to complete (SATA 2/3 mix, about 1 minute per gigabyte) and i'm not sure if that helps. I guess same with balance. Mabye there is a quicker way. I can do without some data if it's corrupted. I have a backup, but i want

[PATCH 05/21] btrfs: pass root to various extent ref mod functions

2017-09-29 Thread Josef Bacik
We need the actual root for the ref verifier tool to work, so change these functions to pass the root around instead. This will be used in a subsequent patch. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.c | 2 +- fs/btrfs/ctree.h | 7 --- fs/btrfs/extent-tree.c | 24

[PATCH 10/21] btrfs: breakout empty head cleanup to a helper

2017-09-29 Thread Josef Bacik
Move this code out to a helper function to further simplivy __btrfs_run_delayed_refs. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 80 -- 1 file changed, 45 insertions(+), 35 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/e

[PATCH 04/21] btrfs: add ref-verify mount option

2017-09-29 Thread Josef Bacik
This adds the infrastructure for turning ref verify on and off for a mount, to be used by a later patch. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 1 + fs/btrfs/super.c | 15 +++ 2 files changed, 16 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 93e767

[PATCH 16/21] btrfs: add a comp_refs() helper

2017-09-29 Thread Josef Bacik
Instead of open-coding the delayed ref comparisons, add a helper to do the comparisons generically and use that everywhere. We compare sequence numbers last for following patches. Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 54 -- 1 fi

[PATCH 07/21] Btrfs: only check delayed ref usage in should_end_transaction

2017-09-29 Thread Josef Bacik
We were only doing btrfs_check_space_for_delayed_refs() if the metadata space was full, ie we couldn't allocate chunks. This assumes we'll be able to allocate chunks during transaction commit, but since nothing does a LIMIT flush during the transaction commit this won't actually happen unless we h

[PATCH 06/21] Btrfs: add a extent ref verify tool

2017-09-29 Thread Josef Bacik
We were having corruption issues that were tied back to problems with the extent tree. In order to track them down I built this tool to try and find the culprit, which was pretty successful. If you compile with this tool on it will live verify every ref update that the fs makes and make sure it i

[PATCH 03/21] btrfs: make the delalloc block rsv per inode

2017-09-29 Thread Josef Bacik
The way we handle delalloc metadata reservations has gotten progressively more complicated over the years. There is so much cruft and weirdness around keeping the reserved count and outstanding counters consistent and handling the error cases that it's impossible to understand. Fix this by making

[PATCH 08/21] btrfs: add a helper to return a head ref

2017-09-29 Thread Josef Bacik
Simplify the error handling in __btrfs_run_delayed_refs by breaking out the code used to return a head back to the delayed_refs tree for processing into a helper function. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 30 +++--- 1 file changed, 15 insertions(+),

[PATCH 12/21] btrfs: move all ref head cleanup to the helper function

2017-09-29 Thread Josef Bacik
We do a couple different cleanup operations on the ref head. We adjust counters, we'll free any reserved space if we didn't end up using the ref, and we clear the pending csum bytes. Move all these disparate things into cleanup_ref_head and clean up the logic in __btrfs_run_delayed_refs so that i

[PATCH 13/21] btrfs: remove delayed_ref_node from ref_head

2017-09-29 Thread Josef Bacik
This is just excessive information in the ref_head, and makes the code complicated. It is a relic from when we had the heads and the refs in the same tree, which is no longer the case. With this removal I've cleaned up a bunch of the cruft around this old assumption as well. Signed-off-by: Josef

[PATCH 19/21] btrfs: don't call btrfs_start_delalloc_roots in flushoncommit

2017-09-29 Thread Josef Bacik
We're holding the sb_start_intwrite lock at this point, and doing async filemap_flush of the inodes will result in a deadlock if we freeze the fs during this operation. This is because we could do a btrfs_join_transaction() in the thread we are waiting on which would block at sb_start_intwrite, an

[PATCH 17/21] btrfs: track refs in a rb_tree instead of a list

2017-09-29 Thread Josef Bacik
If we get a significant amount of delayed refs for a single block (think modifying multiple snapshots) we can end up spending an ungodly amount of time looping through all of the entries trying to see if they can be merged. This is because we only add them to a list, so we have O(2n) for every ref

[PATCH 18/21] btrfs: fix send ioctl on 32bit with 64bit kernel

2017-09-29 Thread Josef Bacik
We pass in a pointer in our send arg struct, this means the struct size doesn't match with 32bit user space and 64bit kernel space. Fix this by adding a compat mode and doing the appropriate conversion. Signed-off-by: Josef Bacik --- fs/btrfs/ioctl.c | 53 +++

[PATCH 21/21] btrfs: add assertions for releasing trans handle reservations

2017-09-29 Thread Josef Bacik
These are useful for debugging problems where we mess with trans->block_rsv to make sure we're not screwing something up. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[PATCH 14/21] btrfs: remove type argument from comp_tree_refs

2017-09-29 Thread Josef Bacik
We can get this from the ref we've passed in. Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index b9b41c838da4..a2973340a94f 100644 --- a/fs/btrfs/delayed-ref.c

[PATCH 01/21] Btrfs: rework outstanding_extents

2017-09-29 Thread Josef Bacik
Right now we do a lot of weird hoops around outstanding_extents in order to keep the extent count consistent. This is because we logically transfer the outstanding_extent count from the initial reservation through the set_delalloc_bits. This makes it pretty difficult to get a handle on how and wh

[PATCH 00/21] My current btrfs patch queue

2017-09-29 Thread Josef Bacik
ertions for releasing trans handle Fixes for various problems I found while running xfstests. One is for 32bit ioctls on a 64bit kernel, two are lockdep fixes, and the last one is just so I can catch myself doing stupid stuff in the future. Dave this is based on for-next-20170929, and is in my btr

[PATCH 11/21] btrfs: move ref_mod modification into the if (ref) logic

2017-09-29 Thread Josef Bacik
We only use this logic if our ref isn't a ref_head, so move it up into the if (ref) case since we know that this is a normal ref and not a delayed ref head. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/ex

[PATCH 20/21] btrfs: move btrfs_truncate_block out of trans handle

2017-09-29 Thread Josef Bacik
Since we do a delalloc reserve in btrfs_truncate_block we can deadlock with freeze. If somebody else is trying to allocate metadata for this inode and it gets stuck in start_delalloc_inodes because of freeze we will deadlock. Be safe and move this outside of a trans handle. This also has a side-

[PATCH 15/21] btrfs: switch args for comp_*_refs

2017-09-29 Thread Josef Bacik
Because seriously? ref2 and then ref1? Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index a2973340a94f..bc940bb374cf 100644 --- a/fs/btrfs/delayed-ref.c +++ b/fs/

[PATCH 02/21] btrfs: add tracepoints for outstanding extents mods

2017-09-29 Thread Josef Bacik
This is handy for tracing problems with modifying the outstanding extents counters. Signed-off-by: Josef Bacik --- fs/btrfs/btrfs_inode.h | 2 ++ include/trace/events/btrfs.h | 21 + 2 files changed, 23 insertions(+) diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrf

[PATCH 09/21] btrfs: move extent_op cleanup to a helper

2017-09-29 Thread Josef Bacik
Move the extent_op cleanup for an empty head ref to a helper function to help simplify __btrfs_run_delayed_refs. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 77 ++ 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/fs/btrfs

Re: [PATCH v3 2/2] btrfs: Remove received_uuid during received snapshot ro->rw switch

2017-09-29 Thread Nikolay Borisov
On 29.09.2017 20:56, David Sterba wrote: > On Thu, Sep 28, 2017 at 10:53:18AM +0300, Nikolay Borisov wrote: >> Currently when a read-only snapshot is received and subsequently its ro >> property >> is set to false i.e. switched to rw-mode the received_uuid of that subvol >> remains >> intact. H

Re: delayed inodes implementation questions

2017-09-29 Thread David Sterba
On Fri, Sep 29, 2017 at 04:00:29PM +0800, Qu Wenruo wrote: > On 2017年09月29日 15:15, Nikolay Borisov wrote: > > I've been studying the implementation of the delayed inodes feature and > > have a couple of questions. > > Sorry I can't answer your question, but as you may already know by the > reject

Re: [PATCH] btrfs: Use bd_dev to generate index when dev_state_hashtable add items.

2017-09-29 Thread David Sterba
On Fri, Sep 29, 2017 at 04:16:35PM +0800, Gu Jinxiang wrote: > From: Gu JinXiang > > Fix bug of > (). Please explain what bug are you fixing. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at h

Re: [PATCH v3 2/2] btrfs: Remove received_uuid during received snapshot ro->rw switch

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 10:53:18AM +0300, Nikolay Borisov wrote: > Currently when a read-only snapshot is received and subsequently its ro > property > is set to false i.e. switched to rw-mode the received_uuid of that subvol > remains > intact. However, once the received volume is switched to RW

Re: [PATCH 1/2] btrfs: Explicitly handle btrfs_update_root failure

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 10:53:17AM +0300, Nikolay Borisov wrote: > btrfs_udpate_root can fail and it aborts the transaction, the correct way to > handle an aborted transaction is to explicitly end with btrfs_end_transaction. > Even now the code is correct since btrfs_commit_transaction would handle

Re: [PATCH v4 1/3] btrfs: undo writable when sprouting fails

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 02:51:09PM +0800, Anand Jain wrote: > When new device is being added to seed FS, seed FS is marked writable, > but when we fail to bring in the new device, we missed to undo the > writable part. This patch fixes it. > > Signed-off-by: Anand Jain > Reviewed-by: Nikolay Bori

Re: [PATCH] btrfs: take the error path out if btrfs_attach_transaction() fails

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 03:13:06PM +0800, Anand Jain wrote: > > I'd suggest to first get rid of the in-place returns and add necessary > > labels or separate exit sequences and then address the new error > > handling. > >As it goes deeper there are quite a number of things which aren't >un

Re: [PATCH 1/7] btrfs-progs: check: supplement extent backref list with rbtree

2017-09-29 Thread David Sterba
On Tue, Jul 25, 2017 at 04:51:32PM -0400, je...@suse.com wrote: > From: Jeff Mahoney > > For the pathlogical case, like xfstests generic/297 that creates a > large file consisting of one, repeating reflinked extent, fsck can > take hours. The root cause is that calling find_data_backref while >

Re: [PATCH v3 07/14] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-29 Thread Goffredo Baroncelli
On 09/28/2017 02:00 AM, Qu Wenruo wrote: > > > On 2017年09月28日 00:20, David Sterba wrote: >> On Mon, Sep 25, 2017 at 07:15:30AM -0400, Austin S. Hemmelgarn wrote: >>> On 2017-09-24 10:08, Goffredo Baroncelli wrote: On 09/24/2017 12:10 PM, Anand Jain wrote: >> >> A lot of points in this thread

Re: [PATCH] btrfs-progs: fix invalid assert in backref.c

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 04:55:00PM -0400, Josef Bacik wrote: > This should be verify'ing that we have an empty key, not that we have a > filled out key. > > Signed-off-by: Josef Bacik > --- > Dave this is on top of your ext/jeffm/extent-cache branch and fixes the > segfault > you reported. Grea

Re: [PATCH v8 0/6] Btrfs: populate heuristic with code

2017-09-29 Thread David Sterba
On Thu, Sep 28, 2017 at 05:33:35PM +0300, Timofey Titovets wrote: > Compile tested, hand tested on live system > > Change v7 -> v8 > - All code moved to compression.c (again) > - Heuristic workspaces inmplemented another way > i.e. only share logic with compression workspaces > - Some st

[GIT PULL] Btrfs fixes for 4.14-rc3

2017-09-29 Thread David Sterba
Hi, we've collected a bunch of isolated fixes, for crashes, user-visible behaviour or missing bits from other subsystem cleanups from the past. The overall number is not small but I was not able to make it significantly smaller. Most of the patches are supposed to go to stable. There are no merge

Re: [PATCH 0/4 v2] add subvolume support to mkfs

2017-09-29 Thread David Sterba
On Wed, Aug 30, 2017 at 10:24:46AM -0700, Yingyi Luo wrote: > From: yingyil > > Hi all, > > Thanks for all the comments you gave to me. I really appriciate it. I've > updated the patch, split it into small ones so that it may be easier > for you to see the changes and review. > > One use case o

Re: [PATCH v3 07/14] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-29 Thread Austin S. Hemmelgarn
On 2017-09-27 20:00, Qu Wenruo wrote: On 2017年09月28日 00:20, David Sterba wrote: On Mon, Sep 25, 2017 at 07:15:30AM -0400, Austin S. Hemmelgarn wrote: On 2017-09-24 10:08, Goffredo Baroncelli wrote: On 09/24/2017 12:10 PM, Anand Jain wrote: A lot of points in this thread, let me address the

[PATCH] btrfs: Use bd_dev to generate index when dev_state_hashtable add items.

2017-09-29 Thread Gu Jinxiang
From: Gu JinXiang Fix bug of (). Signed-off-by: Gu JinXiang --- fs/btrfs/check-integrity.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 9d3854839038..86d79bc4cfb3 100644 --- a/fs/btrfs/check-integrity.c +++ b

Re: delayed inodes implementation questions

2017-09-29 Thread Nikolay Borisov
On 29.09.2017 11:00, Qu Wenruo wrote: > > > On 2017年09月29日 15:15, Nikolay Borisov wrote: >> Hello, >> >> >> I've been studying the implementation of the delayed inodes feature and >> have a couple of questions. > > Sorry I can't answer your question, but as you may already know by the > reject

Re: delayed inodes implementation questions

2017-09-29 Thread Qu Wenruo
On 2017年09月29日 15:15, Nikolay Borisov wrote: Hello, I've been studying the implementation of the delayed inodes feature and have a couple of questions. Sorry I can't answer your question, but as you may already know by the rejection mail, Miao Xie has left Fujitsu and has been for several

[PATCH] btrfs: fix max chunk size on dup

2017-09-29 Thread Naohiro Aota
Balancing a fresh METADATA=dup btrfs file system (with size < 50G) generates a 128MB sized block group. While we set max_stripe_size = max_chunk_size = 256MB, we get this half sized block group: $ btrfs ins dump-t -t CHUNK_TREE btrfs.img|grep length length 8388608 owner 2 stripe_le

delayed inodes implementation questions

2017-09-29 Thread Nikolay Borisov
Hello, I've been studying the implementation of the delayed inodes feature and have a couple of questions. 1. Why are there 2 lists when each and every node is on both lists at the same time? So if we happen to be running the async worker we are processing nodes based on the prepared_list, at th