Re: `btrfs receive` almost coming to a halt

2013-05-10 Thread Arne Jansen
On 09.05.2013 17:14, Remco Hosman - Yerf IT wrote: kernel: 3.9.0 btrfs-progs: pulled from git this morning Trying to receive a 5gig send file. the first bit is fast, doing 10 - 50MB/sec. then it slows down. cpu usage is 50% (dual core machine). when i do a strace, it looks like this,

Re: [PATCH 0/2] introduce list_for_each_entry_del

2013-06-04 Thread Arne Jansen
On 06/04/13 16:53, Chris Mason wrote: Quoting Christoph Hellwig (2013-06-04 10:48:56) On Mon, Jun 03, 2013 at 03:55:55PM -0400, J??rn Engel wrote: Actually, when I compare the two invocations, I prefer the list_for_each_entry_del() variant over list_pop_entry(). while ((ref =

Re: [PATCH 0/2] introduce list_for_each_entry_del

2013-06-05 Thread Arne Jansen
On 05.06.2013 04:09, Jörn Engel wrote: On Tue, 4 June 2013 14:44:35 -0400, Jörn Engel wrote: Or while_list_drain? I'm fine with while_list_drain, although a name starting with list_ like all other list macros would be nice. How about just list_drain? The next question is where to put it in the

Re: [PATCH] xfstests btrfs/314: test send / receive

2013-06-07 Thread Arne Jansen
On 07.06.2013 16:50, Eric Sandeen wrote: On 6/7/13 5:29 AM, Dave Chinner wrote: On Fri, Jun 07, 2013 at 09:18:58AM +0200, Jan Schmidt wrote: (cc Arne for far-progs discussion) On Thu, June 06, 2013 at 19:54 (+0200), Eric Sandeen wrote: On 6/6/13 10:20 AM, Jan Schmidt wrote: Basic send /

Re: btrfs quota examples?

2013-06-10 Thread Arne Jansen
On 10.06.2013 08:21, Tomasz Chmielewski wrote: I'm trying to use btrfs quotas, but I'm unable to find reliable documentation on its usage. Let's start with an empty subvolume: - assign 200 MB space to it: # btrfs qgroup limit 200m /mnt/btrfs-backup/tester/ - see the usage: #

Re: btrfs quota examples?

2013-08-02 Thread Arne Jansen
On 07/29/13 10:05, Tomasz Chmielewski wrote: On Mon, 10 Jun 2013 09:41:39 +0200 Arne Jansen sensi...@gmx.net wrote: Now, my questions: - what do both 104882176 104882176 numbers represent? The first number represents the amount of data in that subvolume, regardless whether that data

Re: btrfs qgroup assign - ERROR: bad relation requested

2013-08-02 Thread Arne Jansen
On 07/31/13 07:39, Tomasz Chmielewski wrote: On Wed, 31 Jul 2013 13:13:37 +0800 Wang Shilong wangsl.f...@cn.fujitsu.com wrote: # git pull origin master Oops, i am sorry, here should: git pull origin qgroup would you please try it again Excellent, it works: # btrfs qgroup show -p

Re: Heavy memory leak when using quota groups

2013-08-05 Thread Arne Jansen
On 05.08.2013 18:35, Tomasz Chmielewski wrote: I am trying to use qgroups functionality with a basic random-write workload, it constantly keeps leaking memory within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that

Re: [PATCH 6/7] btrfs: cleanup: removed unused 'btrfs_reada_detach'

2013-08-08 Thread Arne Jansen
On 07.08.2013 23:43, Sergei Trofimovich wrote: From: Sergei Trofimovich sly...@gentoo.org Found by uselex.rb: btrfs_reada_detach: [R]: exported from: fs/btrfs/btrfs.o fs/btrfs/built-in.o fs/btrfs/reada.o even though the function is currently unused, I'm hesitating to remove it as it's part

Re: [PATCH 6/7] btrfs: cleanup: removed unused 'btrfs_reada_detach'

2013-08-08 Thread Arne Jansen
On 08/08/13 19:46, Zach Brown wrote: even though the function is currently unused, I'm hesitating to remove it as it's part of the reada-API and might be handy for anyone going to use the API in the future. I agree. As replied here,

Re: [PATCH 1/3] Btrfs: fix oops when writing dirty qgroups to disk

2013-08-09 Thread Arne Jansen
) { - struct btrfs_qgroup *qgroup = find_qgroup_rb(fs_info, qgroupid); - struct btrfs_qgroup_list *list; + struct btrfs_qgroup_list *list = NULL; Why do you initialize list to NULL here? It's always assigned before used. otherwise, Reviewed-by: Arne Jansen sensi...@gmx.net - if (!qgroup

Re: [PATCH 3/3] Btrfs: remove reduplicate check when disabling quota

2013-08-09 Thread Arne Jansen
); - if (!quota_root) { - ret = -EINVAL; - goto out; - } - ret = btrfs_clean_quota_tree(trans, quota_root); if (ret) goto out; Reviewed-by: Arne Jansen sensi...@gmx.net -- To unsubscribe from this list: send the line unsubscribe linux-btrfs

Re: [PATCH 10/15] btrfs-progs: fix qgroup realloc inheritance

2013-08-18 Thread Arne Jansen
, so I guess this isn't a very important thing to do :). It only seems to hit if you give -[cx] before -i. I guess only very few people use these options in the first place. They are primarily for hosting providers. Reviewed-by: Arne Jansen sensi...@gmx.net Signed-off-by: Zach Brown z

Purpose of block reserves

2010-12-09 Thread Arne Jansen
Hi, while reading btrfs source code, I try to make sense of the various uses of block reserves. The working theory is as follows: 1. Every operation has to reserve upfront every single byte it needs to complete its operation fully. 2. If an operation cannot determine how much space it will

Re: Purpose of block reserves

2010-12-10 Thread Arne Jansen
Arne Jansen wrote: while reading btrfs source code, I try to make sense of the various uses of block reserves. The working theory is as follows: Since there have been no protests, I added this information to the wiki, section Code documentation. -- Arne 1. Every operation has to reserve

[PATCH] Fixing the chunk allocator to allow it to better utilize the devices

2010-12-11 Thread Arne Jansen
balancing the devices better. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/extent-tree.c |4 +- fs/btrfs/volumes.c | 135 +--- 2 files changed, 73 insertions(+), 66 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent

Re: [PATCH 4/6] btrfs: restructure find_free_dev_extent()

2010-12-22 Thread Arne Jansen
Hi, this patch seems to have the same intention as the patch I sent to the list on Dec 11 Fixing the chunk allocator to allow it to better utilize the devices. The result is quite similar, except that you left the line search_start = max(root-fs_info-alloc_start, search_start); in place, which

[PATCH v1 2/6] btrfs: make struct map_lookup public

2011-03-08 Thread Arne Jansen
definition of struct map_lookup moved from volumes.c to the header Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 14 -- fs/btrfs/volumes.h | 14 ++ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs

[PATCH v1 0/6] btrfs: scrub

2011-03-08 Thread Arne Jansen
the new roots. For future improvements please see the inline comments. The accompanying user mode patches will follow shortly. Thanks, Arne Arne Jansen (5): Jan Schmidt (1): btrfs: add parameter to btrfs_lookup_csum_range btrfs: make struct map_lookup public btrfs: add state information for scrub

[PATCH v1 1/6] btrfs: add parameter to btrfs_lookup_csum_range

2011-03-08 Thread Arne Jansen
A parameter is added to search the commit root instead of the live root. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 ++-- fs/btrfs/file-item.c |8 +++- fs/btrfs/inode.c |2 +- fs/btrfs/relocation.c |2 +- fs/btrfs/tree-log.c |4

[PATCH v1 4/6] btrfs: add scrub code and prototypes

2011-03-08 Thread Arne Jansen
This is the main scrub code. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/Makefile |2 +- fs/btrfs/ctree.h | 14 + fs/btrfs/scrub.c | 1462 + 3 files changed, 1477 insertions(+), 1 deletions(-) diff --git a/fs/btrfs

[PATCH v1 5/6] btrfs: sync scrub with commit device removal

2011-03-08 Thread Arne Jansen
for the particular device gets cancelled Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/disk-io.c |1 + fs/btrfs/transaction.c |3 +++ fs/btrfs/tree-log.c|2 ++ fs/btrfs/volumes.c |2 ++ 4 files changed, 8 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/disk-io.c

[PATCH v1 3/6] btrfs: add state information for scrub

2011-03-08 Thread Arne Jansen
Add structures and state information needed for scrub Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h | 26 ++ fs/btrfs/disk-io.c | 15 +++ fs/btrfs/ioctl.h | 29 + fs/btrfs/volumes.h |3 +++ 4 files

[PATCH v2 2/6] btrfs: make struct map_lookup public

2011-03-11 Thread Arne Jansen
definition of struct map_lookup moved from volumes.c to the header Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 14 -- fs/btrfs/volumes.h | 14 ++ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs

[PATCH v2 0/6] btrfs: scrub

2011-03-11 Thread Arne Jansen
the new roots. For future improvements please see the inline comments. The accompanying user mode patches will follow shortly. This v2 mainly changes the dev_info ioctl interface. Thanks, Arne Arne Jansen (5): btrfs: add parameter to btrfs_lookup_csum_range btrfs: make struct map_lookup public

[PATCH v2 3/6] btrfs: add scrub code and prototypes

2011-03-11 Thread Arne Jansen
This is the main scrub code. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/Makefile |2 +- fs/btrfs/ctree.h | 14 + fs/btrfs/scrub.c | 1463 + 3 files changed, 1478 insertions(+), 1 deletions(-) diff --git a/fs/btrfs

[PATCH v2 5/6] btrfs: add state information for scrub

2011-03-11 Thread Arne Jansen
Add structures and state information needed for scrub Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h | 26 ++ fs/btrfs/disk-io.c | 15 +++ fs/btrfs/ioctl.h | 17 + fs/btrfs/volumes.h |3 +++ 4 files changed, 61

[PATCH v2 6/6] btrfs: new ioctls for scrub

2011-03-11 Thread Arne Jansen
canceled. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |2 - fs/btrfs/ioctl.c | 131 ++ fs/btrfs/ioctl.h | 38 3 files changed, 169 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs

[PATCH v2 1/6] btrfs: add parameter to btrfs_lookup_csum_range

2011-03-11 Thread Arne Jansen
A parameter is added to search the commit root instead of the live root. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 ++-- fs/btrfs/file-item.c |8 +++- fs/btrfs/inode.c |2 +- fs/btrfs/relocation.c |2 +- fs/btrfs/tree-log.c |4

Re: [PATCH v2 3/6] btrfs: add scrub code and prototypes

2011-03-12 Thread Arne Jansen
Hi David, thanks for your reviews. I'll update the code accordingly, comments follow inline. -- Arne David Sterba wrote: On Fri, Mar 11, 2011 at 03:49:40PM +0100, Arne Jansen wrote: This is the main scrub code. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/Makefile |2 +- fs

Re: [PATCH v2 5/6] btrfs: add state information for scrub

2011-03-12 Thread Arne Jansen
David Sterba wrote: On Fri, Mar 11, 2011 at 03:49:42PM +0100, Arne Jansen wrote: Add structures and state information needed for scrub Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h | 26 ++ fs/btrfs/disk-io.c | 15 +++ fs/btrfs

Re: [PATCH v2 0/6] btrfs: scrub

2011-03-12 Thread Arne Jansen
Ric Wheeler wrote: On 03/11/2011 09:49 AM, Arne Jansen wrote: Great work! I do wonder if we should also worry about the unallocated part of the storage device. Often, with local disks specifically, your unused space might accumulate errors over time. What you add to your scrub phase

[PATCH v3 4/6] btrfs: sync scrub with commit device removal

2011-03-12 Thread Arne Jansen
for the particular device gets cancelled Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/disk-io.c |1 + fs/btrfs/transaction.c |3 +++ fs/btrfs/tree-log.c|2 ++ fs/btrfs/volumes.c |2 ++ 4 files changed, 8 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/disk-io.c

[PATCH v3 2/6] btrfs: make struct map_lookup public

2011-03-12 Thread Arne Jansen
definition of struct map_lookup moved from volumes.c to the header Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 14 -- fs/btrfs/volumes.h | 14 ++ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs

[PATCH v3 0/6] btrfs: scrub

2011-03-12 Thread Arne Jansen
the new roots. For future improvements please see the inline comments. The accompanying user mode patches will follow shortly. V2 mainly changes the dev_info ioctl interface. V3 corrects some EIO handling and cleanup issues. Thanks, Arne Arne Jansen (5): btrfs: add parameter

[PATCH v3 6/6] btrfs: new ioctls for scrub

2011-03-12 Thread Arne Jansen
canceled. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |2 - fs/btrfs/ioctl.c | 131 ++ fs/btrfs/ioctl.h | 38 3 files changed, 169 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs

[PATCH v3 1/6] btrfs: add parameter to btrfs_lookup_csum_range

2011-03-12 Thread Arne Jansen
A parameter is added to search the commit root instead of the live root. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 ++-- fs/btrfs/file-item.c |8 +++- fs/btrfs/inode.c |2 +- fs/btrfs/relocation.c |2 +- fs/btrfs/tree-log.c |4

[PATCH v3 5/6] btrfs: add state information for scrub

2011-03-12 Thread Arne Jansen
Add structures and state information needed for scrub Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h | 25 + fs/btrfs/disk-io.c | 15 +++ fs/btrfs/ioctl.h | 17 + fs/btrfs/volumes.h |3 +++ 4 files changed, 60

[PATCH v3 3/6] btrfs: add scrub code and prototypes

2011-03-12 Thread Arne Jansen
This is the main scrub code. Updates v3: - fixed EIO handling, need to reallocate bio instead of reusing it - Updated according to David Sterba's review - don't clobber bi_flags on reuse, just set UPTODATE - allocate long living bios with bio_kmalloc instead of bio_alloc Signed-off-by: Arne

Re: [PATCH v3 3/6] btrfs: add scrub code and prototypes

2011-03-14 Thread Arne Jansen
On 14.03.2011 00:50, Ilya Dryomov wrote: On Sat, Mar 12, 2011 at 03:50:42PM +0100, Arne Jansen wrote: This is the main scrub code. + int nstripes; It should be u64 nstripes. A few lines below you assign the stripe length, it is a 64 bit quantity, also do_div() expects a 64 bit

[PATCH] btrfs-map-logical: usage update

2011-03-14 Thread Arne Jansen
The option to give the number of bytes to read is -b, not -s. Updated usage accordingly. Signed-off-by: Arne Jansen sensi...@gmx.net --- btrfs-map-logical.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/btrfs-map-logical.c b/btrfs-map-logical.c index aa4b5b5..f4695aa

[PATCH] btrfs scrub: don't restart chunk on commit

2011-03-15 Thread Arne Jansen
those out. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/scrub.c | 44 ++-- 1 files changed, 22 insertions(+), 22 deletions(-) diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 915641d..f36f5bf 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs

Re: [PATCH v3 3/6] btrfs: add scrub code and prototypes

2011-03-16 Thread Arne Jansen
On 16.03.2011 23:07, Andi Kleen wrote: Arne Jansensensi...@gmx.net writes: +*/ + mutex_lock(fs_info-scrub_lock); + atomic_inc(fs_info-scrubs_running); + mutex_unlock(fs_info-scrub_lock); It seems odd to protect an atomic_inc with a mutex. Is that done for some side

Re: [PATCH] btrfs: quasi-round-robin for chunk allocation

2011-03-17 Thread Arne Jansen
On 09.02.2011 04:03, Miao Xie wrote: On tue, 8 Feb 2011 19:03:32 +0100, Arne Jansen wrote: In a multi device setup, the chunk allocator currently always allocates chunks on the devices in the same order. This leads to a very uneven distribution, especially with RAID1 or RAID10 and an uneven

[PATCH] btrfs scrub: extent_map reference not freed

2011-03-17 Thread Arne Jansen
scrub_chunk failed to free all extent_maps it referenced, leading to a leak of these maps on unmount. Reported-by: Ilya Dryomov idryo...@gmail.com Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/scrub.c | 13 - 1 files changed, 8 insertions(+), 5 deletions(-) diff --git

[PATCH] Btrfs scrub: revised locking scheme

2011-03-17 Thread Arne Jansen
This patch removes unnecessary mutex locks around scrubs_running. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/scrub.c | 12 +++- 1 files changed, 3 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index ee7671f..85a4d4b 100644 --- a/fs/btrfs

scrub git access

2011-03-17 Thread Arne Jansen
Hello all, The collection of patches is now also available via git at git://git.kernel.org/pub/scm/linux/kernel/git/arne/btrfs-unstable- arne.git and git://git.kernel.org/pub/scm/linux/kernel/git/arne/btrfs-progs-unstable- arne.git both in the 'scrub' branch. It is based on chris' current

[PATCH v4 4/6] btrfs: sync scrub with commit device removal

2011-03-18 Thread Arne Jansen
for the particular device gets cancelled Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/disk-io.c |1 + fs/btrfs/transaction.c |3 +++ fs/btrfs/tree-log.c|2 ++ fs/btrfs/volumes.c |2 ++ 4 files changed, 8 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/disk-io.c

[PATCH v4 1/6] btrfs: add parameter to btrfs_lookup_csum_range

2011-03-18 Thread Arne Jansen
A parameter is added to search the commit root instead of the live root. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 ++-- fs/btrfs/file-item.c |8 +++- fs/btrfs/inode.c |2 +- fs/btrfs/relocation.c |2 +- fs/btrfs/tree-log.c |4

[PATCH v4 6/6] btrfs: new ioctls for scrub

2011-03-18 Thread Arne Jansen
canceled. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |2 - fs/btrfs/ioctl.c | 131 ++ fs/btrfs/ioctl.h | 38 3 files changed, 169 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs

[PATCH v4 3/6] btrfs: add scrub code and prototypes

2011-03-18 Thread Arne Jansen
This is the main scrub code. Updates v3: - fixed EIO handling, need to reallocate bio instead of reusing it - Updated according to David Sterba's review - don't clobber bi_flags on reuse, just set UPTODATE - allocate long living bios with bio_kmalloc instead of bio_alloc Updates v4: -

[PATCH v4 0/6] btrfs: scrub

2011-03-18 Thread Arne Jansen
, leaking them on unmount - remove unnecessary mutex locks around 'scrubs_running' Thanks, Arne Arne Jansen (5): btrfs: add parameter to btrfs_lookup_csum_range btrfs: make struct map_lookup public btrfs: add scrub code and prototypes btrfs: sync scrub with commit device removal btrfs: add

[PATCH v4 2/6] btrfs: make struct map_lookup public

2011-03-18 Thread Arne Jansen
definition of struct map_lookup moved from volumes.c to the header Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 14 -- fs/btrfs/volumes.h | 14 ++ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs

[PATCH v4 5/6] btrfs: add state information for scrub

2011-03-18 Thread Arne Jansen
Add structures and state information needed for scrub Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h | 25 + fs/btrfs/disk-io.c | 15 +++ fs/btrfs/ioctl.h | 17 + fs/btrfs/volumes.h |3 +++ 4 files changed, 60

Re: [PATCH] btrfs: quasi-round-robin for chunk allocation

2011-03-18 Thread Arne Jansen
On 18.03.2011 17:25, cwillu wrote: On Fri, Mar 18, 2011 at 8:40 AM, Chris Masonchris.ma...@oracle.com wrote: I think that filling all the devices fully is more important than the initial spread. Miao is correct that the administrator will probably complain if all the devices aren't used for

Re: [PATCH] btrfs scrub: make fixups sync, don't reuse fixup bios

2011-03-23 Thread Arne Jansen
Hi Ilya, On 18.03.2011 17:21, Ilya Dryomov wrote: [CC'ing the list to make development public, the patch is againt Arne's tree at kernel.org] Below is a quite long diff the primary purpose of which is to make fixups totally sync. They are already sync for checksum failures, this patch

[RFC] Tree fragmentation and prefetching

2011-03-23 Thread Arne Jansen
While looking into the performance of scrub I noticed that a significant amount of time is being used for loading the extent tree and the csum tree. While this is no surprise I did some prototyping on how to improve on it. The main idea is to load the tree (or parts of it) top-down, order the

Re: [PATCH v2 3/6] btrfs: add scrub code and prototypes

2011-03-23 Thread Arne Jansen
On 22.03.2011 17:38, David Sterba wrote: David Sterba wrote: On Fri, Mar 11, 2011 at 03:49:40PM +0100, Arne Jansen wrote: This is the main scrub code. sizeof(struct scrub_dev) == 18760 on an x86_64, an order 3 allocation in scrub_setup_dev() Is this a problem? There are only few

Re: [RFC] Tree fragmentation and prefetching

2011-03-23 Thread Arne Jansen
On 23.03.2011 14:06, Arne Jansen wrote: While looking into the performance of scrub I noticed that a significant amount of time is being used for loading the extent tree and the csum tree. While this is no surprise I did some prototyping on how to improve on it. The main idea is to load

Re: [RFC] Tree fragmentation and prefetching

2011-03-23 Thread Arne Jansen
On 23.03.2011 20:26, Andrey Kuzmin wrote: On Wed, Mar 23, 2011 at 4:06 PM, Arne Jansensensi...@gmx.net wrote: While looking into the performance of scrub I noticed that a significant amount of time is being used for loading the extent tree and the csum tree. While this is no surprise I did

Re: [RFC] Tree fragmentation and prefetching

2011-03-24 Thread Arne Jansen
On 24.03.2011 02:38, Miao Xie wrote: On wed, 23 Mar 2011 21:28:25 +0100, Arne Jansen wrote: On 23.03.2011 20:26, Andrey Kuzmin wrote: On Wed, Mar 23, 2011 at 4:06 PM, Arne Jansensensi...@gmx.net wrote: The main idea is to load the tree (or parts of it) top-down, order the needed blocks

Re: [PATCH v4 3/6] btrfs: add scrub code and prototypes

2011-03-24 Thread Arne Jansen
On 23.03.2011 18:18, David Sterba wrote: Hi, I'm reviewing the atomic counters and the wait/wake infrastructure, just found two missed mutex_unlock()s in btrfs_scrub_dev() in error paths. On Fri, Mar 18, 2011 at 04:55:06PM +0100, Arne Jansen wrote: This is the main scrub code

Re: [PATCH v4 4/6] btrfs: sync scrub with commit device removal

2011-03-24 Thread Arne Jansen
On 23.03.2011 18:28, David Sterba wrote: Hi, you are adding a new smp_mb, can you please explain why it's needed and document it? thanks, dave On Fri, Mar 18, 2011 at 04:55:07PM +0100, Arne Jansen wrote: This adds several synchronizations: - for a transaction commit, the scrub gets

Re: [PATCH v4 4/6] btrfs: sync scrub with commit device removal

2011-03-24 Thread Arne Jansen
On 24.03.2011 13:58, Arne Jansen wrote: On 23.03.2011 18:28, David Sterba wrote: Hi, you are adding a new smp_mb, can you please explain why it's needed and document it? thanks, dave On Fri, Mar 18, 2011 at 04:55:07PM +0100, Arne Jansen wrote: This adds several synchronizations

Re: [RFC] Tree fragmentation and prefetching

2011-03-25 Thread Arne Jansen
On 23.03.2011 20:32, Chris Mason wrote: Excerpts from Arne Jansen's message of 2011-03-23 09:06:02 -0400: For the implementation I'd need an interface which I haven't been able to find yet. Currently I can trigger the read of several pages / tree blocks and wait for the completion of each of

Re: [RFC] Tree fragmentation and prefetching

2011-03-25 Thread Arne Jansen
On 25.03.2011 21:15, Chris Mason wrote: Excerpts from Arne Jansen's message of 2011-03-25 16:14:35 -0400: On 23.03.2011 20:32, Chris Mason wrote: Excerpts from Arne Jansen's message of 2011-03-23 09:06:02 -0400: For the implementation I'd need an interface which I haven't been able to find

Re: [PATCH 1/2] Btrfs: fix OOPS of empty filesystem after balance

2011-03-30 Thread Arne Jansen
Am 10.03.2011 13:28, schrieb Chris Mason: Excerpts from liubo's message of 2011-03-10 03:50:27 -0500: On 03/07/2011 10:13 AM, liubo wrote: btrfs will remove unused block groups after balance. When a empty filesystem is balanced, the block group with tag DATA may be dropped, and after umount

Re: [PATCH 0/2] btrfs: allow cross-subvolume BTRFS_IOC_CLONE

2011-03-31 Thread Arne Jansen
On 31.03.2011 08:36, Christoph Hellwig wrote: On Thu, Mar 31, 2011 at 12:00:11AM -0400, Larry D'Anna wrote: This is a simple patch to allow reflinks to be made crossing subvolume boundaries. NAK. subvolumes will have to become vfsmounts sooner or later, and we really must not support any

Re: Odd rebalancing behavior

2011-04-05 Thread Arne Jansen
On 04.04.2011 12:14, Michel Alexandre Salim wrote: I have an external 4-disk enclosure, connected through USB 2.0 (my laptop does not have a USB 3.0 connector, and the eSATA connector somehow does not work); it initially had a 2-disk btrfs soft-RAID1 file system (both data and metadata are

Re: [PATCH] Btrfs: don't kzalloc the ordered extents

2011-04-06 Thread Arne Jansen
On 06.04.2011 01:20, Josef Bacik wrote: We initialize almost all of the fields when we allocate an ordered extent, so use kmalloc instead of kzalloc and just initialize the other fields that we don't already initialize yet. Thanks, Is it really worth it? we have seen a few bugs in the past

Re: cloning single-device btrfs file system onto multi-device one

2011-04-06 Thread Arne Jansen
On 28.03.2011 15:17, Stephane Chazelas wrote: I then did a btrfs fi balance again and let it run through. However here is what I get: $ df -h /mnt FilesystemSize Used Avail Use% Mounted on /dev/sdb 8.2T 3.5T 3.2T 53% /mnt Only 3.2T left. How would I reclaim

[RFC][PATCH] btrfs: using cached extent_state in set/unlock combinations

2011-04-06 Thread Arne Jansen
the cache. Only end_bio_extent_readpage is updated that it first gets a cached state to pass it to the readpage_end_io_hook as the prototype requested and is later on being used for set/unlock. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/extent_io.c | 70

Re: cloning single-device btrfs file system onto multi-device one

2011-04-06 Thread Arne Jansen
On 06.04.2011 14:05, Stephane Chazelas wrote: 2011-04-06 10:25:00 +0200, Arne Jansen: On 28.03.2011 15:17, Stephane Chazelas wrote: I then did a btrfs fi balance again and let it run through. However here is what I get: $ df -h /mnt FilesystemSize Used Avail Use% Mounted

Re: wrong values in df and btrfs filesystem df

2011-04-11 Thread Arne Jansen
On 11.04.2011 09:29, Stephane Chazelas wrote: 2011-04-10 18:13:51 +0800, Miao Xie: [...] What's the implication of having disks of differing sizes? Does that mean that the extra space on larger disks is lost? Yes. Currently the allocator cannot handle different sizes well, especially when

Re: [PATCH] btrfs: quasi-round-robin for chunk allocation

2011-04-11 Thread Arne Jansen
On 08.02.2011 19:03, Arne Jansen wrote: In a multi device setup, the chunk allocator currently always allocates chunks on the devices in the same order. This leads to a very uneven distribution, especially with RAID1 or RAID10 and an uneven number of devices. This patch always sorts the device

[PATCH v2 1/3] btrfs: move btrfs_cmp_device_free_bytes to super.c

2011-04-12 Thread Arne Jansen
this function won't be used here anymore, so move it super.c where it is used for df-calculation Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/super.c | 25 + fs/btrfs/volumes.c | 13 - fs/btrfs/volumes.h | 15 --- 3 files

[PATCH v2 0/3] btrfs: quasi-round-robin for chunk allocation

2011-04-12 Thread Arne Jansen
. We still avoid allocating stripes that are not a multiple of STRIPE_SIZE. Changes from v1: - split into multiple parts - added some comments - generated with --patience for better readability Arne Jansen (3): btrfs: move btrfs_cmp_device_free_bytes to super.c btrfs: heed alloc_start

[PATCH v2 2/3] btrfs: heed alloc_start

2011-04-12 Thread Arne Jansen
as the allocator failed to take alloc_start into account when calculating the request chunk size. As this gets fixed by this patch, the workaround is not necessary anymore. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c |5 + 1 files changed, 1 insertions(+), 4 deletions(-) diff

[PATCH v2 3/3] btrfs: quasi-round-robin for chunk allocation

2011-04-12 Thread Arne Jansen
. We still avoid allocating stripes that are not a multiple of STRIPE_SIZE. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 474 +++- fs/btrfs/volumes.h |1 + 2 files changed, 169 insertions(+), 306 deletions(-) diff --git

Re: [PATCH] Btrfs: don't reserve metadata when we're using the delalloc reserve

2011-04-13 Thread Arne Jansen
On 13.04.2011 18:06, Josef Bacik wrote: There are a bunch of places where we do btrfs_join_transaction(root, 1), but really we set the block_rsv to the delalloc reserve because our metadata was reserved at delalloc time. This means we don't need to reserve space at all, and can just join the

Re: [PATCH] Btrfs: do not release delalloc space until after we end the transaction

2011-04-13 Thread Arne Jansen
On 13.04.2011 20:54, Josef Bacik wrote: There have been many sporadic reports of the following panic [ cut here ] kernel BUG at fs/btrfs/extent-tree.c:5498! invalid opcode: [#1] PREEMPT SMP last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map

[PATCH v3 0/3] btrfs: quasi-round-robin for chunk allocation

2011-05-02 Thread Arne Jansen
initialization lacked a case for the 'single' type, thus leaving devs_max at the wrong value Arne Jansen (3): btrfs: move btrfs_cmp_device_free_bytes to super.c btrfs: heed alloc_start btrfs: quasi-round-robin for chunk allocation fs/btrfs/super.c | 25 +++ fs/btrfs/volumes.c | 494

[PATCH v3 1/3] btrfs: move btrfs_cmp_device_free_bytes to super.c

2011-05-02 Thread Arne Jansen
this function won't be used here anymore, so move it super.c where it is used for df-calculation Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/super.c | 25 + fs/btrfs/volumes.c | 13 - fs/btrfs/volumes.h | 15 --- 3 files

[PATCH v3 2/3] btrfs: heed alloc_start

2011-05-02 Thread Arne Jansen
as the allocator failed to take alloc_start into account when calculating the request chunk size. As this gets fixed by this patch, the workaround is not necessary anymore. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c |5 + 1 files changed, 1 insertions(+), 4 deletions(-) diff

[PATCH v3 3/3] btrfs: quasi-round-robin for chunk allocation

2011-05-02 Thread Arne Jansen
. We still avoid allocating stripes that are not a multiple of STRIPE_SIZE. Changes from v2: - bugfix for 'single' raid type; the initial parameter initialization lacked a case for the 'single' type, thus leaving devs_max at the wrong value Signed-off-by: Arne Jansen sensi...@gmx.net --- fs

Re: btrfs raid health status

2011-05-02 Thread Arne Jansen
On 02.05.2011 15:13, Roman Kapusta wrote: I have unmounted and again mounted btrfs filesystem, now I can see errors in dmesg. Is there also some error/health summary, like in btrfs fi show? There is no error accounting or reparing yet, but you can check out the scrub patches. They'll read and

[PATCH] btrfs progs: fix extra metadata chunk allocation in --mixed case

2011-05-05 Thread Arne Jansen
be to pull the full changeset for the mixed case from the kernel into the user mode (or, even better, share the code) The additional chunk probably confused block_rsv calculation, which in turn led to severeal ENOSPC Oopses. Signed-off-by: Arne Jansen sensi...@gmx.net --- extent-tree.c |7

[PATCH v4 0/3] btrfs: quasi-round-robin for chunk allocation

2011-05-13 Thread Arne Jansen
: - move declarations from function-level to local scopes where possible - remove an unnecessary BUG_ON - change BUG - WARN if a device is not writable and ignore the device - rename *num_bytes to num_bytes_out for consistency - transform a while loop that mimics a for loop into a for loop Arne

[PATCH v4 2/3] btrfs: heed alloc_start

2011-05-13 Thread Arne Jansen
as the allocator failed to take alloc_start into account when calculating the request chunk size. As this gets fixed by this patch, the workaround is not necessary anymore. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c |5 + 1 files changed, 1 insertions(+), 4 deletions(-) diff

[PATCH v4 1/3] btrfs: move btrfs_cmp_device_free_bytes to super.c

2011-05-13 Thread Arne Jansen
this function won't be used here anymore, so move it super.c where it is used for df-calculation Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/super.c | 26 ++ fs/btrfs/volumes.c | 13 - fs/btrfs/volumes.h | 15 --- 3 files

[PATCH v4 3/3] btrfs: quasi-round-robin for chunk allocation

2011-05-13 Thread Arne Jansen
. We still avoid allocating stripes that are not a multiple of STRIPE_SIZE. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/volumes.c | 503 +++- fs/btrfs/volumes.h |1 + 2 files changed, 188 insertions(+), 316 deletions(-) diff --git

Re: Backref walking utilities

2011-05-23 Thread Arne Jansen
Hi liubo, On 23.05.2011 11:53, liubo wrote: As one of my plans, I'm going to take this project over unless someone has been working on it. Jan Schmidt has a patch for scrub nearly ready, that does some ref-walking to report affected files to the user. While this is kernel code and you're

[PATCH v1 2/5] btrfs: state information for readahead

2011-05-23 Thread Arne Jansen
Add state information for readahead to btrfs_fs_info and btrfs_device Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 fs/btrfs/disk-io.c |4 fs/btrfs/volumes.c |8 fs/btrfs/volumes.h |8 4 files changed, 24 insertions(+), 0

[PATCH v1 4/5] btrfs: hooks for readahead

2011-05-23 Thread Arne Jansen
would wait indefinitely for the extent to finish. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/disk-io.c | 16 fs/btrfs/extent_io.c |2 +- 2 files changed, 17 insertions(+), 1 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 3d4f9c5..7949f30

[PATCH v1 1/5] btrfs: add READAHEAD extent state

2011-05-23 Thread Arne Jansen
Add a READAHEAD extent state flag. Add a function to trigger a read with this flag set. Reset the flag in the end_io hook Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/disk-io.c | 36 fs/btrfs/disk-io.h |2 ++ fs/btrfs/extent_io.c | 10

[PATCH v1 0/5] btrfs: generic readeahead interface

2011-05-23 Thread Arne Jansen
the extent tree (traditional): 351s - enumerating the extent tree (readahead): 41s - enumerating extents+csum tree (readahead): 49s The implementation is also tested with this tool in various combinations of parallel reads of the same and of different trees. Arne Jansen (5): btrfs: add READAHEAD

[PATCH v1 3/5] btrfs: initial readahead code and prototypes

2011-05-23 Thread Arne Jansen
faster than the 2 started one after another. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/Makefile |2 +- fs/btrfs/ctree.h |9 + fs/btrfs/reada.c | 963 + 3 files changed, 973 insertions(+), 1 deletions(-) diff --git a/fs

[PATCH v1 5/5] btrfs: test ioctl for readahead

2011-05-23 Thread Arne Jansen
This ioctl is added to trigger a readahead from user mode. It implements a readahead using the new interface and also a traditional tree walk. This way it's possible to measure the two side by side. Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ioctl.c | 86

[PATCH] btrfs-progs: btrfs-reada, test program for readahead

2011-05-23 Thread Arne Jansen
-by: Arne Jansen sensi...@gmx.net --- Makefile |5 +- btrfs-reada.c | 180 + ioctl.h | 16 + 3 files changed, 200 insertions(+), 1 deletions(-) create mode 100644 btrfs-reada.c diff --git a/Makefile b/Makefile index 6e6f6c6

Re: [PATCH v1 2/5] btrfs: state information for readahead

2011-05-24 Thread Arne Jansen
On 25.05.2011 07:22, liubo wrote: On 05/23/2011 08:59 AM, Arne Jansen wrote: Add state information for readahead to btrfs_fs_info and btrfs_device Signed-off-by: Arne Jansen sensi...@gmx.net --- fs/btrfs/ctree.h |4 fs/btrfs/disk-io.c |4 fs/btrfs/volumes.c |8

  1   2   3   4   >