[PATCH RFC 01/14] btrfs: file-item: Introduce btrfs_setup_file_extent function.

2015-07-28 Thread Qu Wenruo
This new function is just used to fill the file extent with given numbers. This is mainly used for later cleanup of duplicated file extent setup codes in inode.c, but also centralized the safety check for later expansion. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/ctree.h

[PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Qu Wenruo
Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. [[CORE FEATURES]] The main design concept is the following: 1) Controllable memory usage 2) No guarantee to dedup every duplication. 3) No on-disk format change or new format 4)

[PATCH v2] Btrfs: fix warning in backref walking

2015-07-28 Thread Liu Bo
When we do backref walking, we search firstly in queued delayed refs and then the on-disk backrefs, but we parse differently for shared references, for delayed refs we also add 'ref-root' while for on-disk backrefs we don't, this can prevent us from merging refs indexed by the same bytenr and

Re: [GIT PULL] Fix for btrfs/070 checksum error

2015-07-28 Thread Qu Wenruo
Chris Mason wrote on 2015/07/23 21:57 -0400: On Fri, Jul 24, 2015 at 08:29:05AM +0800, Qu Wenruo wrote: [ deadlock with the 070 patches ] Thanks Chris We will investigate it with highest priority. Thanks, Qu Thanks! I'm doing a few more runs to make sure the lockup is new with these

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Qu Wenruo
Oh, there seems to be something wrong with the internal mail server. The codes and patches can also get from github, as only the first 4 patches are successfully sent... https://github.com/adam900710/linux/tree/dedup Thanks, Qu Qu Wenruo wrote on 2015/07/28 16:30 +0800: Although Liu Bo has

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Liu Bo
On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. [[CORE FEATURES]] The main design concept is the following: 1) Controllable memory usage 2) No guarantee to dedup

[PATCH] btrfs-progs: Increase running state's priority in stat output

2015-07-28 Thread Zhaolei
From: Zhao Lei zhao...@cn.fujitsu.com Anthony Plack anth...@plack.net reported a output bug in maillist: title: btrfs-progs SCRUB reporting aborted but still running - minor btrfs scrub status report it was aborted but still runs to completion. # btrfs scrub status /mnt/data scrub status

Re: [PATCH] Btrfs: fix warning in backref walking

2015-07-28 Thread Liu Bo
On Mon, Jul 27, 2015 at 10:03:44AM +0100, Filipe David Manana wrote: On Mon, Jul 27, 2015 at 9:15 AM, Liu Bo bo.li@oracle.com wrote: When we do backref walking, we search firstly in queued delayed refs and then the on-disk backrefs, but we parse differently for shared references, for

[PATCH RFC 02/14] btrfs: Use btrfs_fill_file_extent to reduce duplicated codes

2015-07-28 Thread Qu Wenruo
Use btrfs_fill_file_extent() function to replace the hand-coded codes. As it has better check and takes less effort to maintain. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/file-item.c | 20 fs/btrfs/inode.c | 38 ++

[PATCH RFC 04/14] btrfs: dedup: Add internal add/remove/search function for btrfs dedup.

2015-07-28 Thread Qu Wenruo
Add basic internal add/remove/search functions for btrfs_dedup. They are all internal use only as caller shouldn't call this low level functions Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/dedup.c | 169 ++- 1 file changed,

[PATCH RFC 03/14] btrfs: dedup: Add basic init/free functions for inband dedup.

2015-07-28 Thread Qu Wenruo
Add basic data structures and their init/free functions for later inband dedup implment. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/Makefile | 2 +- fs/btrfs/ctree.h | 5 fs/btrfs/dedup.c | 88 ++ fs/btrfs/dedup.h

[PATCH RFC 03/14] btrfs: dedup: Add basic init/free functions for inband dedup.

2015-07-28 Thread Qu Wenruo
Add basic data structures and their init/free functions for later inband dedup implment. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/Makefile | 2 +- fs/btrfs/ctree.h | 5 fs/btrfs/dedup.c | 88 ++ fs/btrfs/dedup.h

[PATCH RFC 02/14] btrfs: Use btrfs_fill_file_extent to reduce duplicated codes

2015-07-28 Thread Qu Wenruo
Use btrfs_fill_file_extent() function to replace the hand-coded codes. As it has better check and takes less effort to maintain. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/file-item.c | 20 fs/btrfs/inode.c | 38 ++

[PATCH RFC 04/14] btrfs: dedup: Add internal add/remove/search function for btrfs dedup.

2015-07-28 Thread Qu Wenruo
Add basic internal add/remove/search functions for btrfs_dedup. They are all internal use only as caller shouldn't call this low level functions Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/dedup.c | 169 ++- 1 file changed,

[PATCH RFC 01/14] btrfs: file-item: Introduce btrfs_setup_file_extent function.

2015-07-28 Thread Qu Wenruo
This new function is just used to fill the file extent with given numbers. This is mainly used for later cleanup of duplicated file extent setup codes in inode.c, but also centralized the safety check for later expansion. Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- fs/btrfs/ctree.h

[PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Qu Wenruo
Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. [[CORE FEATURES]] The main design concept is the following: 1) Controllable memory usage 2) No guarantee to dedup every duplication. 3) No on-disk format change or new format 4)

mkfs.btrfs with invalid -d option uses default single drive, rather than errors

2015-07-28 Thread james harvey
Doing some fast-paced benchmarking of lots of raid levels, some in a kvm, some via RDMA access over InfiniBand using different procotols, etc. Was shocked to see horrible raid0 performance in one of the tests. Looked back through the logs and found: (Note that I typo'ed -d raid**e**0 = $

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Liu Bo
On Wed, Jul 29, 2015 at 09:47:12AM +0800, Qu Wenruo wrote: First, thanks David for the review. David Sterba wrote on 2015/07/28 16:50 +0200: On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Qu Wenruo
Liu Bo wrote on 2015/07/28 17:52 +0800: On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. [[CORE FEATURES]] The main design concept is the following: 1) Controllable

[PATCH] btrfs-progs: Add missing exit for parse_profile function

2015-07-28 Thread Qu Wenruo
In parse_profile() function, in error handling route, it output error message but forgot to exit(1), causing even profile is not valid, it will just fallback to single. Reported-by: James Harvey jamespharve...@gmail.com Signed-off-by: Qu Wenruo quwen...@cn.fujitsu.com --- mkfs.c | 1 + 1 file

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Chris Mason
On Tue, Jul 28, 2015 at 04:50:21PM +0200, David Sterba wrote: On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. What's the reason to start another implementation?

Re: Error removing an HDD from a RAID1 BTRFS array

2015-07-28 Thread Chris Murphy
On Tue, Jul 28, 2015 at 4:09 PM, Alejandro Recarey a...@recarey.org wrote: # ./btrfs device delete /dev/sda /storage ERROR: error removing the device '/dev/sda' - No space left on device # uname -a Linux nerd-server 3.13.0-48-generic \ I can't even begin to tell you how much has

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread Qu Wenruo
First, thanks David for the review. David Sterba wrote on 2015/07/28 16:50 +0200: On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. What's the reason to start

Re: [PATCH] block: add a bi_error field to struct bio

2015-07-28 Thread Jens Axboe
On 07/28/2015 05:12 AM, Christoph Hellwig wrote: On Fri, Jul 24, 2015 at 10:36:45AM -0600, Jens Axboe wrote: Right, I don't think we need to do that though. If you look at the flags usage, it's all over the map. Some use test/set_bit, some set it just by OR'ing the mask. There's no reason we

Re: mkfs.btrfs with invalid -d option uses default single drive, rather than errors

2015-07-28 Thread Chris Murphy
On Tue, Jul 28, 2015 at 8:16 PM, james harvey jamespharve...@gmail.com wrote: Doing some fast-paced benchmarking of lots of raid levels, some in a kvm, some via RDMA access over InfiniBand using different procotols, etc. Was shocked to see horrible raid0 performance in one of the tests.

Re: [PATCH RFC 00/14] Yet Another In-band(online) deduplication implement

2015-07-28 Thread David Sterba
On Tue, Jul 28, 2015 at 04:30:36PM +0800, Qu Wenruo wrote: Although Liu Bo has already submitted a V10 version of his deduplication implement, here is another implement for it. What's the reason to start another implementation? [[CORE FEATURES]] The main design concept is the following: 1)

Re: [PATCH] block: add a bi_error field to struct bio

2015-07-28 Thread Christoph Hellwig
On Fri, Jul 24, 2015 at 10:36:45AM -0600, Jens Axboe wrote: Right, I don't think we need to do that though. If you look at the flags usage, it's all over the map. Some use test/set_bit, some set it just by OR'ing the mask. There's no reason we can't make this work without relying on

Re: [PATCH v2 10/10] btrfs-progs: Allow open_ctree use backup tree root or search it automatically if primary tree root is corrupted.

2015-07-28 Thread David Sterba
On Tue, Jul 28, 2015 at 08:34:50AM +0800, Qu Wenruo wrote: Yes, you were against the automatic use of backup root or especially iteration all metadata space to find the latest tree root. I'll try to add a new option like --full-scan to enable the automatic search of all metadata space.

Re: mkfs.btrfs with invalid -d option uses default single drive, rather than errors

2015-07-28 Thread Qu Wenruo
Chris Murphy wrote on 2015/07/28 21:14 -0600: On Tue, Jul 28, 2015 at 8:16 PM, james harvey jamespharve...@gmail.com wrote: Doing some fast-paced benchmarking of lots of raid levels, some in a kvm, some via RDMA access over InfiniBand using different procotols, etc. Was shocked to see

mount btrfs takes 30 minutes, btrfs check runs out of memory

2015-07-28 Thread Georgi Georgiev
Using BTRFS on a very large filesystem, and as we put and more data to it, the time it takes to mount it grew to, presently, about 30 minutes. Is there something wrong with the filesystem? Is there a way to bring this time down? ... Here is a snippet from dmesg, showing how long it takes to

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

2015-07-28 Thread Omar Sandoval
On Tue, Jul 28, 2015 at 06:52:07PM +0200, David Sterba wrote: On Tue, Jul 28, 2015 at 12:22:56AM +0200, Brendan Hide wrote: It does not, I apparently forgot that you could use single to concatenate multiple devices. I'll fix that in v2. Thanks for reviewing! Late to the party.

Error removing an HDD from a RAID1 BTRFS array

2015-07-28 Thread Alejandro Recarey
Hi all, I setup a btrfs filesystem on 3 disks with raid 1. Somehow, a small portion (2GiB) got used as RAID0. At the time, no amount of running balance -dconvert raid1 would change it so I left it as is. Lately /dev/sda is starting to show SMART errors and general issues. So I added 2 other