Deduplication and redundancy questions

2015-10-02 Thread -
hey everybody. 1. Deduplication "When given a list of files it will hash their contents on a block by block basis" - are those static blocks or is the length of a block defined by its content? (that would be more resilient regarding inserts of data and the shift of the following data caused by

Re: [PATCH 1/1] Btrfs: consolidate btrfs_error() to btrfs_std_error()

2015-10-02 Thread Anand Jain
On 09/25/2015 06:31 PM, David Sterba wrote: On Fri, Sep 25, 2015 at 02:43:01PM +0800, Anand Jain wrote: btrfs_error() and btrfs_std_error() does the same thing and calls _btrfs_std_error(), so consolidate them together. And the main motivation is that btrfs_error() is closely named with btrfs_

Re: [PATCH 00/23] btrfs device related patch set

2015-10-02 Thread Anand Jain
On 10/02/2015 12:17 AM, David Sterba wrote: On Thu, Oct 01, 2015 at 11:00:39PM +0800, Anand Jain wrote: Its been challenging to realign the patch sequence at my end as well, I am trying. I think it will take a bit longer than expected. I'd suggest to send several small patchsets instead o

Re: [PATCH 1/1] Btrfs: consolidate btrfs_error() to btrfs_std_error()

2015-10-02 Thread Qu Wenruo
在 2015年10月02日 15:41, Anand Jain 写道: On 09/25/2015 06:31 PM, David Sterba wrote: On Fri, Sep 25, 2015 at 02:43:01PM +0800, Anand Jain wrote: btrfs_error() and btrfs_std_error() does the same thing and calls _btrfs_std_error(), so consolidate them together. And the main motivation is that btr

Re: [PATCH v2] fstests: generic: Check if a bull fallocate will change extent number

2015-10-02 Thread Qu Wenruo
Hi Dave, I updated the patch and moved it to btrfs. But I still has some question about the fallocate behavior. Just as the new btrfs test case, I changed the fallocate range, not to cover the last part, to make the problem more obvious: Btrfs will truncate beyond EOF even that's *not covered

Re: Btrfs check dont repair

2015-10-02 Thread Vackář František
I'm not sure if it was my last email with attachement delivered. My last email is below. Attachement is uploaded here: http://franta.naobzoru.cz/files/btrfs-failed.txt Thank you. Frantisek 2015-09-27 11:51 GMT+02:00 Vackář František : > So I copy with dd all my 1.7 TB partition with broken btrf

Re: Deduplication and redundancy questions

2015-10-02 Thread Hugo Mills
On Fri, Oct 02, 2015 at 09:06:17AM +0200, - wrote: > hey everybody. > > 1. Deduplication > "When given a list of files it will hash their contents on a block > by block basis" - are those static blocks or is the length of a > block defined by its content? (that would be more resilient > regarding

Re: [PATCH 1/1] Btrfs: consolidate btrfs_error() to btrfs_std_error()

2015-10-02 Thread Anand Jain
It only needs to log an error, no need to mark FS_ERROR bit. As it may just be an invalid parameter. This ans-ed my question. Then there are two bugs. one which David pointed out. two In original code (before this patch), it should have been calling btrfs_err() instead of btrfs_error(). T

Re: Btrfs check dont repair

2015-10-02 Thread Vackář František
So... I find a solution! :) I tried to play with mount options. Finally I sucessfully mounted corrupted btrfs with mount -t btrfs -oro,degraded,recovery /dev/sdb2 /mnt/hdd-root/ Thank you for help. Now I copy all data to external drive and make new filestystem. Frantisek 2015-10-02 11:05 GM

[PATCH] Btrfs: send, fix file corruption due to incorrect cloning operations

2015-10-02 Thread fdmanana
From: Filipe Manana If we have a file that shares an extent with other files, when processing the extent item relative to a shared extent, we blindly issue a clone operation that will target a length matching the length in the extent item and uses as a source some other file the receiver already

[PATCH] fstests: test btrfs send for reflinked files

2015-10-02 Thread fdmanana
From: Filipe Manana Test that a send operation works correctly with reflinked files (cloned extents which multiple files point to). The btrfs issue was fixed by the linux kernel patch titled: "Btrfs: send, fix file corruption due to incorrect cloning operations" Signed-off-by: Filipe Manana --

[PATCH] Btrfs: pass the error code to the btrfs_std_error and log ret

2015-10-02 Thread Anand Jain
This patch will log return value of add/del_qgroup_relation() and pass the err code of btrfs_run_qgroups to the btrfs_std_error(). Signed-off-by: Anand Jain --- fs/btrfs/ioctl.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index f7

Re: [PATCH 1/1] Btrfs: consolidate btrfs_error() to btrfs_std_error()

2015-10-02 Thread Anand Jain
> btrfs_std_error() naming is quite confusing for me. > > Std_error() means more like stderr, for my first glance, I'd think > it's just a new printk() warpper, until I checked the code. > It does more than printk, but also set FS_STATE_ERROR bit and set fs > to readonly. > I'd like it to be so

[PATCH v2] Btrfs: pass the error code to the btrfs_std_error and log ret

2015-10-02 Thread Anand Jain
This patch will log return value of add/del_qgroup_relation() and pass the err code of btrfs_run_qgroups to the btrfs_std_error(). Signed-off-by: Anand Jain --- v2: fix the forgotten git commit amend, to take in compile fail, sorry fs/btrfs/ioctl.c | 7 ++- 1 file changed, 6 insertions(+),

Re: [PATCH v4 0/9] Btrfs: free space B-tree

2015-10-02 Thread Austin S Hemmelgarn
On 2015-09-29 23:50, Omar Sandoval wrote: Hi, Here's one more reroll of the free space B-tree patches, a more scalable alternative to the free space cache. Minimal changes this time around, I mainly wanted to resend this after Holger and I cleared up his bug report here: http://www.spinics.net/l

Re: BTRFS as image store for KVM?

2015-10-02 Thread Austin S Hemmelgarn
On 2015-10-02 00:21, Russell Coker wrote: On Sat, 26 Sep 2015 12:20:41 AM Austin S Hemmelgarn wrote: FYI: Linux pagecache use LRU cache algo, and in general case it's working good enough I'd argue that 'general usage' should be better defined in this statement. Obviously, ZFS's ARC implementa

Re: [PATCH V5 02/13] Btrfs: Compute and look up csums based on sectorsized blocks

2015-10-02 Thread Chandan Rajendra
On Thursday 01 Oct 2015 10:39:29 Josef Bacik wrote: > On 09/30/2015 06:28 AM, Chandan Rajendra wrote: > > Checksums are applicable to sectorsize units. The current code uses > > bio->bv_len units to compute and look up checksums. This works on machines > > where sectorsize == PAGE_SIZE. This patch

Re: [PATCH V5 12/13] Btrfs: prepare_pages: Retry adding a page to the page cache

2015-10-02 Thread Chandan Rajendra
On Thursday 01 Oct 2015 10:50:30 Josef Bacik wrote: > On 09/30/2015 06:28 AM, Chandan Rajendra wrote: > > When reading the page from the disk, we can race with Direct I/O which can > > get the page lock (before prepare_uptodate_page() gets it) and can go > > ahead and invalidate the page. Hence if

Re: [PATCH v4 6/9] Btrfs: implement the free space B-tree

2015-10-02 Thread kbuild test robot
Hi Omar, [auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore] config: mips-ip27_defconfig (attached as .config) reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/

[PATCH] btrfs-progs: fix error checking in load_device_info

2015-10-02 Thread David Sterba
load_device_info queries the FS_INFO ioctl and this may fail with EPERM on older kernels. The check did not verify the ioctl return value and incorrectly returned EPERM if it was previously stored in errno. This fixes 'btrfs fi usage' that will print the overall summary for all users (provided tha

[PATCH] btrfs-progs: fix double free during scanning

2015-10-02 Thread David Sterba
If there are different devices mounted to the same directory we can run into double free issue in the scanning code and this can lead to a crash. The dev_info_arg buffer allocation get_fs_info might be skipped, eg. if the FS_INFO ioctl fails due to EPERM in older kernels. Reset the pointer before e

Re: [PATCH V5 11/13] Btrfs: Clean pte corresponding to page straddling i_size

2015-10-02 Thread Chandan Rajendra
On Thursday 01 Oct 2015 10:57:52 Josef Bacik wrote: > On 09/30/2015 06:28 AM, Chandan Rajendra wrote: > > When extending a file by either "truncate up" or by writing beyond i_size, > > the page which had i_size needs to be marked "read only" so that future > > writes to the page via mmap interface

[PATCH 4/4] btrfs-progs: change -t option for subvolume list to print a simple space-separated table (making it machine-readable)

2015-10-02 Thread axel
Old implementation used tabs "\t", and tried to work around problems by guessing amount of tabs needed (e.g. "\t\t" after top level", with buggy output as soon as empty uuids are printed). This will never work correctly, as tab width is a user-defined setting in the terminal. Keep it simple and do

[PATCH 2/4] btrfs-progs: add "flags" column for subvolume list (shows "readonly" flag with -A)

2015-10-02 Thread axel
Signed-off-by: Axel Burri --- btrfs-list.c | 14 ++ btrfs-list.h | 1 + 2 files changed, 15 insertions(+) diff --git a/btrfs-list.c b/btrfs-list.c index 7529e11..ff337f9 100644 --- a/btrfs-list.c +++ b/btrfs-list.c @@ -80,6 +80,11 @@ static struct { .need_print =

[PATCH 3/4] btrfs-progs: add option "--time-format=short|iso|unix|locale" to subvolume list

2015-10-02 Thread axel
Affects time format of "otime". Supporting unix time and ISO8601 makes the output of "subvolume list" machine-readable. Default is "short", keeping default output as it was before (I suggest changing the default to "iso", as with "short" is it not clear to the user if localtime or gmtime is printe

[PATCH 0/4] btrfs-progs: improve output of btrfs subvolume list command

2015-10-02 Thread axel
Improvements to "subvolume list" command, with the goal to improve machine-readability. While the first three patches simply add new options (-A and --time-format), the last patch [PATCH 4/4] is more intrusive and changes the output format of the '-t' option. Consider it "my view of how tabular out

[PATCH 1/4] btrfs-progs: add -A option for subvolume list (print all available information)

2015-10-02 Thread axel
Signed-off-by: Axel Burri --- Documentation/btrfs-subvolume.asciidoc | 2 ++ cmds-subvolume.c | 6 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/Documentation/btrfs-subvolume.asciidoc b/Documentation/btrfs-subvolume.asciidoc index c187fd8..afbec83 100

[PATCH] Btrfs: fix deadlock when finalizing block group creation

2015-10-02 Thread fdmanana
From: Filipe Manana Josef ran into a deadlock while a transaction handle was finalizing the creation of its block groups, which produced the following trace: [260445.593112] fio D 88022a9df468 0 8924 4518 0x0084 [260445.593119] 88022a9df468 81c134c0 f

Re: [PATCH] Btrfs: fix deadlock when finalizing block group creation

2015-10-02 Thread Josef Bacik
On 10/02/2015 01:43 PM, fdman...@kernel.org wrote: From: Filipe Manana Josef ran into a deadlock while a transaction handle was finalizing the creation of its block groups, which produced the following trace: [260445.593112] fio D 88022a9df468 0 8924 4518 0x0084

btrfs: user space tool to force compression

2015-10-02 Thread Matwey V. Kornilov
Hello, Is there a way (and appropriate user space tool) to force compression of existing file? I mean there was directory with some content and then I set compression attribute (property) to it. How to force data recompression? -- To unsubscribe from this list: send the line "unsubscribe linux-bt

Re: btrfs: user space tool to force compression

2015-10-02 Thread Hugo Mills
On Fri, Oct 02, 2015 at 11:41:00PM +0300, Matwey V. Kornilov wrote: > Hello, > > Is there a way (and appropriate user space tool) to force compression of > existing file? I mean there was directory with some content and then I > set compression attribute (property) to it. How to force data recompr

raid5 + HDFS

2015-10-02 Thread Jim Dowling
Hi I am interested in combining BtrFS RAID-5 with erasure-coded replication for HDFS. We have an implementation of Reed-Solomon replication for our HDFS distribution called HopsFS (www.hops.io). Some of the nice features of HDFS that make it suitable are: * not many small files * not excessive

Re: raid5 + HDFS

2015-10-02 Thread Martin Tippmann
2015-10-03 0:07 GMT+02:00 Jim Dowling : > Hi Hi, I'm not a btrfs developer but we run HDFS on top of btrfs (mainly due to other use-cases that profit from checksumming data) > I am interested in combining BtrFS RAID-5 with erasure-coded replication for > HDFS. We have an implementation of Reed-So

[josef-btrfs:allocator-fixes 14/15] fs/btrfs/free-space-cache.c:3596:14: error: 'struct btrfs_free_space' has no member named 'max_exten_size'

2015-10-02 Thread kbuild test robot
tree: https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git allocator-fixes head: 5be061b1329c6f3b1926496d44560f924c582deb commit: af8df18b295f2e6ff26d27386a5183c545044e48 [14/15] Btrfs: keep track of largest extent in bitmaps config: i386-randconfig-n0-201539 (attached as .con