Re: [PATCH v2] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Michael Kerrisk (man-pages)
On 11/23/2016 05:49 AM, Omar Sandoval wrote: > On Tue, Nov 22, 2016 at 08:48:16PM -0800, Darrick J. Wong wrote: >> Fix the discussion of the limitations on the dest_count and src_length >> parameters to the fideduperange ioctl to reflect what's actually in the >> kernel. >> >> Signed-off-by: Darric

Re: [PATCH v2] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Michael Kerrisk (man-pages)
On 11/23/2016 05:48 AM, Darrick J. Wong wrote: > Fix the discussion of the limitations on the dest_count and src_length > parameters to the fideduperange ioctl to reflect what's actually in the > kernel. Thanks Darrick. Applied. Cheers, Michael > Signed-off-by: Darrick J. Wong > --- > man2/i

Re: Inconsistent free space with false ENOSPC

2016-11-22 Thread Duncan
Martin Raiber posted on Tue, 22 Nov 2016 17:43:46 + as excerpted: > On 22.11.2016 15:16 Martin Raiber wrote: >> ... >> Interestingly, >> after running "btrfs check --repair" "df" shows 0 free space (Used >> 516456408 Available 0), being inconsistent with the below other btrfs >> free space inf

Re: [PATCH v2] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Omar Sandoval
On Tue, Nov 22, 2016 at 08:48:16PM -0800, Darrick J. Wong wrote: > Fix the discussion of the limitations on the dest_count and src_length > parameters to the fideduperange ioctl to reflect what's actually in the > kernel. > > Signed-off-by: Darrick J. Wong > --- > man2/ioctl_fideduperange.2 |

[PATCH v2] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Darrick J. Wong
Fix the discussion of the limitations on the dest_count and src_length parameters to the fideduperange ioctl to reflect what's actually in the kernel. Signed-off-by: Darrick J. Wong --- man2/ioctl_fideduperange.2 | 20 +--- 1 file changed, 17 insertions(+), 3 deletions(-) diff

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Omar Sandoval
On Wed, Nov 23, 2016 at 12:32:26PM +0800, Anand Jain wrote: > > > On 11/23/16 12:21, Omar Sandoval wrote: > > On Wed, Nov 23, 2016 at 12:21:41PM +0800, Anand Jain wrote: > > > > > > > > > > > Can anyone help me on how to get test coverage for the compression > > > > > code? > > > > > > > > I'm

Re: [PATCH] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Darrick J. Wong
On Tue, Nov 22, 2016 at 08:39:18PM -0800, Omar Sandoval wrote: > On Tue, Nov 22, 2016 at 08:22:22PM -0800, Darrick J. Wong wrote: > > Fix the discussion of the limitations on the dest_count and src_length > > parameters to the fideduperange ioctl to reflect what's actually in the > > kernel. > > >

Re: [PATCH] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Omar Sandoval
On Tue, Nov 22, 2016 at 08:22:22PM -0800, Darrick J. Wong wrote: > Fix the discussion of the limitations on the dest_count and src_length > parameters to the fideduperange ioctl to reflect what's actually in the > kernel. > > Signed-off-by: Darrick J. Wong > --- > man2/ioctl_fideduperange.2 |

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Anand Jain
On 11/23/16 12:21, Omar Sandoval wrote: On Wed, Nov 23, 2016 at 12:21:41PM +0800, Anand Jain wrote: Can anyone help me on how to get test coverage for the compression code? I'm not surprised xfstests missed this one since it's just readahead. You might be able to get better coverage with

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-22 Thread Dave Chinner
On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the > >implicit EOF gets around this in the existing XFS implementation. I > >copied this for the B

[PATCH] fideduperange.2: fix the discussion of maximum sizes

2016-11-22 Thread Darrick J. Wong
Fix the discussion of the limitations on the dest_count and src_length parameters to the fideduperange ioctl to reflect what's actually in the kernel. Signed-off-by: Darrick J. Wong --- man2/ioctl_fideduperange.2 | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Omar Sandoval
On Wed, Nov 23, 2016 at 12:21:41PM +0800, Anand Jain wrote: > > > > > Can anyone help me on how to get test coverage for the compression > > > code? > > > > I'm not surprised xfstests missed this one since it's just readahead. > > You might be able to get better coverage with > > > > export MOU

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Anand Jain
Can anyone help me on how to get test coverage for the compression code? I'm not surprised xfstests missed this one since it's just readahead. You might be able to get better coverage with export MOUNT_OPTS="-o compress-force" And where the data is /dev/urandom the btrfs compression will

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

2016-11-22 Thread Anand Jain
On 11/22/16 21:16, David Sterba wrote: On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote: On 11/14/16 20:13, David Sterba wrote: On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote: This patch isn't integrated, any idea why ? Because it does not cover all usecases.

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-22 Thread Darrick J. Wong
On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the > >implicit EOF gets around this in the existing XFS implementation. I > >copied this for the B

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-22 Thread Zygo Blaxell
On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the >implicit EOF gets around this in the existing XFS implementation. I >copied this for the Btrfs implementation. Somewhat tangential to this patch, but on the de

Re: [PATCH] btrfs: raid56: Use correct stolen pages to calculate P/Q

2016-11-22 Thread Qu Wenruo
At 11/23/2016 02:58 AM, Chris Mason wrote: On 11/21/2016 03:50 AM, Qu Wenruo wrote: In the following situation, scrub will calculate wrong parity to overwrite correct one: RAID5 full stripe: Before | Dev 1 | Dev 2 | Dev 3 | | Data stripe 1 | Data stripe 2 | Parity

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-22 Thread Darrick J. Wong
On Thu, Nov 17, 2016 at 09:38:53PM -0800, Christoph Hellwig wrote: > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > So now we have 3 options: > > > > a) Apply these patches as-is. > > b) Fix XFS to both return the actual bytes_deduped and cap the length > >for the EOF case.

Re: [PATCH 1/9] btrfs: use bio iterators for the decompression handlers

2016-11-22 Thread Liu Bo
On Tue, Nov 22, 2016 at 01:38:46AM -0800, Christoph Hellwig wrote: > On Fri, Nov 18, 2016 at 05:29:06PM -0800, Liu Bo wrote: > > On Wed, Nov 16, 2016 at 01:52:08PM +0100, Christoph Hellwig wrote: > > > Pass the full bio to the decompression routines and use bio iterators > > > to iterate over the d

Re: [PATCH] Btrfs: fix truncate down when no_holes feature is enabled

2016-11-22 Thread Liu Bo
On Tue, Nov 22, 2016 at 02:13:21PM -0500, Chris Mason wrote: > On 11/11/2016 05:27 PM, Liu Bo wrote: > > For such a file mapping, > > > > [0-4k][hole][8k-12k] > > > > In NO_HOLES mode, we don't have the [hole] extent any more. > > Commit c1aa45759e90 ("Btrfs: fix shrinking truncate when the no_ho

Re: [PATCH] Btrfs: adjust len of writes if following a preallocated extent

2016-11-22 Thread Chris Mason
On 11/04/2016 03:20 PM, Liu Bo wrote: If we have |0--hole--4095||4096--preallocate--12287| instead of using preallocated space, a 8K direct write will just create a new 8K extent and it'll end up with |0--new extent--8191||8192--preallocate--12287| It's because we find a hole em and then go t

Re: [PATCH v2] btrfs: introduce priority based delalloc shrink mechanism

2016-11-22 Thread Chris Mason
On 10/24/2016 11:43 AM, David Sterba wrote: Hi Josef, are you fine with V2? On Thu, Oct 13, 2016 at 05:31:25PM +0800, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create a

Re: [PATCH] Btrfs: fix truncate down when no_holes feature is enabled

2016-11-22 Thread Chris Mason
On 11/11/2016 05:27 PM, Liu Bo wrote: For such a file mapping, [0-4k][hole][8k-12k] In NO_HOLES mode, we don't have the [hole] extent any more. Commit c1aa45759e90 ("Btrfs: fix shrinking truncate when the no_holes feature is enabled") fixed disk isize not being updated in NO_HOLES mode when d

Re: btrfs-progs-4.8.3 libbtrfs missing symbols?

2016-11-22 Thread Mike Gilbert
On Tue, Nov 22, 2016 at 1:49 PM, Mike Gilbert wrote: > Hi, > > I received a bug report about a build failure in a package (snapper) > that utilizes libbtrfs. > > https://bugs.gentoo.org/show_bug.cgi?id=600078 > > To summarize the issue, the package has a configure test that executes > the followin

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Omar Sandoval
On Tue, Nov 22, 2016 at 10:42:40AM +0100, Christoph Hellwig wrote: > On Fri, Nov 18, 2016 at 12:04:38PM -0800, Omar Sandoval wrote: > > > +static u64 bio_end_offset(struct bio *bio) > > > +{ > > > + struct bio_vec *last = &bio->bi_io_vec[bio->bi_vcnt - 1]; > > > + > > > + return page_offset(last->b

Re: [PATCH] btrfs: raid56: Use correct stolen pages to calculate P/Q

2016-11-22 Thread Chris Mason
On 11/21/2016 03:50 AM, Qu Wenruo wrote: In the following situation, scrub will calculate wrong parity to overwrite correct one: RAID5 full stripe: Before | Dev 1 | Dev 2 | Dev 3 | | Data stripe 1 | Data stripe 2 | Parity Stripe |

btrfs-progs-4.8.3 libbtrfs missing symbols?

2016-11-22 Thread Mike Gilbert
Hi, I received a bug report about a build failure in a package (snapper) that utilizes libbtrfs. https://bugs.gentoo.org/show_bug.cgi?id=600078 To summarize the issue, the package has a configure test that executes the following: configure:16618: x86_64-pc-linux-gnu-gcc -o conftest -O2 -pipe -m

Re: [PATCH] btrfs: raid56: Use correct stolen pages to calculate P/Q

2016-11-22 Thread Goffredo Baroncelli
On 2016-11-22 01:28, Qu Wenruo wrote: > > > At 11/22/2016 02:48 AM, Goffredo Baroncelli wrote: >> Hi Qu, >> >> I tested this succefully for RAID5 when doing a scrub (i.e.: I mount a >> corrupted disks, then I ran "btrfs scrub start ...", then I check the disks). >> >> However if I do a "cat mnt/

Re: Inconsistent free space with false ENOSPC

2016-11-22 Thread Martin Raiber
On 22.11.2016 15:16 Martin Raiber wrote: > ... > Interestingly, > after running "btrfs check --repair" "df" shows 0 free space (Used > 516456408 Available 0), being inconsistent with the below other btrfs > free space information. > > btrfs fi usage output: > > Overall: > Device size:

Inconsistent free space with false ENOSPC

2016-11-22 Thread Martin Raiber
Hi, I'm having a file system which is currently broken because of ENOSPC issues. It is a single device file system with no compression and no quotas enabled but with some snapshots. Creation and initial ENOSPC/free space inconsistency with 4.4.20 and 4.4.30 (both vanilla). Currently I am on 4.9.0

Re: [Bug 186671] New: OOM on system with just rsync running 32GB of ram 30GB of pagecache

2016-11-22 Thread Vlastimil Babka
On 11/22/2016 02:58 PM, E V wrote: > System OOM'd several times last night with 4.8.10, I attached the > page_owner output from a morning cat ~8 hours after OOM's to the > bugzilla case, split and compressed to fit under the 5M attachment > limit. Let me know if you need anything else. Looks like

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread Chris Mason
On 11/22/2016 04:20 AM, Wang Xiaoguang wrote: hello, On 11/22/2016 12:39 AM, David Sterba wrote: On Tue, Nov 08, 2016 at 05:30:58PM +0800, Wang Xiaoguang wrote: The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big v

Re: [PATCH 00/19] Offline scrub support

2016-11-22 Thread David Sterba
On Tue, Nov 22, 2016 at 09:09:18AM +0800, Qu Wenruo wrote: > Hi David, > > Any comment on this patchset? I haven't looked at the patchset yet, there's enough unfinished work in progs and I'm not familiar with the radi56 code to work on that in parallel. -- To unsubscribe from this list: send the

Re: [Bug 186671] New: OOM on system with just rsync running 32GB of ram 30GB of pagecache

2016-11-22 Thread E V
System OOM'd several times last night with 4.8.10, I attached the page_owner output from a morning cat ~8 hours after OOM's to the bugzilla case, split and compressed to fit under the 5M attachment limit. Let me know if you need anything else. On Fri, Nov 18, 2016 at 10:02 AM, E V wrote: > Yes, t

[CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

2016-11-22 Thread bepi
Hi. My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64 Testing the remote differential receive (via ssh and in local network) of 24 sequential snapshots, and simultaneously deleting the snapshot, (in the same file system, but in a different subvolume), there has

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

2016-11-22 Thread David Sterba
On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote: > On 11/14/16 20:13, David Sterba wrote: > > On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote: > >> This patch isn't integrated, any idea why ? > > > > Because it does not cover all usecases. > > You have stated one of the

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread David Sterba
On Tue, Nov 22, 2016 at 05:20:10PM +0800, Wang Xiaoguang wrote: > hello, > > On 11/22/2016 12:39 AM, David Sterba wrote: > > On Tue, Nov 08, 2016 at 05:30:58PM +0800, Wang Xiaoguang wrote: > >> The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). > >> For 4K page, the total

Re: [PATCH 0/3] introduce type based delalloc metadata reserve to fix some false enospc issues

2016-11-22 Thread Wang Xiaoguang
hello, Any comments about this patchset? The first two patches almost don't do any functional change which I think could be review firstly. btrfs: improve inode's outstanding_extents computation btrfs: introduce type based delalloc metadata reserve Regards, Xiaoguang Wang On 11/11/201

Re: [PATCH 6/9] btrfs: calculate end of bio offset properly

2016-11-22 Thread Christoph Hellwig
On Fri, Nov 18, 2016 at 12:04:38PM -0800, Omar Sandoval wrote: > > +static u64 bio_end_offset(struct bio *bio) > > +{ > > + struct bio_vec *last = &bio->bi_io_vec[bio->bi_vcnt - 1]; > > + > > + return page_offset(last->bv_page) + last->bv_len - last->bv_offset; > > Why is this minus bv_offset

Re: [PATCH 1/9] btrfs: use bio iterators for the decompression handlers

2016-11-22 Thread Christoph Hellwig
On Fri, Nov 18, 2016 at 05:29:06PM -0800, Liu Bo wrote: > On Wed, Nov 16, 2016 at 01:52:08PM +0100, Christoph Hellwig wrote: > > Pass the full bio to the decompression routines and use bio iterators > > to iterate over the data in the bio. > > One question below, It would be nice to cut down the

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread Wang Xiaoguang
hello, On 11/22/2016 01:59 AM, Chris Mason wrote: On 11/08/2016 04:30 AM, Wang Xiaoguang wrote: The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big value, I think in most cases, this limit will not work, here I se

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread Wang Xiaoguang
hello, On 11/22/2016 12:39 AM, David Sterba wrote: On Tue, Nov 08, 2016 at 05:30:58PM +0800, Wang Xiaoguang wrote: The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big value, I think in most cases, this limit will no

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

2016-11-22 Thread Anand Jain
(sorry for the delay due to my vacation). On 11/14/16 20:13, David Sterba wrote: On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote: This patch isn't integrated, any idea why ? Because it does not cover all usecases. David, You have stated one of the use case here https

[RFC PATCH 2/3] fstests: common/ondisk.btrfs: Introduce function to get btrfs ondisk info

2016-11-22 Thread Qu Wenruo
Introduce a new common header, ondisk.btrfs, to allow we manipulate btrfs on-disk data directly, to corrupt or verify data. Since the designed tool, btrfs-corrupt-block is no longer default installed, and furthermore it doesn't support to corrupt given mirror or stripe, it's near useless for later

[RFC PATCH 3/3] fstests: btrfs: Add new test case to check scrub recovery and report

2016-11-22 Thread Qu Wenruo
In fact, desipte the existing btrfs scrub test cases, we didn't even test if scrub can really recovery data. Due to the recent exposed several critical RAID56 scrub problem, it's really needed to test the fundamental function before things get worse and worse. This test case add verification for

[RFC PATCH 1/3] fstests: common: rename _require_btrfs to _require_btrfs_subcommand

2016-11-22 Thread Qu Wenruo
Rename _require_btrfs() to _require_btrfs_subcommand() to avoid confusion, as all other _require_btrfs_* has a quite clear suffix, like _require_btrfs_mkfs_feature() or _require_btrfs_fs_feature(). Signed-off-by: Qu Wenruo --- common/rc | 2 +- tests/btrfs/004 | 2 +- tests/btrfs/048 | 2 +

[RFC PATCH 0/3] Add testcase for fundamental scrub recovery

2016-11-22 Thread Qu Wenruo
Despite the scrub test cases in fstests, there is not even one test case which really checked if scrub can recover data. In fact, btrfs scrub for RAID56 will even corrupt correct data stripes. So let's start from the needed facilities and check scrub starting from RAID1. The main reason the patc

Re: [RFC] btrfs: make max inline data can be equal to sectorsize

2016-11-22 Thread Wang Xiaoguang
hello, On 11/19/2016 04:58 AM, Chris Mason wrote: On 11/16/2016 11:10 AM, David Sterba wrote: On Mon, Nov 14, 2016 at 09:55:34AM +0800, Qu Wenruo wrote: At 11/12/2016 04:22 AM, Liu Bo wrote: On Tue, Oct 11, 2016 at 02:47:42PM +0800, Wang Xiaoguang wrote: If we use mount option "-o max_inli