Re: [PATCH 0/3] introduce type based delalloc metadata reserve to fix some false enospc issues

2016-11-22 Thread Wang Xiaoguang
/2016 04:39 PM, Wang Xiaoguang wrote: When having compression enabled, Stefan Priebe ofen got enospc errors though fs still has much free space. Qu Wenruo also has submitted a fstests test case which can reproduce this bug steadily, please see url: https://patchwork.kernel.org/patch/9420527 First

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread Wang Xiaoguang
hello, On 11/22/2016 01:59 AM, Chris Mason wrote: On 11/08/2016 04:30 AM, Wang Xiaoguang wrote: The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big value, I think in most cases, this limit will not work, here I

Re: [PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-22 Thread Wang Xiaoguang
hello, On 11/22/2016 12:39 AM, David Sterba wrote: On Tue, Nov 08, 2016 at 05:30:58PM +0800, Wang Xiaoguang wrote: The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big value, I think in most cases, this limit

Re: [RFC] btrfs: make max inline data can be equal to sectorsize

2016-11-22 Thread Wang Xiaoguang
hello, On 11/19/2016 04:58 AM, Chris Mason wrote: On 11/16/2016 11:10 AM, David Sterba wrote: On Mon, Nov 14, 2016 at 09:55:34AM +0800, Qu Wenruo wrote: At 11/12/2016 04:22 AM, Liu Bo wrote: On Tue, Oct 11, 2016 at 02:47:42PM +0800, Wang Xiaoguang wrote: If we use mount option &qu

[PATCH 1/3] btrfs: improve inode's outstanding_extents computation

2016-11-11 Thread Wang Xiaoguang
es() won't need to handle cases cross 2 extents. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/inode.c | 65 ++-- fs/btrfs/ioctl.c | 6 ++ 3 files changed, 62 insertions(+), 11 dele

[PATCH 2/3] btrfs: introduce type based delalloc metadata reserve

2016-11-11 Thread Wang Xiaoguang
types will follow soon. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Signed-off-by: Qu Wenruo <quwen...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 34 ++--- fs/btrfs/extent-tree.c | 50 +-- fs/btrfs/file.c | 22 ++-

[PATCH 3/3] btrfs: Introduce COMPRESS reserve type to fix false enospc for compression

2016-11-11 Thread Wang Xiaoguang
And in Step 5) we reserve correct amount of metadata space. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Signed-off-by: Qu Wenruo <quwen...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/extent-tree.c | 2 ++ fs/btrfs/extent_io.c | 61 ++

[PATCH 0/3] introduce type based delalloc metadata reserve to fix some false enospc issues

2016-11-11 Thread Wang Xiaoguang
ave also run whole fstests multiple times, no regression occurs, thanks. Wang Xiaoguang (3): btrfs: improve inode's outstanding_extents computation btrfs: introduce type based delalloc metadata reserve btrfs: Introduce COMPRESS reserve type to fix false enospc for compression fs/btrfs

[PATCH] btrfs: limit the number of asynchronous delalloc pages to reasonable value

2016-11-08 Thread Wang Xiaoguang
The current limit of number of asynchronous delalloc pages is (10 * SZ_1M). For 4K page, the total ram bytes would be 40G, very big value, I think in most cases, this limit will not work, here I set limit of the number of asynchronous delalloc pages to SZ_1M(4GB ram bytes). Signed-off-by: Wang

[PATCH 2/2] btrfs: increase tickets_id even for failed metadata request

2016-11-07 Thread Wang Xiaoguang
Not functional change, it just makes codes logic more reasonable, then at least tickets_id can reflect the number of metadata requests we already handled. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 6 -- 1 file changed, 4 insertions

[PATCH 1/2] btrfs: add necessary comments about tickets_id

2016-11-07 Thread Wang Xiaoguang
Tickets_id's name may result in some misunderstandings, it just indicates the next ticket will be handled and is not stored per ticket. Fixes: ce12965 ("btrfs: introduce tickets_id to determine whether asynchronous metadata reclaim work makes progress") Signed-off-by: Wang Xiaoguang

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

2016-11-02 Thread Wang Xiaoguang
hi, On 11/02/2016 09:27 AM, Dave Chinner wrote: On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote: In btrfs, sometimes though the number of created files' consumed disk space are not larger than fs's free space, we can still get some ENOSPC error, it may be that btrfs does not try

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

2016-11-02 Thread Wang Xiaoguang
hi Eryu, There has already be a generic/102 doing this test... Thanks for you kindly review and sorry for wasting your time. Regards, Xiaoguang Wang On 11/01/2016 08:26 PM, Eryu Guan wrote: On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote: In btrfs, sometimes though the number

[PATCH v5 2/2] generic: make 17[1-4] work well when btrfs compression is enabled

2016-11-01 Thread Wang Xiaoguang
When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs() in common/rc, which'll keep creating and writing files until enospc error occurs. Note _fill_fs is copied from tests/generic/256, but with some minor modifications. Signed-off-by: Wang

[PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place

2016-11-01 Thread Wang Xiaoguang
/populate, so here I choose to put _require_xfs_io_command "falloc" or "punch" in helper function which really use falloc and fpunch. And xfs/120 uses fpunch, add _require_xfs_io_command "fpunch". Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- com

[PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

2016-11-01 Thread Wang Xiaoguang
error will not always happen even in kernel without my fixing patch). Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- tests/generic/389 | 78 +++

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-11-01 Thread Wang Xiaoguang
hi, I have rebased these 2 patches against Linux 4.9-rc2, sorry for being late. After applying these patches, Stefan does not see any ENOSPC error :) If you have free time, please have a check, thanks. Regards, Xiaoguang Wang On 11/01/2016 06:18 PM, Wang Xiaoguang wrote: When testing btrfs

[PATCH 2/2] btrfs: fix false enospc for compression

2016-11-01 Thread Wang Xiaoguang
through compression path. With this patch, we can fix these false enospc error for compression. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Tested-by: Holger Hoffstätte <hol...@applied-asynchrony.com> Tested-by: Stefan Priebe <s.pri...@profihost.ag> --- fs/btrfs/ctree.h

[PATCH 1/2] btrfs: improve inode's outstanding_extents computation

2016-11-01 Thread Wang Xiaoguang
es() won't need to handle cases cross 2 extents. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Tested-by: Holger Hoffstätte <hol...@applied-asynchrony.com> Tested-by: Stefan Priebe <s.pri...@profihost.ag> --- fs/b

Re: [PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

2016-11-01 Thread Wang Xiaoguang
hi Darrick, Common/populate needs xfs_io supports falloc and fpunch, so I didn't put _fill_fs() in common/populate. Regards, Xiaoguang Wang On 11/01/2016 04:45 PM, Wang Xiaoguang wrote: When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs

[PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

2016-11-01 Thread Wang Xiaoguang
When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs() in common/rc, which'll keep creating and writing files until enospc error occurs. Note _fill_fs is copied from tests/generic/256, but with some minor modifications. Signed-off-by: Wang

Re: [PATCH v3] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-28 Thread Wang Xiaoguang
hi, On 10/27/2016 07:25 PM, Eryu Guan wrote: On Wed, Oct 26, 2016 at 05:52:11PM +0800, Wang Xiaoguang wrote: When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs() in common/rc, which'll keep creating and writing files until enospc error occurs

Re: [PATCH v3] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-28 Thread Wang Xiaoguang
hi, On 10/28/2016 01:13 AM, Darrick J. Wong wrote: On Wed, Oct 26, 2016 at 05:52:11PM +0800, Wang Xiaoguang wrote: When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs() in common/rc, which'll keep creating and writing files until enospc error

[PATCH v3] btrfs: imporve delayed refs iterations

2016-10-26 Thread Wang Xiaoguang
] _raw_spin_unlock_irq For normal files, this patch also gives help, at least we do not need to iterate whole list to found BTRFS_ADD_DELAYED_REF nodes. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Reviewed-by: Liu Bo <bo.li@oracle.com> Tested-by: Holger Hoffstätte <

[PATCH v3] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-26 Thread Wang Xiaoguang
When enabling btrfs compression, original codes can not fill fs correctly, here we introduce _fill_fs() in common/rc, which'll keep creating and writing files until enospc error occurs. Note _fill_fs is copied from tests/generic/256, but with some minor modifications. Signed-off-by: Wang

Re: [RFC] btrfs: make max inline data can be equal to sectorsize

2016-10-26 Thread Wang Xiaoguang
hi, On 10/11/2016 02:47 PM, Wang Xiaoguang wrote: If we use mount option "-o max_inline=sectorsize", say 4096, indeed even for a fresh fs, say nodesize is 16k, we can not make the first 4k data completely inline, I found this conditon causing this issue: !compressed_size &

[PATCH v2] btrfs: cleanup: use already calculated value in btrfs_should_throttle_delayed_refs()

2016-10-26 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 9aa6d2c..3c8f0ec 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-

[PATCH] btrfs: minor codes cleanup

2016-10-25 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- V1: Just one small codes cleanup, if you think it's not appropriate to make a individual patch for it, please ignore it :) --- fs/btrfs/extent-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-25 Thread Wang Xiaoguang
Am 29.09.2016 um 09:27 schrieb Stefan Priebe - Profihost AG: Am 29.09.2016 um 09:13 schrieb Wang Xiaoguang: I found that compress sometime report ENOSPC error even in 4.8-rc8, currently I cannot confirm that as i do not have anough space to test this without compression ;-( But yes i've

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-10-25 Thread Wang Xiaoguang
hi, On 10/19/2016 10:23 PM, David Sterba wrote: On Mon, Oct 17, 2016 at 05:01:46PM +0800, Wang Xiaoguang wrote: [..] int btrfs_set_extent_delalloc(struct inode *inode, u64 start, u64 end, - struct extent_state **cached_state

[PATCH v2] btrfs: imporve delayed refs iterations

2016-10-25 Thread Wang Xiaoguang
] _raw_spin_unlock_irq For normal files, this patch also gives help, at least we do not need to iterate whole list to found BTRFS_ADD_DELAYED_REF nodes. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Reviewed-by: Liu Bo <bo.li@oracle.com> Tested-by: Holger Hoffstätte <

Re: [PATCH] btrfs: imporve delayed refs iterations

2016-10-25 Thread Wang Xiaoguang
hi, On 10/25/2016 03:00 AM, Liu Bo wrote: On Fri, Oct 21, 2016 at 05:05:07PM +0800, Wang Xiaoguang wrote: This issue was found when I tried to delete a heavily reflinked file, when deleting such files, other transaction operation will not have a chance to make progress, for example

[PATCH] btrfs: imporve delayed refs iterations

2016-10-21 Thread Wang Xiaoguang
] _raw_spin_unlock_irq For normal files, this patch also gives help, at least we do not need to iterate whole list to found BTRFS_ADD_DELAYED_REF nodes. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 8 fs

Re: [PATCH] btrfs: pass correct args to btrfs_async_run_delayed_refs()

2016-10-18 Thread Wang Xiaoguang
hi, On 10/18/2016 06:32 PM, Holger Hoffstätte wrote: On Tue, 18 Oct 2016 15:56:13 +0800, Wang Xiaoguang wrote: In btrfs_truncate_inode_items()->btrfs_async_run_delayed_refs(), we swap the arg2 and arg3 wrongly, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com>

[PATCH] btrfs: pass correct args to btrfs_async_run_delayed_refs()

2016-10-18 Thread Wang Xiaoguang
In btrfs_truncate_inode_items()->btrfs_async_run_delayed_refs(), we swap the arg2 and arg3 wrongly, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/inode.c b

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-10-17 Thread Wang Xiaoguang
hi, On 10/14/2016 09:59 PM, Holger Hoffstätte wrote: On 10/06/16 04:51, Wang Xiaoguang wrote: When testing btrfs compression, sometimes we got ENOSPC error, though fs still has much free space, xfstests generic/171, generic/172, generic/173, generic/174, generic/175 can reveal this bug in my

[PATCH v2] btrfs: introduce priority based delalloc shrink mechanism

2016-10-13 Thread Wang Xiaoguang
nd as the priority increases, we will try wo write more delalloc bytes, meanwhile if "reclaim_priority == 0" returns true, we'll also wait all current ordered extents to finish. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> ---

Re: [PATCH 1/2] btrfs: introduce priority based delalloc shrink mechanism

2016-10-12 Thread Wang Xiaoguang
hi, On 10/13/2016 01:20 AM, Josef Bacik wrote: On 10/12/2016 05:03 AM, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete tests, it's because shrink_delalloc

[PATCH] btrfs: make file clone aware of fatal signals

2016-10-12 Thread Wang Xiaoguang
Indeed this just make the behavior similar to xfs when process has fatal signals pending, and it'll make fstests/generic/298 happy. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/ioctl.c | 5 + 1 file changed, 5 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs

[PATCH 1/2] btrfs: introduce priority based delalloc shrink mechanism

2016-10-12 Thread Wang Xiaoguang
r of metadata we want to reserve. Meanwhile if "reclaim_priority >= 3" returns true, we'll also wait all current ordered extents to finish. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 49 ---

[PATCH 2/2] btrfs: try to satisfy metadata requests if wen can overcommit

2016-10-12 Thread Wang Xiaoguang
In shrink_delalloc(), if can_overcommit() returns true, shrink_delalloc() will give up shrinking delalloc bytes, in this case we should check whether some tickcts' requests can overcommit, if some can, we can satisfy them timely and directly. Signed-off-by: Wang Xiaoguang <wangx

Re: [PATCH 1/2] btrfs: try to satisfy metadata requests when every flush_space() returns

2016-10-12 Thread Wang Xiaoguang
hi, On 10/07/2016 09:16 PM, Josef Bacik wrote: On 09/21/2016 02:59 AM, Wang Xiaoguang wrote: In flush_space()->shrink_delalloc(), if can_overcommit() returns true, though no bytes added to space_info, we still may satisfy some requests. Signed-off-by: Wang Xiaoguang <wa

Re: [RFC] btrfs: make max inline data can be equal to sectorsize

2016-10-11 Thread Wang Xiaoguang
hi, On 10/11/2016 11:49 PM, Chris Murphy wrote: On Tue, Oct 11, 2016 at 12:47 AM, Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> wrote: If we use mount option "-o max_inline=sectorsize", say 4096, indeed even for a fresh fs, say nodesize is 16k, we can not make the first 4k data c

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-10-11 Thread Wang Xiaoguang
, Xiaoguang Wang On 10/06/2016 10:51 AM, Wang Xiaoguang wrote: When testing btrfs compression, sometimes we got ENOSPC error, though fs still has much free space, xfstests generic/171, generic/172, generic/173, generic/174, generic/175 can reveal this bug in my test environment when compression is enabled

[RFC] btrfs: make max inline data can be equal to sectorsize

2016-10-11 Thread Wang Xiaoguang
If it retuns true, we'll not make data inline. For 4k sectorsize, 0~4094 dara range, we can make it inline, but 0~4095, it can not. I don't think this limition is useful, so here remove it which will make max inline data can be equal to sectorsize. Signed-off-by: Wang Xiaoguang <wangxg.f...@c

Re: [PATCH] generic/175: disable inline data feature for btrfs

2016-10-10 Thread Wang Xiaoguang
hi, On 10/11/2016 07:09 AM, Dave Chinner wrote: On Mon, Oct 10, 2016 at 01:06:47PM +0800, Wang Xiaoguang wrote: For btrfs, if compression is enabled, it may generate inline data for a blocksize data range, this inline data is stored in fs tree, will not have a individual extent, try to reflink

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-10 Thread Wang Xiaoguang
hi, On 10/11/2016 04:06 AM, Stefan Priebe - Profihost AG wrote: Dear Wang, Am 06.10.2016 um 05:04 schrieb Wang Xiaoguang: Hi, On 09/29/2016 03:27 PM, Stefan Priebe - Profihost AG wrote: Am 29.09.2016 um 09:13 schrieb Wang Xiaoguang: I found that compress sometime report ENOSPC error even

Re: [PATCH 1/2] btrfs: try to satisfy metadata requests when every flush_space() returns

2016-10-10 Thread Wang Xiaoguang
hi, On 10/07/2016 09:16 PM, Josef Bacik wrote: On 09/21/2016 02:59 AM, Wang Xiaoguang wrote: In flush_space()->shrink_delalloc(), if can_overcommit() returns true, though no bytes added to space_info, we still may satisfy some requests. Signed-off-by: Wang Xiaoguang <wa

Re: [RFC 3/3] btrfs: make shrink_delalloc() try harder to reclaim metadata space

2016-10-10 Thread Wang Xiaoguang
Hi, On 10/07/2016 09:24 PM, Josef Bacik wrote: On 09/22/2016 05:25 AM, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete test, it's because shrink_delalloc

[PATCH v2] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-10 Thread Wang Xiaoguang
When enabling btrfs compression, original codes can not fill fs correctly, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- V2: In common/, I did't find an existing function suitable for these 4 test cases to fill fs, so I still use _pwrite_byte() with a big

[PATCH] generic/175: disable inline data feature for btrfs

2016-10-09 Thread Wang Xiaoguang
. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- tests/generic/175 | 7 +++ 1 file changed, 7 insertions(+) diff --git a/tests/generic/175 b/tests/generic/175 index 964580c..b3f90dc 100755 --- a/tests/generic/175 +++ b/tests/generic/175 @@ -50,6 +50,13 @@ rm -f "$

Re: [PATCH] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-09 Thread Wang Xiaoguang
hi, On 10/10/2016 05:04 AM, Darrick J. Wong wrote: On Sat, Oct 08, 2016 at 01:36:10AM +1100, Dave Chinner wrote: On Fri, Oct 07, 2016 at 03:00:42PM +0800, Wang Xiaoguang wrote: When enabling btrfs compression, original codes can not fill fs correctly, fix this. Signed-off-by: Wang Xiaoguang

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-07 Thread Wang Xiaoguang
hello, On 10/07/2016 04:16 PM, Stefan Priebe - Profihost AG wrote: Am 07.10.2016 um 10:07 schrieb Wang Xiaoguang: hello, On 10/07/2016 04:06 PM, Stefan Priebe - Profihost AG wrote: and it shows: PAG | scan 33829e5 | steal 1968e3 | stall 0

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-07 Thread Wang Xiaoguang
, there will be no OOMs in your test environment? I want to confirm whether this OOM is caused by my patches... Regards, Xiaoguang Wang greets, Stefan Am 07.10.2016 um 09:17 schrieb Wang Xiaoguang: Hi, On 10/07/2016 03:03 PM, Stefan Priebe - Profihost AG wrote: Dear Wang, can't use v4.8.0

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-07 Thread Wang Xiaoguang
didn't see any such OOMs... Can you try holger's tree with my patches. Regards, Xiaoguang Wang Greets, Stefan Am 06.10.2016 um 05:04 schrieb Wang Xiaoguang: Hi, On 09/29/2016 03:27 PM, Stefan Priebe - Profihost AG wrote: Am 29.09.2016 um 09:13 schrieb Wang Xiaoguang: I found that compress

[PATCH] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-07 Thread Wang Xiaoguang
When enabling btrfs compression, original codes can not fill fs correctly, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- tests/generic/171 | 4 +--- tests/generic/172 | 2 +- tests/generic/173 | 4 +--- tests/generic/174 | 4 +--- 4 files changed, 4 insertions(

Re: [RFC 3/3] btrfs: make shrink_delalloc() try harder to reclaim metadata space

2016-10-07 Thread Wang Xiaoguang
Hi, Any comments about these 3 patches, thanks. Regards, Xiaoguang Wang On 09/22/2016 05:25 PM, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete test, it's

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-10-05 Thread Wang Xiaoguang
Hi, On 10/06/2016 10:51 AM, Wang Xiaoguang wrote: When testing btrfs compression, sometimes we got ENOSPC error, though fs still has much free space, xfstests generic/171, generic/172, generic/173, generic/174, generic/175 can reveal this bug in my test environment when compression is enabled

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-05 Thread Wang Xiaoguang
Hi, On 09/29/2016 03:27 PM, Stefan Priebe - Profihost AG wrote: Am 29.09.2016 um 09:13 schrieb Wang Xiaoguang: I found that compress sometime report ENOSPC error even in 4.8-rc8, currently I cannot confirm that as i do not have anough space to test this without compression ;-( But yes i've

[PATCH 2/2] btrfs: fix false enospc for compression

2016-10-05 Thread Wang Xiaoguang
through compression path. With this patch, we can fix these false enospc error for compression. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 31 ++-- fs/btrfs/extent-tree.c | 55 + fs/btrfs/extent_io.c

[PATCH 1/2] btrfs: improve inode's outstanding_extents computation

2016-10-05 Thread Wang Xiaoguang
es() won't need to handle cases cross 2 extents. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/inode.c | 65 ++-- fs/btrfs/ioctl.c | 6 ++ 3 files changed, 62 insertions(+), 11 dele

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-09-29 Thread Wang Xiaoguang
Hi, On 09/29/2016 03:09 PM, Stefan Priebe - Profihost AG wrote: Am 29.09.2016 um 08:55 schrieb Wang Xiaoguang: Hi, On 09/29/2016 02:49 PM, Stefan Priebe - Profihost AG wrote: Hi, Am 28.09.2016 um 14:10 schrieb Wang Xiaoguang: OK, I see. But given that you often run into enospc errors, can

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-09-29 Thread Wang Xiaoguang
Hi, On 09/29/2016 02:49 PM, Stefan Priebe - Profihost AG wrote: Hi, Am 28.09.2016 um 14:10 schrieb Wang Xiaoguang: OK, I see. But given that you often run into enospc errors, can you work out a reproduce script according to you work load. That will give us great help. You got ENOSPC errors

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-09-28 Thread Wang Xiaoguang
hello, On 09/28/2016 08:02 PM, Stefan Priebe - Profihost AG wrote: Hi Xiaoguang Wang, Am 28.09.2016 um 13:35 schrieb Wang Xiaoguang: hello, On 09/28/2016 07:15 PM, Stefan Priebe - Profihost AG wrote: Dear list, is there any chance anybody wants to work with me on the following issue

[RFC 3/3] btrfs: make shrink_delalloc() try harder to reclaim metadata space

2016-09-22 Thread Wang Xiaoguang
want to reclaim metadata space, we can try harder, after all, false enospc error is not acceptable. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btr

Re: [PATCH v2] generic: check whether we can truncate heavily reflinked file

2016-09-22 Thread Wang Xiaoguang
hi, On 09/22/2016 02:18 PM, Darrick J. Wong wrote: On Thu, Sep 22, 2016 at 10:13:08AM +0800, Wang Xiaoguang wrote: In btrfs, there is a bug about btrfs' truncate codes, it'll leak some fs space as the truncate operation proceeds. If this truncate operation is very large, later metadata request

[PATCH 1/2] btrfs: try to satisfy metadata requests when every flush_space() returns

2016-09-21 Thread Wang Xiaoguang
In flush_space()->shrink_delalloc(), if can_overcommit() returns true, though no bytes added to space_info, we still may satisfy some requests. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 39 +++ 1 file ch

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-09-18 Thread Wang Xiaoguang
reproduce that even with the following commit: commit 18513091af9483ba84328d42092bd4d42a3c958f Author: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Date: Mon Jul 25 15:51:40 2016 +0800 btrfs: update btrfs_space_info's bytes_may_use timely And the btrfs_free_reserved_data_space_noquota WARN

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-09-12 Thread Wang Xiaoguang
hello, On 08/13/2016 01:36 AM, Ronan Arraes Jardim Chagas wrote: Hi guys, I'm facing a daily problem with BTRFS. Almost everyday, I get the message "No space left on device". Sometimes I can recover by balancing the system but sometimes even balancing does not work due to the lack of space. In

Re: [PATCH] generic: check whether we can truncate heavily reflinked file

2016-09-12 Thread Wang Xiaoguang
Hi, On 09/12/2016 05:18 PM, Eryu Guan wrote: On Mon, Sep 12, 2016 at 02:36:59PM +0800, Wang Xiaoguang wrote: Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> It's better to describe the test a bit in the commit log, e.g. why this test is needed etc., which at least could g

Re: [PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-12 Thread Wang Xiaoguang
hello, On 09/09/2016 06:56 PM, Holger Hoffstätte wrote: On 09/09/16 12:18, Holger Hoffstätte wrote: On Fri, 09 Sep 2016 16:17:48 +0800, Wang Xiaoguang wrote: cleaner_kthread() may run at any time, in which it'll call btrfs_delete_unused_bgs() to delete unused block groups. Because this work

Re: [PATCH 1/3] btrfs-progs: check: remove unused found_key variable in walk_down_tree()

2016-09-12 Thread Wang Xiaoguang
Hi, On 08/30/2016 12:22 AM, David Sterba wrote: On Thu, Aug 25, 2016 at 01:20:59PM +0800, Wang Xiaoguang wrote: Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- cmds-check.c | 5 - 1 file changed, 5 deletions(-) diff --git a/cmds-check.c b/cmds-check.c index 0

[PATCH] generic: check whether we can truncate heavily reflinked file

2016-09-12 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- tests/generic/377 | 74 +++ tests/generic/377.out | 2 ++ tests/generic/group | 1 + 3 files changed, 77 insertions(+) create mode 100755 tests/generic/377 create mode

bug report about patch "Btrfs: kill the btree_inode"

2016-09-09 Thread Wang Xiaoguang
hello, When we rebase dedupe patches to David's for-next-20160906 branch, we found below panic. By bisect, it seems that "Btrfs: kill the btree_inode" causing this bug, please check. Fstests case btrfs/060 can easily reproduce this bug. localhost login: [ 43.694734] BUG: unable to handle

Re: [PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-09 Thread Wang Xiaoguang
, which I think can resolve these issues, please check, thanks. Regards, Xiaoguang Wang On 09/09/2016 04:17 PM, Wang Xiaoguang wrote: cleaner_kthread() may run at any time, in which it'll call btrfs_delete_unused_bgs() to delete unused block groups. Because this work is asynchronous, it may also

[PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-09 Thread Wang Xiaoguang
: Stefan Priebe <s.pri...@profihost.ag> Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- V2: fix a deadlock revealed by fstests case btrfs/071, we call start_transaction() before in down_write(bg_delete_sem) in btrfs_delete_unused_bgs(). v3: Stefan Priebe reported

Re: [PATCH] btrfs: fix false enospc error when truncating heavily reflinked file

2016-09-07 Thread Wang Xiaoguang
Hi, On 09/07/2016 11:56 PM, Darrick J. Wong wrote: On Wed, Sep 07, 2016 at 08:17:38PM +0800, Wang Xiaoguang wrote: Below test script can reveal this bug: dd if=/dev/zero of=fs.img bs=$((1024*1024)) count=100 dev=$(losetup --show -f fs.img) mkdir -p /mnt/mntpoint mkfs.btrfs

[PATCH] btrfs: fix false enospc error when truncating heavily reflinked file

2016-09-07 Thread Wang Xiaoguang
data reservation. In this case, it's "btrfs_start_transaction(root, 2);" fails for enospc error, which make this truncate operation fail. Call btrfs_block_rsv_release() to fix this bug. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/inode.c | 1 + 1 file chan

Re: [PATCH] btrfs: introduce tickets_id to determine whether asynchronous metadata reclaim work makes progress

2016-09-06 Thread Wang Xiaoguang
hello, On 09/02/2016 09:28 PM, Josef Bacik wrote: On 09/01/2016 10:58 PM, Wang Xiaoguang wrote: In btrfs_async_reclaim_metadata_space(), we use ticket's address to determine whether asynchronous metadata reclaim work is making progress. ticket = list_first_entry(_info->tick

[PATCH] btrfs: introduce tickets_id to determine whether asynchronous metadata reclaim work makes progress

2016-09-01 Thread Wang Xiaoguang
ult in some enospc issues(I have seen this in my test machine). Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/ctree.h | 1 + fs/btrfs/extent-tree.c | 11 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrf

Re: [lkp] [btrfs] 18513091af: WARNING: CPU: 3 PID: 14091 at fs/btrfs/extent-tree.c:10008 btrfs_free_block_groups+0x29b/0x420 [btrfs]

2016-09-01 Thread Wang Xiaoguang
Hi, Thanks for your report. I've sent a patch to btrfs community to fix this issue, currently it's not merged. Regards, Xiaoguang Wang On 09/01/2016 04:32 PM, kernel test robot wrote: FYI, we noticed the following commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

[PATCH] btrfs: fix one bug that process may endlessly wait for ticket in wait_reserve_ticket()

2016-08-31 Thread Wang Xiaoguang
/172 with mount option "-o compress=lzo" have revealed this bug in my test machine. Here if we have tickets to handle, we must handle them first. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 10 +- 1 file changed, 5 insertions

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

2016-08-30 Thread Wang Xiaoguang
Hi, On 08/30/2016 07:32 PM, David Sterba wrote: On Tue, Aug 30, 2016 at 09:50:20AM +0800, Qu Wenruo wrote: Are they not? The low-memory patchset has been released in 4.7.1, the devel branch is always on top of master branch. I see both branches pushed to the public git repos so I don't see

Re: [PATCH] generic/159: record error value correctly

2016-08-29 Thread Wang Xiaoguang
hello, On 08/30/2016 12:45 PM, Darrick J. Wong wrote: On Tue, Aug 30, 2016 at 12:00:44PM +0800, Wang Xiaoguang wrote: Hi, On 08/30/2016 11:59 AM, Eryu Guan wrote: On Tue, Aug 30, 2016 at 11:51:14AM +0800, Wang Xiaoguang wrote: This case failed in btrfs, it's because when trying to open

Re: [PATCH] generic/159: record error value correctly

2016-08-29 Thread Wang Xiaoguang
Hi, On 08/30/2016 11:59 AM, Eryu Guan wrote: On Tue, Aug 30, 2016 at 11:51:14AM +0800, Wang Xiaoguang wrote: This case failed in btrfs, it's because when trying to open an immutable file, vfs returns EPERM, not EACCESS, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.

[PATCH] generic/159: record error value correctly

2016-08-29 Thread Wang Xiaoguang
This case failed in btrfs, it's because when trying to open an immutable file, vfs returns EPERM, not EACCESS, fix this. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- tests/generic/159.out | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/generic/1

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-08-29 Thread Wang Xiaoguang
hello, On 08/29/2016 11:52 PM, Ronan Arraes Jardim Chagas wrote: Hi guys, I just have the problem again. Now, it happens during the lunch time when the machine was idle. Only the system processes were running. It was not the first time that I saw this problem just after lunch when the machine

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-08-29 Thread Wang Xiaoguang
hello, On 08/13/2016 01:36 AM, Ronan Arraes Jardim Chagas wrote: Hi guys, I'm facing a daily problem with BTRFS. Almost everyday, I get the message "No space left on device". Sometimes I can recover by balancing the system but sometimes even balancing does not work due to the lack of space. In

[PATCH] btrfs: add ENOSPC check for temporarily allocated btrfs_block_rsv

2016-08-29 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- fs/btrfs/extent-tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 56a9c59..602e410 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -5552,6 +

[PATCH v2] btrfs: should block unused block groups deletion work when allocating data space

2016-08-29 Thread Wang Xiaoguang
sem) before start_transaction(), but becasue bg_delete_sem is rw_semaphore, there will be no deadlock. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- V2: fix a deadlock revealed by fstests case btrfs/071, we call start_transaction() before in down_write(bg_delete_sem) in

[PATCH] btrfs: do not decrease bytes_may_use when replaying extents

2016-08-25 Thread Wang Xiaoguang
When replaying extents, there is no need to update bytes_may_use in btrfs_alloc_logged_file_extent(), otherwise it'll trigger a WARN_ON about bytes_may_use. Fixes: ("btrfs: update btrfs_space_info's bytes_may_use timely") Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com&g

Re: [PULL] Btrfs fixes, part 3

2016-08-25 Thread Wang Xiaoguang
. Looks like this commit is deadlocking btrfs/071, I'm double checking with it removed: commit 9711669d602004f7658cf7fe003181de69e5ae6d Author: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Date: Wed Jul 27 13:50:21 2016 +0800 btrfs: should block unused block groups deletion work when allo

Re: [PULL] Btrfs fixes, part 3

2016-08-25 Thread Wang Xiaoguang
fs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html >From 37a68c327b8090ecf59e7019433638d7a569d3ab Mon Sep 17 00:00:00 2001 From: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> Date: Thu, 25 Aug 2016 10:09:09 +08

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

2016-08-24 Thread Wang Xiaoguang
Hi, On 08/24/2016 08:44 PM, David Sterba wrote: On Fri, Aug 19, 2016 at 05:59:46PM +0800, Wang Xiaoguang wrote: The basic idea is simple. Assume a middle tree node A is shared and its referenceing fs/file tree root ids are 5, 258 and 260, then we only check node A in the tree who has

[PATCH 1/3] btrfs-progs: check: remove unused found_key variable in walk_down_tree()

2016-08-24 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- cmds-check.c | 5 - 1 file changed, 5 deletions(-) diff --git a/cmds-check.c b/cmds-check.c index 0ddfd24..1cd0421 100644 --- a/cmds-check.c +++ b/cmds-check.c @@ -3737,7 +3737,6 @@ static int check_fs_root(struct btrf

[PATCH 2/3] btrfs-progs: check: make low memory mode support partially dropped snapshots

2016-08-24 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- cmds-check.c | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/cmds-check.c b/cmds-check.c index 1cd0421..bce586c 100644 --- a/cmds-check.c +++ b/cmds-check.c @@ -9016,9 +9016,10 @@ stat

[PATCH 3/3] btrfs-progs: tests: add 021-partially-dropped-snapshot-case

2016-08-24 Thread Wang Xiaoguang
Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- .../dropped-snapshot.img | Bin 0 -> 24576 bytes .../021-partially-dropped-snapshot-case/test.sh | 18 ++ 2 files changed, 18 insertions(+) create mode 100644 tests/fsck-

Re: [lkp] [btrfs] 994cdc1331: WARNING: CPU: 0 PID: 14378 at fs/btrfs/extent-tree.c:10030 btrfs_free_block_groups+0x29b/0x420 [btrfs]

2016-08-23 Thread Wang Xiaoguang
Hi, On 08/23/2016 03:45 PM, kernel test robot wrote: FYI, we noticed the following commit: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 994cdc133173fd9d69eae45cd228eec21dbed080 ("btrfs: update btrfs_space_info's bytes_may_use timely") in testcase:

Re: [PATCH 00/13] btrfs-progs: check: check fs roots in low_memory mode

2016-08-19 Thread Wang Xiaoguang
Hi On 08/18/2016 01:00 AM, David Sterba wrote: On Thu, Jul 28, 2016 at 03:08:12PM +0800, Lu Fengqi wrote: This patchset implements check fs trees in low memory mode, and fix the return value bug of the function cmd_check(). Lu Fengqi (13): btrfs-progs: move btrfs_extref_hash() to hash.h

[PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

2016-08-19 Thread Wang Xiaoguang
partition with linux source codes and create 3 snapshots in it. Before this patch, it averagely took 46s to finish one btrfsck execution, with this patch, it averagely took 15s. Signed-off-by: Wang Xiaoguang <wangxg.f...@cn.fujitsu.com> --- cmds-check.c

  1   2   >