Re: [PATCH 4/5] btrfs: raid56: Don't keep rbio for later steal

At 03/17/2017 12:44 PM, Liu Bo wrote: On Fri, Feb 03, 2017 at 04:20:22PM +0800, Qu Wenruo wrote: Before this patch, btrfs raid56 will keep raid56 rbio even all its IO is done. This may save some time allocating rbio, but it can cause deadly use-after-free bug, for the following case:

Re: [PATCH 4/5] btrfs: raid56: Don't keep rbio for later steal

On Fri, Feb 03, 2017 at 04:20:22PM +0800, Qu Wenruo wrote: > Before this patch, btrfs raid56 will keep raid56 rbio even all its IO is > done. > This may save some time allocating rbio, but it can cause deadly > use-after-free bug, for the following case: > > Original fs: 4 devices RAID5 > >

help : "bad tree block start" -> btrfs forced readonly

Hi, our largest BTRFS filesystem is damaged but I'm unclear if it is recoverable or not. This is a 20TB filesystem with ~13TB used in a virtual machine using virtio-scsi backed by Ceph (Firefly 0.8.10). The following messages have become more frequent : fileserver kernel: sd 0:0:1:0: [sdb] tag#

Re: help!!! error when mount a btrfs file system

At 03/17/2017 01:36 AM, Liu Bo wrote: On Thu, Mar 16, 2017 at 08:23:05PM +0800, 李云甫 wrote: hi, buddy I have a file server with btrfs file system, it's work well for several months. but after last system reboot, the /dev/sdb become not mountable. below is the details. is there any

[PATCH 2/2] btrfs-progs: fsck-tests: Add SHARED_DATA_REF test image for lowmem mode

Introduce a new image, which contains external SHARED_DATA_REF items to trigger a lowmem mode false alert. The image only contains external SHARED_DATA_REF and no inlined data backref. Before the image, we only have inlined shared data ref, which is not enough to trigger lowmem mode false alert.

[PATCH 1/2] btrfs-progs: lowmeme check: Fix false alert about backref lost for SHARED_DATA_REF

In check_extent_data_item(), after checking extent item of one data extent, we search inlined data backref, then EXTENT_DATA_REF_KEY. But we didn't search SHARED_DATA_REF, so if the backref is SHARED_DATA_REF, then we will raise a false alert about backref lost. Fix by also checking

Re: [PATCH 5/8] nowait aio: return on congested block device

On Thu, Mar 16, 2017 at 10:33 PM, Jens Axboe wrote: > On 03/15/2017 03:51 PM, Goldwyn Rodrigues wrote: >> diff --git a/block/blk-core.c b/block/blk-core.c >> index 0eeb99e..2e5cba2 100644 >> --- a/block/blk-core.c >> +++ b/block/blk-core.c >> @@ -2014,7 +2019,7 @@ blk_qc_t

Re: [PATCH v2 2/2] btrfs: replace hardcoded value with SEQ_LAST macro

At 03/17/2017 12:04 AM, ednadol...@gmail.com wrote: From: Edmund Nadolski Define the SEQ_LAST macro to replace (u64)-1 in places where said value triggers a special-case ref search behavior. Signed-off-by: Edmund Nadolski Reviewed-by: Jeff Mahoney

Re: btrfs check lowmem vs original

At 03/17/2017 07:22 AM, Chris Murphy wrote: With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the following errors only with --mode=lowmem with a single device volume. ERROR: data extent[16913485824 7577600] backref lost (hundreds of lines of the same thing) btrfs-image (no errors

Re: [PATCH 2/2] btrfs-progs: convert: Make btrfs_reserved_ranges const

At 03/17/2017 12:33 AM, David Sterba wrote: On Thu, Mar 16, 2017 at 11:18:31AM +0800, Qu Wenruo wrote: Since btrfs_reserved_ranges array is just used to store btrfs reserved ranges, no one will nor should modify them at run time, make them static and const will be better. This also

Re: btrfs check lowmem vs original

With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the following errors only with --mode=lowmem with a single device volume. ERROR: data extent[16913485824 7577600] backref lost (hundreds of lines of the same thing) btrfs-image (no errors reported during capture) 518MiB

Re: [PATCH 5/8] nowait aio: return on congested block device

On Wed, Mar 15, 2017 at 04:51:04PM -0500, Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > A new flag BIO_NOWAIT is introduced to identify bio's > orignating from iocb with IOCB_NOWAIT. This flag indicates > to return immediately if a request cannot be made instead > of

Re: help!!! error when mount a btrfs file system

On Thu, Mar 16, 2017 at 08:23:05PM +0800, 李云甫 wrote: > hi, buddy > >I have a file server with btrfs file system, it's work well for several > months. > > but after last system reboot, the /dev/sdb become not mountable. > > below is the details. is there any advise? > > > ##Version 

[PATCH v2 0/2] Cleanup for some hardcoded constants

From: Edmund Nadolski This series replaces several hard-coded values with descriptive symbols. --- v2: + rename SEQ_NONE to SEQ_LAST and move definition to ctree.h + clarify comment at __merge_refs() Edmund Nadolski (2): btrfs: provide enumeration for __merge_refs mode

Re: [PATCH 2/2] btrfs-progs: convert: Make btrfs_reserved_ranges const

On Thu, Mar 16, 2017 at 11:18:31AM +0800, Qu Wenruo wrote: > Since btrfs_reserved_ranges array is just used to store btrfs reserved > ranges, no one will nor should modify them at run time, make them static > and const will be better. > > This also eliminates the use of immediate number 3.

[PATCH v2 1/2] btrfs: provide enumeration for __merge_refs mode argument

From: Edmund Nadolski Replace hardcoded numeric values for __merge_refs 'mode' argument with descriptive constants. Signed-off-by: Edmund Nadolski Reviewed-by: Jeff Mahoney --- fs/btrfs/backref.c | 23 +-- 1 file

[PATCH v2 2/2] btrfs: replace hardcoded value with SEQ_LAST macro

From: Edmund Nadolski Define the SEQ_LAST macro to replace (u64)-1 in places where said value triggers a special-case ref search behavior. Signed-off-by: Edmund Nadolski Reviewed-by: Jeff Mahoney --- fs/btrfs/backref.c | 16

Re: [PATCH v2 2/2] btrfs-progs: dump-tree: Also output log root tree

On Tue, Mar 14, 2017 at 01:35:18PM +0800, Qu Wenruo wrote: > In btrfs-dump-tree, we output any existing log tree, however we don't > output the log root tree, which records all root items for log trees. > > This makes it confusing for any one who want to know where the log tree > comes from. > >

Re: [PATCH 5/8] nowait aio: return on congested block device

On 03/15/2017 03:51 PM, Goldwyn Rodrigues wrote: > diff --git a/block/blk-core.c b/block/blk-core.c > index 0eeb99e..2e5cba2 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -2014,7 +2019,7 @@ blk_qc_t generic_make_request(struct bio *bio) > do { > struct

Re: [PATCH 3/8] nowait aio: return if direct write will trigger writeback

On 03/16/2017 08:20 AM, Matthew Wilcox wrote: > On Wed, Mar 15, 2017 at 04:51:02PM -0500, Goldwyn Rodrigues wrote: >> From: Goldwyn Rodrigues >> >> Find out if the write will trigger a wait due to writeback. If yes, >> return -EAGAIN. >> >> This introduces a new function

Re: [PATCH 3/8] nowait aio: return if direct write will trigger writeback

On 03/16/2017 08:08 AM, Matthew Wilcox wrote: > On Wed, Mar 15, 2017 at 04:51:02PM -0500, Goldwyn Rodrigues wrote: >> This introduces a new function filemap_range_has_page() which >> returns true if the file's mapping has a page within the range >> mentioned. > > I thought you were going to

Re: [PATCH 3/8] nowait aio: return if direct write will trigger writeback

On Wed, Mar 15, 2017 at 04:51:02PM -0500, Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > Find out if the write will trigger a wait due to writeback. If yes, > return -EAGAIN. > > This introduces a new function filemap_range_has_page() which > returns true if the

Re: [PATCH 3/8] nowait aio: return if direct write will trigger writeback

On Wed, Mar 15, 2017 at 04:51:02PM -0500, Goldwyn Rodrigues wrote: > This introduces a new function filemap_range_has_page() which > returns true if the file's mapping has a page within the range > mentioned. I thought you were going to replace this patch with one that starts writeback for these

help!!! error when mount a btrfs file system

hi, buddy I have a file server with btrfs file system, it's work well for several months. but after last system reboot, the /dev/sdb become not mountable. below is the details. is there any advise? ##Version info Fedora 25 Server Kernel 4.9.13-201.fc25.x86_64 btrfs-progs v4.6.1

help!!! error when mount a btrfs file system

hi, buddy I have a file server with btrfs file system, it's work well for several months. but after last system reboot, the /dev/sdb become not mountable. below is the details. is there any advise? ##Version info Fedora 25 Server Kernel 4.9.13-201.fc25.x86_64 btrfs-progs v4.6.1

[PATCH v2 2/3] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test

Btrfs balance with inband dedupe enable/disable will expose a lot of hidden dedupe bug: 1) Enable/disable race bug 2) Btrfs dedupe tree balance corrupted delayed_ref 3) Btrfs disable and balance will cause balance BUG_ON Reported-by: Satoru Takeuchi

[PATCH v2 3/3] fstests: btrfs: Test inband dedupe with data balance.

Btrfs balance will reloate date extent, but its hash is removed too late at run_delayed_ref() time, which will cause extent ref increased during balance, cause either find_data_references() gives WARN_ON() or even run_delayed_refs() fails and cause transaction abort. Add such concurrency test for

[PATCH v2 1/3] fstests: btrfs: Add basic test for btrfs in-band de-duplication

Add basic test for btrfs in-band de-duplication(inmemory backend), including: 1) Enable 3) Dedup rate 4) File correctness 5) Disable Signed-off-by: Qu Wenruo --- common/defrag | 14 +++ tests/btrfs/200 | 110

[PATCH v2 0/3] Btrfs in-band de-duplication test cases

Btrfs in-band de-duplication test cases for in-memory backend, which covers the bugs exposed during the development. v2: Fix double disable and exit Use $KILLALL_PROG Use local variable in _count_extent_uniq() Update date of the test cases Move test case 203 to 202, to make up the

Re: [PATCH 1/3] fstests: btrfs: Add basic test for btrfs in-band de-duplication

At 03/16/2017 04:08 PM, Eryu Guan wrote: On Thu, Mar 16, 2017 at 09:50:25AM +0800, Qu Wenruo wrote: Add basic test for btrfs in-band de-duplication(inmemory backend), including: 1) Enable 3) Dedup rate 4) File correctness 5) Disable Signed-off-by: Qu Wenruo I

Re: [PATCH 3/5] btrfs: raid56: Use correct stolen pages to calculate P/Q

At 03/16/2017 01:36 PM, Liu Bo wrote: On Fri, Feb 03, 2017 at 04:20:21PM +0800, Qu Wenruo wrote: In the following situation, scrub will calculate wrong parity to overwrite correct one: RAID5 full stripe: Before | Dev 1 | Dev 2 | Dev 3 | | Data stripe 1 | Data

Re: [PATCH 2/3] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test

On Thu, Mar 16, 2017 at 09:50:27AM +0800, Qu Wenruo wrote: > Btrfs balance with inband dedupe enable/disable will expose a lot of > hidden dedupe bug: > > 1) Enable/disable race bug > 2) Btrfs dedupe tree balance corrupted delayed_ref > 3) Btrfs disable and balance will cause balance BUG_ON > >

Re: [PATCH 1/3] fstests: btrfs: Add basic test for btrfs in-band de-duplication

On Thu, Mar 16, 2017 at 09:50:25AM +0800, Qu Wenruo wrote: > Add basic test for btrfs in-band de-duplication(inmemory backend), including: > 1) Enable > 3) Dedup rate > 4) File correctness > 5) Disable > > Signed-off-by: Qu Wenruo I haven't looked into this patchset

[PATCH v14.2 08/16] btrfs: delayed-ref: Add support for increasing data ref under spinlock

For in-band dedupe, btrfs needs to increase data ref with delayed_ref locked, so add a new function btrfs_add_delayed_data_ref_lock() to increase extent ref with delayed_refs already locked. Signed-off-by: Qu Wenruo Reviewed-by: Josef Bacik ---

[PATCH v14.2 11/16] btrfs: ordered-extent: Add support for dedupe

From: Wang Xiaoguang Add ordered-extent support for dedupe. Note, current ordered-extent support only supports non-compressed source extent. Support for compressed source extent will be added later. Signed-off-by: Qu Wenruo Signed-off-by:

[PATCH v14.2 09/16] btrfs: dedupe: Introduce function to search for an existing hash

From: Wang Xiaoguang Introduce static function inmem_search() to handle the job for in-memory hash tree. The trick is, we must ensure the delayed ref head is not being run at the time we search the for the hash. With inmem_search(), we can implement the

[PATCH v14.2 16/16] btrfs: dedupe: Introduce new reconfigure ioctl

Introduce new reconfigure ioctl, and new FORCE flag for in-band dedupe ioctls. Now dedupe enable and reconfigure ioctl are stateful. | Current state | Ioctl| Next state | | Disabled | enable

[PATCH v14.2 06/16] btrfs: dedupe: Introduce function to add hash into in-memory tree

From: Wang Xiaoguang Introduce static function inmem_add() to add hash into in-memory tree. And now we can implement the btrfs_dedupe_add() interface. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang

[PATCH v14.2 00/16] Btrfs In-band De-duplication

This patchset can be fetched from github: https://github.com/adam900710/linux.git wang_dedupe_latest This is just a normal rebase update. Now the new base is v4.11-rc2, and the patchset no longer rely qgroup fixes. Normal test cases from auto group exposes no regression, and ib-dedupe group can

[PATCH v14.2 05/16] btrfs: dedupe: Introduce function to initialize dedupe info

From: Wang Xiaoguang Add generic function to initialize dedupe info. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik --- fs/btrfs/Makefile | 2 +-

[PATCH v14.2 10/16] btrfs: dedupe: Implement btrfs_dedupe_calc_hash interface

From: Wang Xiaoguang Unlike in-memory or on-disk dedupe method, only SHA256 hash method is supported yet, so implement btrfs_dedupe_calc_hash() interface using SHA256. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang

[PATCH v14.2 14/16] btrfs: dedupe: Add ioctl for inband dedupelication

From: Wang Xiaoguang Add ioctl interface for inband dedupelication, which includes: 1) enable 2) disable 3) status And a pseudo RO compat flag, to imply that btrfs now supports inband dedup. However we don't add any ondisk format change, it's just a pseudo RO compat

[PATCH v14.2 04/16] btrfs: dedupe: Introduce dedupe framework and its header

From: Wang Xiaoguang Introduce the header for btrfs in-band(write time) de-duplication framework and needed header. The new de-duplication framework is going to support 2 different dedupe methods and 1 dedupe hash. Signed-off-by: Qu Wenruo

[PATCH v14.2 03/16] btrfs: Introduce COMPRESS reserve type to fix false enospc for compression

From: Wang Xiaoguang When testing btrfs compression, sometimes we got ENOSPC error, though fs still has much free space, xfstests generic/171, generic/172, generic/173, generic/174, generic/175 can reveal this bug in my test environment when compression is enabled.

[PATCH v14.2 13/16] btrfs: Introduce DEDUPE reserve type to fix false enospc for in-band dedupe

From: Wang Xiaoguang We can trigger false ENOSPC error if enabling in-band dedupe. This is the same reason of compress false ENOSPC error: Difference in max extent size can lead to too much space reserved for metadata. Since we already have type-based reserve

[PATCH v14.2 15/16] btrfs: relocation: Enhance error handling to avoid BUG_ON

Since the introduce of btrfs dedupe tree, it's possible that balance can race with dedupe disabling. When this happens, dedupe_enabled will make btrfs_get_fs_root() return PTR_ERR(-ENOENT). But due to a bug in error handling branch, when this happens backref_cache->nr_nodes is increased but the

[PATCH v14.2 12/16] btrfs: dedupe: Inband in-memory only de-duplication implement

Core implement for inband de-duplication. It reuse the async_cow_start() facility to do the calculate dedupe hash. And use dedupe hash to do inband de-duplication at extent level. The work flow is as below: 1) Run delalloc range for an inode 2) Calculate hash for the delalloc range at the unit of

[PATCH v14.2 01/16] btrfs: improve inode's outstanding_extents computation

From: Wang Xiaoguang This issue was revealed by modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB, When modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB, fsstress test often gets these warnings from btrfs_destroy_inode():

[PATCH v14.2 07/16] btrfs: dedupe: Introduce function to remove hash from in-memory tree

From: Wang Xiaoguang Introduce static function inmem_del() to remove hash from in-memory dedupe tree. And implement btrfs_dedupe_del() and btrfs_dedup_disable() interfaces. Also for btrfs_dedupe_disable(), add new functions to wait existing writer and block incoming

[PATCH v14.2 02/16] btrfs: introduce type based delalloc metadata reserve

From: Wang Xiaoguang Introduce type based metadata reserve parameter for delalloc space reservation/freeing function. The problem we are going to solve is, btrfs use different max extent size for different mount options. For compression, the max extent size is 128K,