Re: [PATCH v2 1/2] Btrfs: don't wait for unrelated IO to finish before relocation

On Tue, Apr 26, 2016 at 04:39:38PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Before the relocation process of a block group starts, it sets the block > group to readonly mode, then flushes all delalloc writes and then finally > it waits for all ordered

FYI: Kernel crash info

He guys! while testing/stressing (dd'ing 200GB random to the drive) a brand new 8TB seagate drive i ran into an kernel ooops. i think it happend after i finished dd'ing and while removing the drive. saw it a few minutes afterwards. uname -a Linux MacBookPro 4.4.0-22-generic #39~14.04.1-Ubuntu

Re: [PATCH v3 2/2] Btrfs: don't do unnecessary delalloc flushes when relocating

On Tue, Apr 26, 2016 at 06:45:24PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Before we start the actual relocation process of a block group, we do > calls to flush delalloc of all inodes and then wait for ordered extents > to complete. However we do these

Re: Idea on compatibility for old distributions

On 5/9/16 8:16 PM, Qu Wenruo wrote: > Hi David, Mark, > > In the recent test for new btrfs-convert backward compatibility, I > found that cmds-fi-du.c uses FIEMAP_EXTENT_SHARED bits, which is not > present in kernel of old distributions like RHEL6 (Sorry, didn't test > on openSUSE equivalent).

Re: About in-band dedupe for v4.7

On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: > Hi, Chris, Josef and David, > > As merge window for v4.7 is coming, it would be good to hear your > ideas about the inband dedupe. > > We are addressing the ENOSPC problem which Josef pointed out, and we > believe the final fix patch

Re: About in-band dedupe for v4.7

On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: > Hi, Chris, Josef and David, > > As merge window for v4.7 is coming, it would be good to hear your ideas > about the inband dedupe. > > We are addressing the ENOSPC problem which Josef pointed out, and we believe > the final fix patch

Re: [PATCH RFC v2 00/16] Introduce low memory usage btrfsck mode

David Sterba wrote on 2016/05/10 14:44 +0200: On Fri, Apr 29, 2016 at 03:18:54PM +0800, Qu Wenruo wrote: The branch can be fetched from my github: https://github.com/adam900710/btrfs-progs.git low_mem_fsck_rebasing ... Changelog: v1.1: Fix a typo which makes keyed data backref check

Re: About in-band dedupe for v4.7

Chris Mason wrote on 2016/05/10 20:37 -0400: On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: Hi, Chris, Josef and David, As merge window for v4.7 is coming, it would be good to hear your ideas about the inband dedupe. We are addressing the ENOSPC problem which Josef pointed out,

Re: [PATCH] Trivial fix for typos in comments.

On 10 May 2016 at 09:33, David Sterba wrote: > On Mon, May 09, 2016 at 08:13:29PM -0400, Nicholas D Steeves wrote: >> Trivial fix for typos in comments; I hope this patch isn't a nuisance! > > No, but I don't see the typos in any of the branches (master or the > for-next

Re: [PATCH v10 02/21] btrfs: dedupe: Introduce function to initialize dedupe info

On Fri, Apr 01, 2016 at 05:59:13PM +0800, kbuild test robot wrote: > Hi Wang, > > [auto build test ERROR on btrfs/next] > [also build test ERROR on v4.6-rc1 next-20160401] > [if your patch is applied to the wrong git tree, please drop us a note to > help improving the system] > > url: >

WARNING at fs/btrfs/inode.c:9261 btrfs_destroy_inode()

Hello, The following warning has been triggering for me since about v4.6-rc3: WARN_ON(BTRFS_I(inode)->csum_bytes); On one machine the warning has occurred 657 times since v4.6-rc5. On another it has occurred 3 times since v4.6-rc3. Both are now on v4.6-rc7, where I have still observed

Re: WARNING at fs/btrfs/inode.c:9261 btrfs_destroy_inode()

On Tue, May 10, 2016 at 06:19:20PM -0500, Eric Biggers wrote: > Hello, > > The following warning has been triggering for me since about v4.6-rc3: > > WARN_ON(BTRFS_I(inode)->csum_bytes); > > On one machine the warning has occurred 657 times since v4.6-rc5. On another > it > has occurred

Re: [PATCH v10 02/21] btrfs: dedupe: Introduce function to initialize dedupe info

Mark Fasheh wrote on 2016/05/10 17:00 -0700: On Fri, Apr 01, 2016 at 05:59:13PM +0800, kbuild test robot wrote: Hi Wang, [auto build test ERROR on btrfs/next] [also build test ERROR on v4.6-rc1 next-20160401] [if your patch is applied to the wrong git tree, please drop us a note to help

Re: FYI: Kernel crash info

On Tue, May 10, 2016 at 9:35 PM, wrote: > He guys! > > > while testing/stressing (dd'ing 200GB random to the drive) a brand new > 8TB seagate drive i ran into an kernel ooops. > > i think it happend after i finished dd'ing and while removing the drive. > saw it a few minutes

Re: About in-band dedupe for v4.7

Mark Fasheh wrote on 2016/05/10 15:11 -0700: On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: Hi, Chris, Josef and David, As merge window for v4.7 is coming, it would be good to hear your ideas about the inband dedupe. We are addressing the ENOSPC problem which Josef pointed out,

Re: About in-band dedupe for v4.7

On 2016/05/11 10:40, Qu Wenruo wrote: Chris Mason wrote on 2016/05/10 20:37 -0400: On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: Hi, Chris, Josef and David, As merge window for v4.7 is coming, it would be good to hear your ideas about the inband dedupe. We are addressing the

Re: [PATCH] fstests: btrfs: Test fiemap ioctl on completely deduped file

Filipe Manana wrote on 2016/05/10 11:01 +0100: On Tue, May 10, 2016 at 9:39 AM, Qu Wenruo wrote: For a completely deduped file, which means all its file extent are pointing to one bytenr, if calling fiemap on it, btrfs will cause soft hang up or just takes years

Re: About in-band dedupe for v4.7

On Wed, May 11, 2016 at 09:03:24AM +0800, Qu Wenruo wrote: > > > Mark Fasheh wrote on 2016/05/10 15:11 -0700: > >On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: > >>Hi, Chris, Josef and David, > >> > >>As merge window for v4.7 is coming, it would be good to hear your > >>ideas about

Re: [PATCH v10 02/21] btrfs: dedupe: Introduce function to initialize dedupe info

Qu Wenruo wrote on 2016/05/11 08:21 +0800: Mark Fasheh wrote on 2016/05/10 17:00 -0700: On Fri, Apr 01, 2016 at 05:59:13PM +0800, kbuild test robot wrote: Hi Wang, [auto build test ERROR on btrfs/next] [also build test ERROR on v4.6-rc1 next-20160401] [if your patch is applied to the

Re: About in-band dedupe for v4.7

On Wed, May 11, 2016 at 09:40:51AM +0800, Qu Wenruo wrote: > > > Chris Mason wrote on 2016/05/10 20:37 -0400: > >On Tue, May 10, 2016 at 03:19:52PM +0800, Qu Wenruo wrote: > >>Hi, Chris, Josef and David, > >> > >>As merge window for v4.7 is coming, it would be good to hear your ideas > >>about

Re: [PATCH] fstests: btrfs: Test fiemap ioctl on completely deduped file

On Tue, May 10, 2016 at 04:39:41PM +0800, Qu Wenruo wrote: > For a completely deduped file, which means all its file extent are > pointing to one bytenr, if calling fiemap on it, btrfs will cause soft > hang up or just takes years long. > > This bug can be reproduced even without any in-band or

Re: [PATCH] fstests: btrfs: Test fiemap ioctl on completely deduped file

On Wed, May 11, 2016 at 10:14:42AM +0800, Qu Wenruo wrote: > BTW, does other fs support reflink file range? > I found a lot xfs test cases using reflink, but I still can't reflink a file > range inside the same inode XFS work is under development and not in mainline yet. Also NFS can support

Re: [PATCH] Btrfs: fix fspath error deallocation

On Tue, May 10, 2016 at 02:56:20PM +0200, Vincent Stehlé wrote: > Make sure to deallocate fspath with vfree() in case of error in > init_ipath(). > > fspath is allocated with vmalloc() in init_data_container() since > commit 425d17a290c0 ("Btrfs: use larger limit for translation of logical to >

[PATCH 13/13] btrfs: check for failed device and hot replace

From: Anand Jain This patch checks for failed device and kicks out auto replace, if when user decided to disable auto replace it can be done by future sysfs or future ioctl interface to set fs_info->no_auto_replace parameter to 1. Signed-off-by: Anand Jain

[PATCH 05/13] btrfs: Cleanup num_tolerated_disk_barrier_failures

From: Qu Wenruo As we use per-chunk degradable check, now the global num_tolerated_disk_barrier_failures is of no use. So cleanup it. Signed-off-by: Qu Wenruo [Btrfs: resolve conflict to apply 'btrfs: Cleanup

[PATCH 1/1] btrfs: introduce device dynamic state transition to offline or failed

From: Anand Jain This patch provides helper functions to force a device to offline or failed, and we need this device states for the following reasons, 1) a. it can be reported that device has failed when it does b. close the device when it goes offline so that

[PATCH 02/13] btrfs: Do per-chunk check for mount time check

From: Qu Wenruo Now use the btrfs_check_degraded() to do mount time degraded check. With this patch, now we can mount with the following case: # mkfs.btrfs -f -m raid1 -d single /dev/sdb /dev/sdc # wipefs -a /dev/sdc # mount /dev/sdb /mnt/btrfs -o degraded As the

[PATCH v6 00/13] Introduce device state 'failed', spare device and auto replace

Thanks for various comments, tests and feedback. Background: Spare device and Auto replace: Spare device is predominately used to mitigate or narrow the time window of a degraded raid mode, as because during which any further disk failure would lead to a catastrophic data loss. Data center

[PATCH 1/1] btrfs: introduce helper functions to perform hot replace

From: Anand Jain Hot replace / auto replace is important volume manager feature and is critical to the data center operations, so that the degraded volume can be brought back to a healthy state at the earliest and without manual intervention. This modifies the existing

[PATCH 01/13] btrfs: Introduce a new function to check if all chunks a OK for degraded mount

From: Qu Wenruo Introduce a new function, btrfs_check_degradable(), to judge if all chunks in btrfs is OK for degraded mount. It provides the new basis for accurate btrfs mount/remount and even runtime degraded mount check other than old one-size-fit-all method.

[PATCH 04/13] btrfs: Allow barrier_all_devices to do per-chunk device check

From: Qu Wenruo The last user of num_tolerated_disk_barrier_failures is barrier_all_devices(). But it's can be easily changed to new per-chunk degradable check framework. Now btrfs_device will have two extra members, representing send/wait error, set at

[PATCH v6 00/13] Introduce device state 'failed', spare device and auto replace

Thanks for various comments, tests and feedback. Background: Spare device and Auto replace: Spare device is predominately used to mitigate or narrow the time window of a degraded raid mode, as because during which any further disk failure would lead to a catastrophic data loss. Data center

[PATCH 03/13] btrfs: Do per-chunk degraded check for remount

From: Qu Wenruo Just the same for mount time check, use new btrfs_check_degraded() to do per chunk check. Signed-off-by: Qu Wenruo Btrfs: use btrfs_error instead of btrfs_err during remount Signed-off-by: Anand Jain

[PATCH 07/13] btrfs: add check not to mount a spare device

From: Anand Jain Spare devices can be scanned but shouldn't be mountable. Signed-off-by: Anand Jain Tested-by: Austin S. Hemmelgarn Tested-by: Yauhen Kharuzhy --- fs/btrfs/disk-io.c | 8

[PATCH 1/1] btrfs: introduce helper functions to perform hot replace

From: Anand Jain Hot replace / auto replace is important volume manager feature and is critical to the data center operations, so that the degraded volume can be brought back to a healthy state at the earliest and without manual intervention. This modifies the existing

[PATCH 1/1] btrfs: introduce helper functions to perform hot replace

From: Anand Jain Hot replace / auto replace is important volume manager feature and is critical to the data center operations, so that the degraded volume can be brought back to a healthy state at the earliest and without manual intervention. This modifies the existing

[PATCH] btrfs: introduce BTRFS_FEATURE_INCOMPAT_SPARE_DEV

From: Anand Jain Add BTRFS_FEATURE_INCOMPAT_SPARE_DEV (400) flag to identify a spare device. Along with this it checks in the mount context that a spare device will fail to mount. As spare devices aren't mountable. Signed-off-by: Anand Jain

[PATCH 08/13] btrfs: support btrfs dev scan for spare device

From: Anand Jain When the user or system calls the BTRFS_IOC_SCAN_DEV, ioctl this patch will make sure it is added to the device list and set it as spare. This operation will be same when BTRFS_IOC_DEVICES_READY as well since BTRFS_IOC_DEVICES_READY ioctl has been doing

[PATCH 09/13] btrfs: provide framework to get and put a spare device

From: Anand Jain This adds functions to get and put a spare device from the list. So that hot repace code can pick a spare device when needed. Signed-off-by: Anand Jain Tested-by: Austin S. Hemmelgarn Tested-by: Yauhen

[PATCH 1/1] btrfs: check device for critical errors and mark failed

From: Anand Jain Write and Flush errors are considered as critical errors, upon which the device will be brought offline and marked as failed. Write and Flush errors are identified using device error statistics. This is monitored using a kthread btrfs_health.

[PATCH] Btrfs: fix fspath error deallocation

Make sure to deallocate fspath with vfree() in case of error in init_ipath(). fspath is allocated with vmalloc() in init_data_container() since commit 425d17a290c0 ("Btrfs: use larger limit for translation of logical to inode"). Signed-off-by: Vincent Stehlé Cc: Chris

[PATCH 01/13] btrfs: Introduce a new function to check if all chunks a OK for degraded mount

From: Qu Wenruo Introduce a new function, btrfs_check_degradable(), to judge if all chunks in btrfs is OK for degraded mount. It provides the new basis for accurate btrfs mount/remount and even runtime degraded mount check other than old one-size-fit-all method.

Re: [PATCH RFC v2 00/16] Introduce low memory usage btrfsck mode

On Fri, Apr 29, 2016 at 03:18:54PM +0800, Qu Wenruo wrote: > The branch can be fetched from my github: > https://github.com/adam900710/btrfs-progs.git low_mem_fsck_rebasing ... > Changelog: > v1.1: > Fix a typo which makes keyed data backref check always fail > v2: > Don't return minus

Re: [PATCH] Trivial fix for typos in comments.

On Mon, May 09, 2016 at 08:13:29PM -0400, Nicholas D Steeves wrote: > Trivial fix for typos in comments; I hope this patch isn't a nuisance! No, but I don't see the typos in any of the branches (master or the for-next snapshots). -- To unsubscribe from this list: send the line "unsubscribe

[PATCH 03/13] btrfs: Do per-chunk degraded check for remount

From: Qu Wenruo Just the same for mount time check, use new btrfs_check_degraded() to do per chunk check. Signed-off-by: Qu Wenruo Btrfs: use btrfs_error instead of btrfs_err during remount Signed-off-by: Anand Jain

[PATCH 06/13] btrfs: introduce BTRFS_FEATURE_INCOMPAT_SPARE_DEV

From: Anand Jain Add BTRFS_FEATURE_INCOMPAT_SPARE_DEV (400) flag to identify a spare device. Along with this it checks in the mount context that a spare device will fail to mount. As spare devices aren't mountable. Signed-off-by: Anand Jain

[PATCH 12/13] btrfs: check device for critical errors and mark failed

From: Anand Jain Write and Flush errors are considered as critical errors, upon which the device will be brought offline and marked as failed. Write and Flush errors are identified using device error statistics. This is monitored using a kthread btrfs_health.

[PATCH 05/13] btrfs: Cleanup num_tolerated_disk_barrier_failures

From: Qu Wenruo As we use per-chunk degradable check, now the global num_tolerated_disk_barrier_failures is of no use. So cleanup it. Signed-off-by: Qu Wenruo [Btrfs: resolve conflict to apply 'btrfs: Cleanup

[PATCH 13/13] btrfs: check for failed device and hot replace

From: Anand Jain This patch checks for failed device and kicks out auto replace, if when user decided to disable auto replace it can be done by future sysfs or future ioctl interface to set fs_info->no_auto_replace parameter to 1. Signed-off-by: Anand Jain

[PATCH 07/13] btrfs: add check not to mount a spare device

From: Anand Jain Spare devices can be scanned but shouldn't be mountable. Signed-off-by: Anand Jain Tested-by: Austin S. Hemmelgarn Tested-by: Yauhen Kharuzhy --- fs/btrfs/disk-io.c | 8

[PATCH 08/13] btrfs: support btrfs dev scan for spare device

From: Anand Jain When the user or system calls the BTRFS_IOC_SCAN_DEV, ioctl this patch will make sure it is added to the device list and set it as spare. This operation will be same when BTRFS_IOC_DEVICES_READY as well since BTRFS_IOC_DEVICES_READY ioctl has been doing

[PATCH 11/13] btrfs: introduce device dynamic state transition to offline or failed

From: Anand Jain This patch provides helper functions to force a device to offline or failed, and we need this device states for the following reasons, 1) a. it can be reported that device has failed when it does b. close the device when it goes offline so that

[PATCH 09/13] btrfs: provide framework to get and put a spare device

From: Anand Jain This adds functions to get and put a spare device from the list. So that hot repace code can pick a spare device when needed. Signed-off-by: Anand Jain Tested-by: Austin S. Hemmelgarn Tested-by: Yauhen

[PATCH 04/13] btrfs: Allow barrier_all_devices to do per-chunk device check

From: Qu Wenruo The last user of num_tolerated_disk_barrier_failures is barrier_all_devices(). But it's can be easily changed to new per-chunk degradable check framework. Now btrfs_device will have two extra members, representing send/wait error, set at

[PATCH 10/13] btrfs: introduce helper functions to perform hot replace

From: Anand Jain Hot replace / auto replace is important volume manager feature and is critical to the data center operations, so that the degraded volume can be brought back to a healthy state at the earliest and without manual intervention. This modifies the existing

[PATCH 02/13] btrfs: Do per-chunk check for mount time check

From: Qu Wenruo Now use the btrfs_check_degraded() to do mount time degraded check. With this patch, now we can mount with the following case: # mkfs.btrfs -f -m raid1 -d single /dev/sdb /dev/sdc # wipefs -a /dev/sdc # mount /dev/sdb /mnt/btrfs -o degraded As the

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On 05/10/2016 10:28 PM, David Sterba wrote: On Tue, May 10, 2016 at 04:40:58PM +0800, Anand Jain wrote: On 05/10/2016 04:21 PM, David Sterba wrote: On Tue, May 10, 2016 at 04:01:05PM +0800, Anand Jain wrote: In more detail: * introduce _btrfs_printk that takes a string pointer as 1st

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On Tue, May 10, 2016 at 04:40:58PM +0800, Anand Jain wrote: > > > On 05/10/2016 04:21 PM, David Sterba wrote: > > On Tue, May 10, 2016 at 04:01:05PM +0800, Anand Jain wrote: > >>> In more detail: > >>> > >>> * introduce _btrfs_printk that takes a string pointer as 1st argument > >>> (this

Re: [PATCH] Trivial fix for typos in comments.

On Mon, May 09, 2016 at 08:13:29PM -0400, Nicholas D Steeves wrote: > Trivial fix for typos in comments; I hope this patch isn't a nuisance! I ran ispell on the strings from comments, there are like 90+ typos (attached) that seem worth fixing, if you like. antyhing seperate alot * any device

Re: [PATCH 1/3] Btrfs-progs: add validation checks for chunk loading

On Mon, May 02, 2016 at 11:18:53AM -0700, Liu Bo wrote: > To survive fuzz filesystem images, we need various validation checks to > make btrfsck detect any invalid value inside chunks including those in > sys_array. > > Note that these checks may not be sufficient to cover all corner cases, > we

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On Tue, May 10, 2016 at 04:01:05PM +0800, Anand Jain wrote: > > In more detail: > > > > * introduce _btrfs_printk that takes a string pointer as 1st argument > >(this could be used for messages before fs_info exists) > > * _btrfs_printk(NULL, ...) will be valid > > * then btrfs_printk(fs_info,

About in-band dedupe for v4.7

Hi, Chris, Josef and David, As merge window for v4.7 is coming, it would be good to hear your ideas about the inband dedupe. We are addressing the ENOSPC problem which Josef pointed out, and we believe the final fix patch would come out at the beginning of the merge window.(Next week) If

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On Tue, May 10, 2016 at 10:34:19AM +0800, Anand Jain wrote: > On 05/09/2016 05:39 PM, David Sterba wrote: > > Currently we lack the identification of the filesystem in most if not > > all mount messages, done via printk/pr_* functions. We can use the > > btrfs_* helpers in open_ctree, as the

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On 05/10/2016 04:21 PM, David Sterba wrote: On Tue, May 10, 2016 at 04:01:05PM +0800, Anand Jain wrote: In more detail: * introduce _btrfs_printk that takes a string pointer as 1st argument (this could be used for messages before fs_info exists) * _btrfs_printk(NULL, ...) will be valid *

[PATCH] fstests: btrfs: Test fiemap ioctl on completely deduped file

For a completely deduped file, which means all its file extent are pointing to one bytenr, if calling fiemap on it, btrfs will cause soft hang up or just takes years long. This bug can be reproduced even without any in-band or out-of-band dedupe, normal clone_file_range() call can create such

Re: [PATCH] btrfs: switch to common message helpers in open_ctree, adjust messages

On 05/10/2016 03:42 PM, David Sterba wrote: On Tue, May 10, 2016 at 10:34:19AM +0800, Anand Jain wrote: On 05/09/2016 05:39 PM, David Sterba wrote: Currently we lack the identification of the filesystem in most if not all mount messages, done via printk/pr_* functions. We can use the btrfs_*

Re: [PATCH 1/2] btrfs: create degraded-RAID1 chunks

-.devs_min= 2, +.devs_min= 1, I think we should introduce another way how to determine the lower limit for the degraded mounts. We need the proper raidX constraints and use the degraded limits only if in case of the degraded mount. .tolerated_failures =

Re: [PATCH 3/3] Btrfs: don't do unnecessary delalloc flushes when relocating

On Tue, May 10, 2016 at 12:56 AM, Liu Bo wrote: > On Mon, Apr 25, 2016 at 02:01:12AM +0100, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Before we start the actual relocation process of a block group, we do >> calls to flush delalloc of all

Re: Parent transid failed wanted x found x + [3-5]

Shoot, sorry all. I am clearly too sleep deprived. I just realized that I was misreading the transid errors, and that the journal was ahead of the disk. Sorry for the mistake. On Tue, May 10, 2016 at 3:23 AM, Zachary Bischof wrote: > Hi all, > > I'm a bit of an

Re: [PATCH] fstests: btrfs: Test fiemap ioctl on completely deduped file

On Tue, May 10, 2016 at 9:39 AM, Qu Wenruo wrote: > For a completely deduped file, which means all its file extent are > pointing to one bytenr, if calling fiemap on it, btrfs will cause soft > hang up or just takes years long. > > This bug can be reproduced even without