Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Qu Wenruo
On 2018年07月13日 13:32, Anand Jain wrote: > > > > > >>> But if you are planning to >>>    record and start at transaction [14] then its an overkill because >>>    transaction [19 and [20] are already in the disk. >> >> Yes, I'm doing it overkilled. > >  Ah. Ok. > >> But it's already much bet

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Anand Jain
But if you are planning to   record and start at transaction [14] then its an overkill because   transaction [19 and [20] are already in the disk. Yes, I'm doing it overkilled. Ah. Ok. But it's already much better than scrub all block groups (my original plan). That's true. Whic

Re: [PATCH v2] btrfs: use customized batch size for total_bytes_pinned

2018-07-12 Thread ethanlien
Omar Sandoval 於 2018-07-13 06:19 寫到: On Wed, Jul 11, 2018 at 11:59:36PM +0800, Ethan Lien wrote: In commit b150a4f10d878 ("Btrfs: use a percpu to keep track of possibly pinned bytes") we use total_bytes_pinned to track how many bytes we are going to free in this transaction. When we are close t

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Qu Wenruo
On 2018年07月13日 08:20, Qu Wenruo wrote: > > > [snip] >>> In this case, it depends on when and how we mark the device resilvering. >>> If we record the generation of write error happens, then just initial a >>> scrub for generation greater than that generation. >> >>  If we record all the degrade

Re: [LKP] [lkp-robot] [mm] 9092c71bb7: blogbench.write_score -12.3% regression

2018-07-12 Thread Huang, Ying
Hi, Chris, Chris Mason writes: > On 19 Jun 2018, at 23:51, Huang, Ying wrote: "Huang, Ying" writes: > Hi, Josef, > > Do you have time to take a look at the regression? > > kernel test robot writes: > >> Greeting, >> >> FYI, we noticed a -12.3% regr

Re: Why original mode doesn't use swap? (Original: Re: btrfs check lowmem, take 2)

2018-07-12 Thread Qu Wenruo
On 2018年07月13日 07:14, Marc MERLIN wrote: > On Thu, Jul 12, 2018 at 01:26:41PM +0800, Qu Wenruo wrote: >> >> >> On 2018年07月12日 01:09, Chris Murphy wrote: >>> On Tue, Jul 10, 2018 at 12:09 PM, Marc MERLIN wrote: Thanks to Su and Qu, I was able to get my filesystem to a point that it's m

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Qu Wenruo
[snip] >> In this case, it depends on when and how we mark the device resilvering. >> If we record the generation of write error happens, then just initial a >> scrub for generation greater than that generation. > >  If we record all the degraded transactions then yes. Not just the last >  faile

Re: [PATCH v2 1/2] btrfs-progs: Rename OPEN_CTREE_FS_PARTIAL to OPEN_CTREE_TEMPORARY_SUPER

2018-07-12 Thread Christoph Anton Mitterer
Hey. Better late than never ;-) Just to confirm: At least since 4.16.1, I could btrfs-restore from the broken fs image again (that I've described in "spurious full btrfs corruption" from around mid March). So the regression in btrfsprogs has in fact been fixed by these patches, it seems. Thank

Re: Why original mode doesn't use swap? (Original: Re: btrfs check lowmem, take 2)

2018-07-12 Thread Marc MERLIN
On Thu, Jul 12, 2018 at 01:26:41PM +0800, Qu Wenruo wrote: > > > On 2018年07月12日 01:09, Chris Murphy wrote: > > On Tue, Jul 10, 2018 at 12:09 PM, Marc MERLIN wrote: > >> Thanks to Su and Qu, I was able to get my filesystem to a point that > >> it's mountable. > >> I then deleted loads of snapshot

[PATCH RESEND 2/2] btrfs-progs: make all programs and libraries optional

2018-07-12 Thread Omar Sandoval
From: Omar Sandoval We have a build system internally which only needs to build the libraries out of a repository, not any binaries. I looked at how this works with other projects, and the best example was util-linux, which makes it possible to enable or disable everything individually. This is n

[PATCH RESEND 0/2] btrfs-progs: build improvements

2018-07-12 Thread Omar Sandoval
From: Omar Sandoval Hi, Dave, This is a resend of a couple of patches I sent back in April, rebased on the current devel branch. Patch 1 cleans up some stale build targets, and patch 2 makes the btrfs-progs build more flexible, so that it's possible to pick and choose what gets built. Please con

[PATCH RESEND 1/2] btrfs-progs: remove stale dir-test and quick-test

2018-07-12 Thread Omar Sandoval
From: Omar Sandoval These don't build anymore and don't appear to be used for anything. Signed-off-by: Omar Sandoval --- Makefile | 10 +- dir-test.c | 518 --- quick-test.c | 226 -- 3 files changed, 1 insertion(+), 75

Re: [PATCH v2] btrfs: use customized batch size for total_bytes_pinned

2018-07-12 Thread Omar Sandoval
On Wed, Jul 11, 2018 at 11:59:36PM +0800, Ethan Lien wrote: > In commit b150a4f10d878 ("Btrfs: use a percpu to keep track of possibly > pinned bytes") we use total_bytes_pinned to track how many bytes we are > going to free in this transaction. When we are close to ENOSPC, we check it > and know if

Re: Transaction aborted (error -28) btrfs_run_delayed_refs*0x163/0x190

2018-07-12 Thread Pete
On 07/12/2018 07:07 PM, Pete wrote: > On 07/12/2018 08:11 AM, Nikolay Borisov wrote: >> >> >> This one shouldn't have gone RO since it has plenty of unallocated and >> free space. What was the workload at the time it went RO? Hard to say, >> it's best if you can provide output with the debug patch

Re: Transaction aborted (error -28) btrfs_run_delayed_refs*0x163/0x190

2018-07-12 Thread Pete
On 07/12/2018 08:11 AM, Nikolay Borisov wrote: > > > This one shouldn't have gone RO since it has plenty of unallocated and > free space. What was the workload at the time it went RO? Hard to say, > it's best if you can provide output with the debug patch applied when > this issue re-appears. >

Re: [PATCH v2] btrfs: use customized batch size for total_bytes_pinned

2018-07-12 Thread ethanlien
Nikolay Borisov 於 2018-07-12 15:07 寫到: On 11.07.2018 18:59, Ethan Lien wrote: In commit b150a4f10d878 ("Btrfs: use a percpu to keep track of possibly pinned bytes") we use total_bytes_pinned to track how many bytes we are going to free in this transaction. When we are close to ENOSPC, we check

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Anand Jain
On 07/12/2018 08:59 PM, Qu Wenruo wrote: On 2018年07月12日 20:33, Anand Jain wrote: On 07/12/2018 01:43 PM, Qu Wenruo wrote: On 2018年07月11日 15:50, Anand Jain wrote: BTRFS Volume operations, Device Lists and Locks all in one page: Devices are managed in two contexts, the scan context a

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Qu Wenruo
On 2018年07月12日 20:33, Anand Jain wrote: > > > On 07/12/2018 01:43 PM, Qu Wenruo wrote: >> >> >> On 2018年07月11日 15:50, Anand Jain wrote: >>> >>> >>> BTRFS Volume operations, Device Lists and Locks all in one page: >>> >>> Devices are managed in two contexts, the scan context and the mounted >>>

Re: About hung task on generic/041

2018-07-12 Thread Lu Fengqi
On Thu, Jul 12, 2018 at 11:40:54AM +0100, Filipe Manana wrote: >On Wed, Jul 11, 2018 at 10:02 AM, Lu Fengqi wrote: >> Hi, >> >> When I run generic/041 with v4.18-rc3 (turn on kasan and hung task >> detection), btrfs-transaction kthread will trigger the hung task timeout >> (stall at wait_event in

Re: [DOC] BTRFS Volume operations, Device Lists and Locks all in one page

2018-07-12 Thread Anand Jain
On 07/12/2018 01:43 PM, Qu Wenruo wrote: On 2018年07月11日 15:50, Anand Jain wrote: BTRFS Volume operations, Device Lists and Locks all in one page: Devices are managed in two contexts, the scan context and the mounted context. In scan context the threads originate from the btrfs_control io

[v10.4 2/5] btrfs-progs: dedupe: Add enable command for dedupe command group

2018-07-12 Thread Lu Fengqi
From: Qu Wenruo Add enable subcommand for dedupe commmand group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- v10.4: 1. s/btrfs-dedupe/btrfs-dedupe-inband 2. replace strerror(errno) with %m Documentation/btrfs-dedupe-inband.asciidoc | 114 +- btrfs-completion

[v10.4 1/5] btrfs-progs: Basic framework for dedupe-inband command group

2018-07-12 Thread Lu Fengqi
From: Qu Wenruo Add basic ioctl header and command group framework for later use. Alone with basic man page doc. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- v10.4: 1. s/btrfs-dedupe/btrfs-dedupe-inband in man page 2. use SZ_* instead of the intermediate number Documentation/Makefil

[v10.4 0/5] In-band de-duplication for btrfs-progs

2018-07-12 Thread Lu Fengqi
Patchset can be fetched from github: https://github.com/littleroad/btrfs-progs.git dedupe_latest Inband dedupe(in-memory backend only) ioctl support for btrfs-progs. v7 changes: Update ctree.h to follow kernel structure change Update print-tree to follow kernel structure change V8 changes:

[v10.4 4/5] btrfs-progs: dedupe: Add status subcommand

2018-07-12 Thread Lu Fengqi
From: Qu Wenruo Add status subcommand for dedupe command group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- v10.4: 1. s/btrfs-dedupe/btrfs-dedupe-inband 2. replace strerror(errno) with %m Documentation/btrfs-dedupe-inband.asciidoc | 3 + btrfs-completion

[v10.4 3/5] btrfs-progs: dedupe: Add disable support for inband dedupelication

2018-07-12 Thread Lu Fengqi
From: Qu Wenruo Add disable subcommand for dedupe command group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- v10.4: 1. s/btrfs-dedupe/btrfs-dedupe-inband 2. replace strerror(errno) with %m Documentation/btrfs-dedupe-inband.asciidoc | 5 +++ btrfs-completion

[v10.4 5/5] btrfs-progs: dedupe: introduce reconfigure subcommand

2018-07-12 Thread Lu Fengqi
From: Qu Wenruo Introduce reconfigure subcommand to co-operate with new kernel ioctl modification. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- v10.4: 1. update btrfs-completion for reconfigure 2. replace strerror(errno) with %m Documentation/btrfs-dedupe-inband.asciidoc | 7 +++ b

Re: About hung task on generic/041

2018-07-12 Thread Filipe Manana
On Wed, Jul 11, 2018 at 10:02 AM, Lu Fengqi wrote: > Hi, > > When I run generic/041 with v4.18-rc3 (turn on kasan and hung task > detection), btrfs-transaction kthread will trigger the hung task timeout > (stall at wait_event in btrfs_commit_transaction). At the same time, you > can see that xfs_i

[PATCH v2] generic: add test for fsync after cloning file range

2018-07-12 Thread fdmanana
From: Filipe Manana Test that if we do a buffered write to a file, fsync it, clone a range from another file into our file that overlaps the previously written range, fsync the file again and then power fail, after we mount again the filesystem, no file data was lost or corrupted. This test is m

[PATCH v2] Btrfs: fix file data corruption after cloning a range and fsync

2018-07-12 Thread fdmanana
From: Filipe Manana When we clone a range into a file we can end up dropping existing extent maps (or trimming them) and replacing them with new ones if the range to be cloned overlaps with a range in the destination inode. When that happens we add the new extent maps to the list of modified ext

[PATCH] generic: add test for fsync after cloning file range

2018-07-12 Thread fdmanana
From: Filipe Manana Test that if we do a buffered write to a file, fsync it, clone a range from another file into our file that overlaps the previously written range, fsync the file again and then power fail, after we mount again the filesystem, no file data was lost or corrupted. This test is m

[PATCH] Btrfs: file data corruption after cloning a range and fsync

2018-07-12 Thread fdmanana
From: Filipe Manana When we clone a range into a file we can end up dropping existing extent maps (or trimming them) and replacing them with new ones if the range to be cloned overlaps with a range in the destination inode. When that happens we add the new extent maps to the list of modified ext

Re: [PATCH 3/3] btrfs: add helper function check device delete able

2018-07-12 Thread Nikolay Borisov
On 10.07.2018 21:22, Anand Jain wrote: > Move the section of the code which performs the check if the device is > indelible, move that into a helper function. > > Signed-off-by: Anand Jain > --- > fs/btrfs/volumes.c | 49 ++--- > 1 file changed, 30

Re: [PATCH 2/3] btrfs: add helper btrfs_num_devices() to deduce num_devices

2018-07-12 Thread Nikolay Borisov
On 10.07.2018 21:22, Anand Jain wrote: > When the replace is running the fs_devices::num_devices also includes > the replace device, however in some operations like device delete and > balance it needs the actual num_devices without the repalce devices, so > now the function btrfs_num_devices()

Re: [PATCH 1/3] btrfs: warn for num_devices below 0

2018-07-12 Thread Nikolay Borisov
On 10.07.2018 21:22, Anand Jain wrote: > In preparation to de-duplicate a section of code where we deduce the > num_devices, use warn instead of bug. > > Signed-off-by: Anand Jain > --- > fs/btrfs/volumes.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/fs/btrfs/vol

Re: Transaction aborted (error -28) btrfs_run_delayed_refs*0x163/0x190

2018-07-12 Thread Nikolay Borisov
On 12.07.2018 10:04, Peter Chant wrote: > On 07/12/2018 07:10 AM, Nikolay Borisov wrote: >> >> >> On 10.07.2018 10:04, Pete wrote: >>> I've just had the error in the subject which caused the file system to >>> go read-only. >>> >>> Further part of error message: >>> WARNING: CPU: 14 PID: 1351 at

Re: [PATCH v2] btrfs: use customized batch size for total_bytes_pinned

2018-07-12 Thread Nikolay Borisov
On 11.07.2018 18:59, Ethan Lien wrote: > In commit b150a4f10d878 ("Btrfs: use a percpu to keep track of possibly > pinned bytes") we use total_bytes_pinned to track how many bytes we are > going to free in this transaction. When we are close to ENOSPC, we check it > and know if we can make the a

Re: Transaction aborted (error -28) btrfs_run_delayed_refs*0x163/0x190

2018-07-12 Thread Peter Chant
On 07/12/2018 07:10 AM, Nikolay Borisov wrote: > > > On 10.07.2018 10:04, Pete wrote: >> I've just had the error in the subject which caused the file system to >> go read-only. >> >> Further part of error message: >> WARNING: CPU: 14 PID: 1351 at fs/btrfs/extent-tree.c:3076 >> btrfs_run_delayed_r