[PATCH v2 4/4] btrfs: Refactor find_free_extent() loops update into find_free_extent_update_loop()

2018-08-21 Thread Qu Wenruo
| | |- If valid, then exit | |- Jump to next block group | |- Push harder to find free extents |- If not found, re-iterate all block groups Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 218 ++--- 1 file changed, 117 insertions(+), 101 deletions

[PATCH v2 1/4] btrfs: Introduce find_free_extent_ctrl structure for later rework

2018-08-21 Thread Qu Wenruo
-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 244 ++--- 1 file changed, 156 insertions(+), 88 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index cb4f7d1cf8b0..7bc0bdda99d4 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent

[PATCH v2 2/4] btrfs: Refactor clustered extent allocation into find_free_extent_clustered()

2018-08-21 Thread Qu Wenruo
in find_free_extent(), the helper function will use return value to indicate different behavior. Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 239 - 1 file changed, 117 insertions(+), 122 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Qu Wenruo
On 2018/8/21 下午3:14, Misono Tomohiro wrote: > On 2018/08/12 10:33, Qu Wenruo wrote: >> Provide an offline tool to disable quota. >> >> For kernel which skip_balance doesn't work, there is no way to disable >> quota on huge fs with balance, as quota will cause balance

[PATCH v2] btrfs: Always check nocow for quota enabled case to make sure we won't reserve unnecessary data space

2018-08-20 Thread Qu Wenruo
ests. Fix it by reverting to old behavior for quota enabled case. Fixes: c6887cd11149 ("Btrfs: don't do nocow check unless we have to") Signed-off-by: Qu Wenruo --- changelog v2: Fix regression for quota+cow case. (Previously it will skip data reservation if quota is enabled, causing

[PATCH v4] btrfs: locking: Add extra check in btrfs_init_new_buffer() to avoid deadlock

2018-08-20 Thread Qu Wenruo
em can only happen in crafted image, we will still trigger kernel warning for later aborted transaction, but with a little more meaningful warning message. Link: https://bugzilla.kernel.org/show_bug.cgi?id=200405 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Modify btrfs_tree

[PATCH v2.1] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-20 Thread Qu Wenruo
such invalid share to avoid possible NULL dereference as root->node can be NULL in this case. Link: https://bugzilla.kernel.org/show_bug.cgi?id=200411 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Merge the same error message output code into one block and add label for the g

[PATCH v2] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-20 Thread Qu Wenruo
such invalid share to avoid possible NULL dereference as root->node can be NULL in this case. Link: https://bugzilla.kernel.org/show_bug.cgi?id=200411 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Merge the same error message output code into one block and add label for the g

Re: [PATCH RFC] btrfs: locking: Add extra check in btrfs_init_new_buffer() to avoid deadlock

2018-08-20 Thread Qu Wenruo
On 2018/8/21 上午12:38, David Sterba wrote: > On Tue, Aug 14, 2018 at 01:51:21PM +0800, Qu Wenruo wrote: >> [BUG] >> For certains crafted image, whose csum root leaf has missing backref, if >> we try to trigger write with data csum, it could cause deadlock with the >&g

Re: [PATCH RFC] btrfs: extent-tree: refactor find_free_extent()

2018-08-20 Thread Qu Wenruo
On 2018/8/20 下午10:17, David Sterba wrote: > On Wed, Jul 25, 2018 at 04:12:42PM +0800, Qu Wenruo wrote: >> extent-tree.c::find_free_extent() could be one of the most >> ill-structured function, it has at least 4 non-exit tags and jumps >> between them. >> &g

Re: [PATCH 2/2] btrfstune: Rename change_header_uuid to change_buffer_header_uuid

2018-08-20 Thread Qu Wenruo
r(), IMHO both ways make sense. Although I have no objection against current modification. > Also remove the root argument since it's used > solely to take a reference to fs_info which can be done via the > mandatory eb argument. No functional changes.> > Signed-off-by: Nikolay B

Re: [PATCH 1/2] btrfstune: Remove fs_info arg from change_device_uuid

2018-08-20 Thread Qu Wenruo
On 2018/8/20 下午9:28, Nikolay Borisov wrote: > This function already takes an extent buffer that contains a reference > to the fs_info. Use that and reduce argument count. No functional > changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Qu Wenruo Thanks, Qu > ---

Re: Are the btrfs mount options inconsistent?

2018-08-20 Thread Qu Wenruo
On 2018/8/20 下午8:24, David Howells wrote: > David Sterba wrote: > >> No it's not. Compression needs the checksums so nodatasum should disable >> compression, which is missing as you found out. > > Thanks. > > Btw, do fs_info->mount_opt end up inscribed on disk as is? I don't see > anywhere

[PATCH] btrfs-progs: image: Rebuild dev extents for restore

2018-08-16 Thread Qu Wenruo
the whole device tree when finishing restore, so kernel or btrfs check will give no complain about restore image any more. This fixes misc/021 test case. Reported-by: Nikolay Borisov Signed-off-by: Qu Wenruo --- ctree.h | 6 +++ image/main.c | 129

Re: btrfsck out of memory for big fs

2018-08-16 Thread Qu Wenruo
data and re-do the fs ? I'd recommend to backup data asap. Thanks, Qu > > ______ > From: Qu Wenruo > Date: 2018-08-17 12:34 >

Re: BTRFS w/ quotas hangs on read-write mount using all available RAM

2018-08-16 Thread Qu Wenruo
On 2018/8/17 上午11:47, Loren M. Lang wrote: > Hello, > > I am unable to mount my btrfs in read-write mode after enabling quotas > and running a full balance on it. My service is running Ubuntu 17.10 > with Linux kernel 4.13.0-17-generic and btrfs-progs 4.12. I am trying to > recover with a

Re: btrfsck out of memory for big fs

2018-08-16 Thread Qu Wenruo
On 2018/8/17 上午10:44, litaibaich...@gmail.com wrote: > Hi Guys, > > I had a big btrfs on a md device,  it be mounted , but after a while it will > become ready only: > # btrfs fi df /data/ > Data, single: total=24.46TiB, used=24.46TiB > System, DUP: total=8.00MiB, used=2.59MiB > System,

Re: Transaction aborted - 4.16.17 kernel

2018-08-16 Thread Qu Wenruo
On 2018/8/16 下午3:51, David Goodwin wrote: > I've just spotted this on one server. > > Running : umount /backups && mount /backups > > seems to allow it to become r/w again, but it does write : > > BTRFS error (device xvdj): cleaner transaction attach returned -30 > > to 'dmesg'. > > >

Re: [PATCH v5] btrfs: qgroup: Remove qgroup items along with subvolume deletion

2018-08-15 Thread Qu Wenruo
On 2018/8/9 下午3:05, Misono Tomohiro wrote: > When qgroup is on, subvolume deletion does not remove qgroup items > of the subvolume (qgroup info, limit, relation) from quota tree and > they need to get removed manually by "btrfs qgroup destroy". > > Since level 0 qgroup cannot be used/inherited

Re: [PATCH v5] btrfs: qgroup: Remove qgroup items along with subvolume deletion

2018-08-15 Thread Qu Wenruo
On 2018/8/15 下午9:06, David Sterba wrote: > On Thu, Aug 09, 2018 at 04:05:36PM +0900, Misono Tomohiro wrote: >> When qgroup is on, subvolume deletion does not remove qgroup items >> of the subvolume (qgroup info, limit, relation) from quota tree and >> they need to get removed manually by "btrfs

[PATCH] btrfs: Always check nocow for quota enabled case to make sure we won't reserve unnecessary data space

2018-08-15 Thread Qu Wenruo
ests. Fix it by reverting to old behavior for quota enabled case. Fixes: c6887cd11149 ("Btrfs: don't do nocow check unless we have to") Signed-off-by: Qu Wenruo --- fs/btrfs/file.c | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/file.c b/f

Re: [RFC PATCH] btrfs: Remove 'objectid' member from struct btrfs_root

2018-08-14 Thread Qu Wenruo
On 2018/8/14 上午1:37, David Sterba wrote: > On Mon, Aug 06, 2018 at 02:17:54PM +0800, Qu Wenruo wrote: >>> - u64 objectid; >> >> Off topic crazy idea here. >> >> I think it is a little crazy, but it should save a lot of objectid >> related modification

[PATCH RFC] btrfs: locking: Add extra check in btrfs_init_new_buffer() to avoid deadlock

2018-08-13 Thread Qu Wenruo
em can only happen in crafted image, we will still trigger kernel warning, but with a little more meaningful warning message. Link: https://bugzilla.kernel.org/show_bug.cgi?id=200405 Reported-by: Xu Wen Signed-off-by: Qu Wenruo --- changelog: v2: Modify btrfs_tree_lock() to be able to return int

[PATCH v2 0/2] btrfs-progs: completion: Small fixes to make debug simpler

2018-08-13 Thread Qu Wenruo
trfs inspect dump-tree" can only complete dir. Make them to use _filedir() so any filename can be completed and save us developer a little time and nerve hitting that holy tab. changelog: v2: Discard _btrfs_dev() macro completely. Qu Wenruo (2): btrfs-progs: completion: Use _filedir() to re

[PATCH v2 1/2] btrfs-progs: completion: Use _filedir() to replace _btrfs_devs()

2018-08-13 Thread Qu Wenruo
mplete any filenames, no matter if it's just a file or a real block device. Signed-off-by: Qu Wenruo --- btrfs-completion | 17 + 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/btrfs-completion b/btrfs-completion index ae683f4ecf61..b7628e6984c5 100644 --- a/btrfs-comple

[PATCH v2 2/2] btrfs-progs: completion: Let dump-tree/dump-super/inode-resolve to accept any file

2018-08-13 Thread Qu Wenruo
For dump-tree/dump-super the completion uses default filedir -d, which is far from convenient. Use filedir for dump-tree/dump-super/inode-resolve just like rootid. Signed-off-by: Qu Wenruo --- btrfs-completion | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/btrfs-completion

[PATCH] fstests: btrfs: Add test for corrupted childless qgroup numbers

2018-08-12 Thread Qu Wenruo
, thus old numbers remain and cause qgroup corruption. Fixed by the following kernel patch: "btrfs: qgroup: Dirty all qgroups before rescan" Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- changelog: v2: Change the adjective for the offending group, from "orphan"

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-11 Thread Qu Wenruo
On 2018/8/12 下午12:18, Dan Merillat wrote: > On Sat, Aug 11, 2018 at 9:36 PM Qu Wenruo wrote: > >>> I'll add a new rescue subcommand, 'btrfs rescue disable-quota' for you >>> to disable quota offline. >> >> Patch set (from my work mailbox), titled "[P

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-11 Thread Qu Wenruo
On 2018/8/12 上午8:30, Qu Wenruo wrote: > > > On 2018/8/12 上午5:10, Dan Merillat wrote: >> 19 hours later, still going extremely slowly and taking longer and >> longer for progress made. Main symptom is the mount process is >> spinning at 100% CPU, interspersed with

[PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-11 Thread Qu Wenruo
Merillat Signed-off-by: Qu Wenruo --- This can patch can be fetched from github repo: https://github.com/adam900710/btrfs-progs/tree/quota_disable --- Documentation/btrfs-rescue.asciidoc | 6 +++ cmds-rescue.c | 80 + 2 files changed, 86

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-11 Thread Qu Wenruo
On 2018/8/12 上午8:59, Dan Merillat wrote: > On Sat, Aug 11, 2018 at 8:30 PM Qu Wenruo wrote: >> >> It looks pretty like qgroup, but too many noise. >> The pin point trace event would btrfs_find_all_roots(). > > I had this half-written when you replied. > > Agre

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-11 Thread Qu Wenruo
On 2018/8/12 上午5:10, Dan Merillat wrote: > 19 hours later, still going extremely slowly and taking longer and > longer for progress made. Main symptom is the mount process is > spinning at 100% CPU, interspersed with btrfs-transaction spinning at > 100% CPU. > So far it's racked up 14h45m of

Re: BUG: scheduling while atomic

2018-08-10 Thread Qu Wenruo
On 8/11/18 6:14 AM, James Courtier-Dutton wrote: > On 6 August 2018 at 07:26, Qu Wenruo wrote: >> >> >>> WARNING: CPU: 3 PID: 803 at >>> /build/linux-hwe-SYRsgd/linux-hwe-4.15.0/fs/btrfs/extent_map.c:77 >>> free_extent_map+0x78/0x90 [btrfs] >>

Re: [RFC PATCH 00/17] btrfs zoned block device support

2018-08-10 Thread Qu Wenruo
On 8/10/18 9:32 PM, Hans van Kranenburg wrote: > On 08/10/2018 09:28 AM, Qu Wenruo wrote: >> >> >> On 8/10/18 2:04 AM, Naohiro Aota wrote: >>> This series adds zoned block device support to btrfs. >>> >>> [...] >> >> And

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-10 Thread Qu Wenruo
On 8/10/18 6:42 PM, Dan Merillat wrote: > On Fri, Aug 10, 2018 at 6:05 AM, Qu Wenruo wrote: > >> >> Although not sure about the details, but the fs looks pretty huge. >> Tons of subvolume and its free space cache inodes. > > 11TB, 3 or so subvol

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-10 Thread Qu Wenruo
On 8/10/18 5:39 PM, Dan Merillat wrote: > On Fri, Aug 10, 2018 at 5:13 AM, Qu Wenruo wrote: >> >> >> On 8/10/18 4:47 PM, Dan Merillat wrote: >>> Unfortunately that doesn't appear to be it, a forced restart and >>> attempted to mount with skip_balanc

Re: [PATCH] fstests: btrfs: Add test for corrupted orphan qgroup numbers

2018-08-10 Thread Qu Wenruo
On 8/10/18 5:42 PM, Eryu Guan wrote: > On Fri, Aug 10, 2018 at 05:10:29PM +0800, Qu Wenruo wrote: >> >> >> On 8/10/18 4:54 PM, Filipe Manana wrote: >>> On Fri, Aug 10, 2018 at 9:46 AM, Qu Wenruo wrote: >>>> >>>> >>>> On 8/9/18

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-10 Thread Qu Wenruo
d like to start on other options. When umounted, would you please also try "btrfs check --readonly " to see if there is anything wrong about the fs? Thanks, Qu > > On Fri, Aug 10, 2018 at 3:59 AM, Qu Wenruo wrote: >> >> >> On 8/10/18 3:40 PM, Dan Merillat wrote:

Re: [PATCH] fstests: btrfs: Add test for corrupted orphan qgroup numbers

2018-08-10 Thread Qu Wenruo
On 8/10/18 4:54 PM, Filipe Manana wrote: > On Fri, Aug 10, 2018 at 9:46 AM, Qu Wenruo wrote: >> >> >> On 8/9/18 5:26 PM, Filipe Manana wrote: >>> On Thu, Aug 9, 2018 at 8:45 AM, Qu Wenruo wrote: >>>> This bug is exposed by populating a high level

Re: [PATCH] fstests: btrfs: Add test for corrupted orphan qgroup numbers

2018-08-10 Thread Qu Wenruo
On 8/9/18 5:26 PM, Filipe Manana wrote: > On Thu, Aug 9, 2018 at 8:45 AM, Qu Wenruo wrote: >> This bug is exposed by populating a high level qgroup, and then make it >> orphan (high level qgroup without child) > > Same comment as in the kernel patch: > > "That

Re: Mount stalls indefinitely after enabling quota groups.

2018-08-10 Thread Qu Wenruo
On 8/10/18 3:40 PM, Dan Merillat wrote: > Kernel 4.17.9, 11tb BTRFS device (md-backed, not btrfs raid) > > I was testing something out and enabled quota groups and started getting > 2-5 minute long pauses where a btrfs-transaction thread spun at 100%. Looks pretty like a running balance and

Re: Report correct filesystem usage / limits on BTRFS subvolumes with quota

2018-08-10 Thread Qu Wenruo
On 8/10/18 3:17 PM, Tomasz Pala wrote: > On Fri, Aug 10, 2018 at 07:35:32 +0800, Qu Wenruo wrote: > >>> when limiting somebody's data space we usually don't care about the >>> underlying "savings" coming from any deduplicating technique - these are >>

[PATCH] btrfs-progs: qgroup: Don't return 1 if qgroup is marked inconsistent during relationship assignment

2018-08-09 Thread Qu Wenruo
ing the return value to 0 for --no-rescan case. Signed-off-by: Qu Wenruo --- cmds-qgroup.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cmds-qgroup.c b/cmds-qgroup.c index b928edc7c408..7234bdc17bb4 100644 --- a/cmds-qgroup.c +++ b/cmds-qgroup.c @@ -123,6 +123,7 @@ static int _cmd_qgroup_ass

[PATCH v2] fstests: btrfs: Add test for corrupted childless qgroup numbers

2018-08-09 Thread Qu Wenruo
, thus old numbers remain and cause qgroup corruption. Fixed by the following kernel patch: "btrfs: qgroup: Dirty all qgroups before rescan" Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- changelog: v2: Change the adjective for the offending group, from "orphan&q

[PATCH v3] btrfs: qgroup: Dirty all qgroups before rescan

2018-08-09 Thread Qu Wenruo
ss qgroups, their QGROUP_INFO items will still get updated during rescan. Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo Reviewed-by: Misono Tomohiro Tested-by: Misono Tomohiro --- changelog: v2: Fix some grammar errors in commit message. v3: Fix the adjective used for qgroups without chil

Re: Report correct filesystem usage / limits on BTRFS subvolumes with quota

2018-08-09 Thread Qu Wenruo
On 8/10/18 1:48 AM, Tomasz Pala wrote: > On Tue, Jul 31, 2018 at 22:32:07 +0800, Qu Wenruo wrote: > >> 2) Different limitations on exclusive/shared bytes >>Btrfs can set different limit on exclusive/shared bytes, further >>complicating the problem. >> &

Re: [PATCH v2] btrfs: qgroup: Dirty all qgroups before rescan

2018-08-09 Thread Qu Wenruo
On 8/9/18 5:17 PM, Filipe Manana wrote: > On Thu, Aug 9, 2018 at 8:08 AM, Qu Wenruo wrote: >> [BUG] >> In the following case, rescan won't zero out the number of qgroup 1/0: >> -- >> $ mkfs.btrfs -fq $DEV >> $ mount $DEV /mnt >> >> $ btrfs quota

[PATCH] fstests: btrfs: Add test for corrupted orphan qgroup numbers

2018-08-09 Thread Qu Wenruo
, their on-disk data is not updated, thus old numbers remain and cause qgroup corruption. Fixed by the following kernel patch: "btrfs: qgroup: Dirty all qgroups before rescan" Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- tests/btrfs/17

Re: [PATCH] btrfs: qgroup: Dirty all qgroups before rescan

2018-08-09 Thread Qu Wenruo
On 8/9/18 3:05 PM, Nikolay Borisov wrote: > > > On 9.08.2018 10:03, Qu Wenruo wrote: >> [BUG] >> In the following case, rescan won't zero out the number of qgroup 1/0: >> -- >> $ mkfs.btrfs -fq $DEV >> $ mount $DEV /mnt >> >> $ btrfs

[PATCH v2] btrfs: qgroup: Dirty all qgroups before rescan

2018-08-09 Thread Qu Wenruo
ave orphan qgroups their QGROUP_INFO items will still get updated during rescan. Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- changelog: v2: Fix some grammar errors in commit message. --- fs/btrfs/qgroup.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrf

[PATCH] btrfs: qgroup: Dirty all qgroups before rescan

2018-08-09 Thread Qu Wenruo
an qgroups their QGROUP_INFO item will still get updated during rescan. Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- fs/btrfs/qgroup.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 48c1c3e7baf3..5a5372b33d96 100644 --- a/fs/btrfs/qgroup.

Re: [PATCH v4] btrfs: qgroup: Remove qgroup items along with subvolume deletion

2018-08-09 Thread Qu Wenruo
On 8/9/18 2:05 PM, Misono Tomohiro wrote: > On 2018/08/09 14:47, Qu Wenruo wrote: >> >> >> On 8/9/18 12:12 PM, Misono Tomohiro wrote: >>> When qgroup is on, subvolume deletion does not remove qgroup items >>> of the subvolume (qgroup info, limit, rela

Re: [PATCH] btrfs: qgroup: Don't populating excl numbers for snapshot src if it belongs to other qgroups

2018-08-08 Thread Qu Wenruo
On 8/8/18 3:48 PM, Qu Wenruo wrote: > > > On 2018年08月08日 15:41, Misono Tomohiro wrote: >> On 2018/08/08 15:04, Qu Wenruo wrote: >>> When quota is enabled and we do a snapshot, we just update the 'excl' >>> number of both snapshot src and dst to src's 'rfer'

Re: [PATCH v4] btrfs: qgroup: Remove qgroup items along with subvolume deletion

2018-08-08 Thread Qu Wenruo
his problem last time I reviewed this patch) Thanks, Qu > In this case manual qgroup rescan is needed. > > Reviewed-by: Lu Fengqi > Reviewed-by: Qu Wenruo > Signed-off-by: Misono Tomohiro > --- > Hi David, > It turned out that this patch may cause qgroup inconsistency i

Re: [PATCH] btrfs: qgroup: Always remove all qgroup relation in btrfs_remove_qgroup()

2018-08-08 Thread Qu Wenruo
iro Indeed we still need to remove all qgroup relationship even we will make qgroup inconsistent. Reviewed-by: Qu Wenruo Thanks, Qu > --- > fs/btrfs/qgroup.c | 15 --- > 1 file changed, 12 insertions(+), 3 deletions(-) > > diff --git a/fs/btrfs/qgroup.c b/fs/btrfs

Re: [PATCH] btrfs: qgroup: Don't populating excl numbers for snapshot src if it belongs to other qgroups

2018-08-08 Thread Qu Wenruo
On 2018年08月08日 15:41, Misono Tomohiro wrote: > On 2018/08/08 15:04, Qu Wenruo wrote: >> When quota is enabled and we do a snapshot, we just update the 'excl' >> number of both snapshot src and dst to src's 'rfer' - nodesize. >> >> It's a quick hack to avoid quota

Re: [RFC PATCH 0/4] undelete subvolume online version

2018-08-08 Thread Qu Wenruo
On 2018年08月08日 14:53, Lu Fengqi wrote: > On Wed, Aug 08, 2018 at 02:11:24PM +0800, Qu Wenruo wrote: >> >> >> On 2018年08月08日 00:39, David Sterba wrote: >>> On Sun, Aug 05, 2018 at 06:39:57PM +0800, Lu Fengqi wrote: >>>> This patchset will add the

Re: [RFC PATCH 0/4] undelete subvolume online version

2018-08-08 Thread Qu Wenruo
On 2018年08月08日 00:39, David Sterba wrote: > On Sun, Aug 05, 2018 at 06:39:57PM +0800, Lu Fengqi wrote: >> This patchset will add the BTRFS_IOC_SUBVOL_UNDELETE ioctl for online >> btrfs subvolume undelete. >> >> And btrfs subvolume undelete subcommand was added to btrfs-progs. >> >> So user can

[PATCH] btrfs: qgroup: Don't populating excl numbers for snapshot src if it belongs to other qgroups

2018-08-08 Thread Qu Wenruo
rescan to do its job. Reported-by: Misono Tomohiro Signed-off-by: Qu Wenruo --- fs/btrfs/qgroup.c | 16 1 file changed, 16 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index ec4351fd7537..2b3d2dd1b735 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c

Re: [PATCH v3] btrfs: qgroup: Remove qgroup items along with subvolume deletion

2018-08-07 Thread Qu Wenruo
to be precise, when the subvolume root is dropped). >>> >>> Reviewed-by: Lu Fengqi >>> Reviewed-by: Qu Wenruo >>> Signed-off-by: Misono Tomohiro >>> --- >>> v2 -> v3: >>> Use root->root_key.objectid instead of root->obj

[PATCH 11/11] btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

2018-08-07 Thread Qu Wenruo
Nothing interesting, since such combination can be handled easily by qgroup-verify. Signed-off-by: Qu Wenruo --- .../mkfs-tests/018-rootdir-with-quota/test.sh | 51 +++ 1 file changed, 51 insertions(+) create mode 100755 tests/mkfs-tests/018-rootdir-with-quota/test.sh diff

[PATCH 10/11] btrfs-progs: test/mkfs: Add test case for -Q|--quota option

2018-08-07 Thread Qu Wenruo
Only test if btrfs check (which will check qgroup by default) and kernel mount success. Comprehensive qgroup test cases still belongs to fstests. Signed-off-by: Qu Wenruo --- tests/mkfs-tests/001-basic-profiles/test.sh | 10 ++ 1 file changed, 10 insertions(+) diff --git a/tests/mkfs

[PATCH 09/11] btrfs-progs: mkfs: Introduce mkfs time quota support

2018-08-07 Thread Qu Wenruo
The result fs will has quota enabled, with consistent qgroup accounting. This is quite handy to test quota with fstests, which doesn't support to call ioctl for btrfs at mount time. Signed-off-by: Qu Wenruo --- Documentation/mkfs.btrfs.asciidoc | 5 + mkfs/main.c

[PATCH 08/11] btrfs-progs: mkfs: Introduce function to setup quota root and rescan

2018-08-07 Thread Qu Wenruo
Introduce a new function, setup_quota_root(), which will create quota root, and do an offline rescan to ensure all quota accounting numbers are correct. Signed-off-by: Qu Wenruo --- mkfs/main.c | 86 + 1 file changed, 86 insertions(+) diff

[PATCH v3 00/11] btrfs-progs: mkfs: Quota support through -Q|--quota

2018-08-07 Thread Qu Wenruo
option. Rebased to v4.16. v3: Revert to -Q|--quota single option interface, since data reloc tree and uuid tree creation is already supported by mkfs. Rebased to v4.17.1. Add a new patch to cooperate with enhanced progress indicator. Qu Wenruo (11): btrfs-progs: qgroup-verify: Avoid NULL

[PATCH 03/11] btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we should repair qgroups

2018-08-07 Thread Qu Wenruo
In fact qgroup-verify is just kind of offline qgroup rescan, and later mkfs qgroup support will reuse it. So qgroup-verify doesn't really need to rely the global variable @repair to check if it should repair qgroups. Instead check fs_info->readonly to do the repair. Signed-off-by: Qu Wen

[PATCH 06/11] btrfs-progs: ctree: Introduce function to create an empty tree

2018-08-07 Thread Qu Wenruo
easily This patch provides the basis to implement quota support in mkfs. Signed-off-by: Qu Wenruo --- ctree.c | 109 ctree.h | 2 ++ 2 files changed, 111 insertions(+) diff --git a/ctree.c b/ctree.c index d8a6883aa85f..96eef9765e82 100644

[PATCH 05/11] btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent repair

2018-08-07 Thread Qu Wenruo
Allow repair_qgroups() to do silent repair, so it can acts as offline qgroup rescan. This provides the basis for later mkfs quota support. Signed-off-by: Qu Wenruo --- check/main.c| 2 +- qgroup-verify.c | 19 +++ qgroup-verify.h | 2 +- 3 files changed, 13 insertions

[PATCH 07/11] btrfs-progs: mkfs: Introduce function to insert qgroup info and limit items

2018-08-07 Thread Qu Wenruo
Introduce a new function, insert_qgroup_items(), to insert qgroup info item and qgroup limit item for later mkfs qgroup support. Signed-off-by: Qu Wenruo --- mkfs/main.c | 34 ++ 1 file changed, 34 insertions(+) diff --git a/mkfs/main.c b/mkfs/main.c index

[PATCH 04/11] btrfs-progs: qgroup-verify: Move qgroup classification out of report_qgroups

2018-08-07 Thread Qu Wenruo
. Signed-off-by: Qu Wenruo --- check/main.c| 18 +++--- qgroup-verify.c | 62 ++--- qgroup-verify.h | 2 +- 3 files changed, 58 insertions(+), 24 deletions(-) diff --git a/check/main.c b/check/main.c index bc2ee22f7943..9ed102c197ab 100644

[PATCH 01/11] btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later silent qgroup repair

2018-08-07 Thread Qu Wenruo
ULL before accessing it. Signed-off-by: Qu Wenruo --- qgroup-verify.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/qgroup-verify.c b/qgroup-verify.c index f5885589f02c..9b509abb78fd 100644 --- a/qgroup-verify.c +++ b/qgroup-verify.c @@ -743,7 +743,8 @@ static int travel_t

[PATCH 02/11] btrfs-progs: qgroup-verify: Also repair qgroup status version

2018-08-07 Thread Qu Wenruo
Current kernel only supports qgroup version 1. Make qgroup-verify to follow this standard. Signed-off-by: Qu Wenruo --- ctree.h | 1 + qgroup-verify.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/ctree.h b/ctree.h index 4719962df67d..61c3624ff5db 100644 --- a/ctree.h +++ b

Re: [PATCH v2 00/12] mkfs: Quota support through -R|--runtime quota

2018-08-07 Thread Qu Wenruo
On 2018年08月07日 15:02, Lu Fengqi wrote: > On Mon, Jul 30, 2018 at 01:03:00PM +0800, Qu Wenruo wrote: >> Ping the 3rd time? >> >> Or should I just rebase the patchset? > > Hi Qu > > Could you rebase this patchset? Because I want to test existing test cases > w

Re: BUG: scheduling while atomic

2018-08-06 Thread Qu Wenruo
On 2018年08月06日 14:07, James Courtier-Dutton wrote: > > > On Mon, 6 Aug 2018, 01:48 Qu Wenruo, <mailto:quwenruo.bt...@gmx.com>> wrote: > > > > On 2018年08月05日 22:09, James Courtier-Dutton wrote: > > I am seeing a server halt and require a manua

Re: [RFC PATCH] btrfs: Remove 'objectid' member from struct btrfs_root

2018-08-06 Thread Qu Wenruo
t a small nitpick about __setup_root() inlined later. (And a personal crazy idea no need to address) > > Signed-off-by: Misono Tomohiro Feel free to add my tag: Reviewed-by: Qu Wenruo > --- > Although being fundamentally independent, this is based on the > patch: https://patch

[PATCH 0/3] btrfs-progs: Minor qgroup subcommand usage update

2018-08-06 Thread Qu Wenruo
s it makes more sense to make --rescan as default behavior, and add more explanation on why quota rescan is needed, and under which case we can skip the rescan. Qu Wenruo (3): btrfs-progs: cmds-qgroup: Use bool to replace int for @rescan btrfs-progs: qgroup: make --rescan as the defaul

[PATCH 2/3] btrfs-progs: qgroup: make --rescan as the default behavior for assign

2018-08-06 Thread Qu Wenruo
Even though we have --rescan option, it still needs user to manually specify it. However even some developer is not aware of the option, it makes even less sense for end-user. So make it the default behavior. Signed-off-by: Qu Wenruo --- Documentation/btrfs-qgroup.asciidoc | 4 ++-- cmds

[PATCH 3/3] btrfs-progs: Doc: Update btrfs-qgroup for the rescan condition

2018-08-06 Thread Qu Wenruo
Add a new section, RESCAN CONDITION, to explain why and when we need a full quota rescan when assigning/removing qgroup relationship. Also, since 'remove' shares the same options of 'assign', add reference to 'assign' options for 'remove' subcommand. Signed-off-by: Qu Wenruo --- Documentation

[PATCH 1/3] btrfs-progs: cmds-qgroup: Use bool to replace int for @rescan

2018-08-06 Thread Qu Wenruo
Just a cleanup to avoid abuse of int. Signed-off-by: Qu Wenruo --- cmds-qgroup.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cmds-qgroup.c b/cmds-qgroup.c index 93206900693d..7a505443d271 100644 --- a/cmds-qgroup.c +++ b/cmds-qgroup.c @@ -40,7 +40,7 @@ static int

Re: BUG: scheduling while atomic

2018-08-05 Thread Qu Wenruo
On 2018年08月05日 22:09, James Courtier-Dutton wrote: > I am seeing a server halt and require a manual restart that I think > might be related to btrfs. > I attach the kernel log from it, in the hope that someone will > understand it better than me. > Any clues? > >

Re: [PATCH 1/2] btrfs-progs: completion: Use _filedir() to replace _btrfs_devs()

2018-08-03 Thread Qu Wenruo
On 2018年08月03日 22:00, David Sterba wrote: > On Tue, Jul 17, 2018 at 02:15:31PM +0800, Qu Wenruo wrote: >> For developers it's pretty common to call "btrfs check" on a raw image >> dump other than real block device. > > I'd be less concerned about deve

Re: [PATCH v2] btrfs: qgroup: Remove qgroup item along with subvolume deletion

2018-08-03 Thread Qu Wenruo
lume, >>>> let's remove them automatically when subvolume is deleted >>>> (to be precise, when the subvolume root is dropped). >>>> >>>> Signed-off-by: Misono Tomohiro >>> >>> Looks good to me. >>> >>> Reviewed-by: Lu Fengq

[PATCH 1/6] btrfs-progs: Exit gracefully if we hit ENOSPC when allocating tree block

2018-08-02 Thread Qu Wenruo
-offset.raw.restored mayfail: returned code 134 (SIGABRT), not ignored test failed for case 003-multi-check-unmounted -- Just remove that BUG_ON() and allow us to exit gracefully. Signed-off-by: Qu Wenruo --- extent-tree.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/extent

[PATCH 0/6] btrfs-progs: Variant fixes for fuzz-tests

2018-08-02 Thread Qu Wenruo
This can be fetched from github: https://github.com/adam900710/btrfs-progs/tree/fixes_for_fuzz_test The base HEAD is: commit d7a1b84756157d544a9ddc399ef48c6132eaafcf (david/devel) Author: Qu Wenruo Date: Thu Jul 5 15:37:31 2018 +0800 btrfs-progs: check/original: Don't overwrite return

[PATCH 2/6] btrfs-progs: Exit gracefully when failed to repair root dir item

2018-08-02 Thread Qu Wenruo
/btrfs check --init-csum-tree /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-161821.raw.restored mayfail: returned code 134 (SIGABRT), not ignored test failed for case 003-multi-check-unmounted -- Just abort current transaction and exit gracefully in this case. Signed-off-by: Qu

[PATCH 3/6] btrfs-progs: Don't report dirty leaked eb using BUG_ON

2018-08-02 Thread Qu Wenruo
i-check-unmounted -- Since we're shifting to using btrfs_abort_transaction() in btrfs-progs, it will be more and more common to see dirty leaked eb. Instead of BUG_ON(), we only needs to report it as warning. Signed-off-by: Qu Wenruo --- extent_io.c | 6 +- 1 file changed, 5 insertion

[PATCH 4/6] btrfs-progs: Fix infinite loop when failed to repair bad key order

2018-08-02 Thread Qu Wenruo
referring to the bad tree block, it won't cause infinite loop any more. Signed-off-by: Qu Wenruo --- check/main.c | 5 + 1 file changed, 5 insertions(+) diff --git a/check/main.c b/check/main.c index e6756e0f852b..571edc7ef928 100644 --- a/check/main.c +++ b/check/main.c @@ -4051,6 +4051,11 @@ s

[PATCH 6/6] btrfs-progs: rescue-super: Don't double free fs_devices

2018-08-02 Thread Qu Wenruo
ult when exiting. Signed-off-by: Qu Wenruo --- super-recover.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/super-recover.c b/super-recover.c index 880fd7712546..86b3df9867dc 100644 --- a/super-recover.c +++ b/super-recover.c @@ -292,9 +292,6 @@ int btrfs_recover_superblocks(const char *dn

[PATCH 5/6] btrfs-progs: Exit gracefull when we failed to alloc dev extent

2018-08-02 Thread Qu Wenruo
IGABRT), not ignored test failed for case 003-multi-check-unmounted -- However the culprit function btrfs_alloc_dev_extent() has proper error handler tag err:, just use that tag would solve the problem easily. Signed-off-by: Qu Wenruo --- volumes.c | 3 ++- 1 file changed, 2 insertions(+), 1 de

Re: btrfs check --repair questions

2018-08-02 Thread Qu Wenruo
On 2018年08月03日 12:55, Nathan Dehnel wrote: > Question 1: The command has been running for a couple days on a 10TB > array with no visible change in output. Is it actually fixing anything? > > enabling repair mode > Checking filesystem on > /dev/disk/by-uuid/e1ee5980-c54b-4b6e-82e2-3dbdcee1dd24

Re: [PATCH] btrfs: qgroup: Remove qgroup item along with subvolume deletion

2018-08-02 Thread Qu Wenruo
On 2018年08月03日 12:08, Misono Tomohiro wrote: > When qgroup is on, subvolume deletion does not remove qgroup items > of the subvolume (qgroup info, limits, relation) from quota tree and > they need to get removed manually by "btrfs qgroup destroy". > > Since level 0 qgroup cannot be

Re: [PATCH v2 0/6] btrfs: Enhanced validation check for fuzzed images

2018-08-02 Thread Qu Wenruo
On 2018年08月03日 00:40, David Sterba wrote: > On Wed, Aug 01, 2018 at 10:37:15AM +0800, Qu Wenruo wrote: >> The branch can be fetched from the following git repo: >> https://github.com/adam900710/linux/tree/tree_checker_enhance >> >> It's based on v4.18-rc1, with 3 patch

Re: BTRFS and databases

2018-08-02 Thread Qu Wenruo
On 2018年08月02日 18:45, Andrei Borzenkov wrote: > > > Отправлено с iPhone > >> 2 авг. 2018 г., в 10:02, Qu Wenruo написал(а): >> >> >> >>> On 2018年08月01日 11:45, MegaBrutal wrote: >>> Hi all, >>> >>> I know it's a decade-

Re: BTRFS and databases

2018-08-02 Thread Qu Wenruo
On 2018年08月02日 18:35, Andrei Borzenkov wrote: > > > Отправлено с iPhone > >> 2 авг. 2018 г., в 12:16, Martin Steigerwald написал(а): >> >> Hugo Mills - 01.08.18, 10:56: On Wed, Aug 01, 2018 at 05:45:15AM +0200, MegaBrutal wrote: I know it's a decade-old question, but I'd like to

Re: [PATCH] btrfs: Use wrapper macro for rcu string to remove duplicate code

2018-08-02 Thread Qu Wenruo
On 2018年08月02日 15:19, Misono Tomohiro wrote: > Cleanup patch and no functional changes. > > Signed-off-by: Misono Tomohiro Looks good to me. Reviewed-by: Qu Wenruo Thanks, Qu > --- > fs/btrfs/ioctl.c | 6 ++ > fs/btrfs/scrub.c | 8 ++-- > fs

Re: [PATCH v2] btrfs: locking: Allow btrfs_tree_lock() to return error to avoid deadlock

2018-08-02 Thread Qu Wenruo
On 2018年08月01日 21:45, Qu Wenruo wrote: > > > On 2018年08月01日 21:27, David Sterba wrote: >> On Tue, Jul 31, 2018 at 02:52:23PM +0800, Qu Wenruo wrote: >>> [BUG] >>> For certains crafted image, whose csum root leaf has missing backref, if >>> we try to tr

Re: BTRFS and databases

2018-08-02 Thread Qu Wenruo
On 2018年08月01日 22:33, Remi Gauvin wrote: > On 2018-07-31 11:45 PM, MegaBrutal wrote: > >> I know that with nodatacow, I take away most of the benefits of BTRFS >> (those are actually hurting database performance – the exact CoW >> nature that is elsewhere a blessing, with databases it's a

Re: BTRFS and databases

2018-08-02 Thread Qu Wenruo
On 2018年08月01日 11:45, MegaBrutal wrote: > Hi all, > > I know it's a decade-old question, but I'd like to hear your thoughts > of today. By now, I became a heavy BTRFS user. Almost everywhere I use > BTRFS, except in situations when it is obvious there is no benefit > (e.g. /var/log, /boot). At

Re: [PATCH v2] btrfs: locking: Allow btrfs_tree_lock() to return error to avoid deadlock

2018-08-01 Thread Qu Wenruo
On 2018年08月01日 21:27, David Sterba wrote: > On Tue, Jul 31, 2018 at 02:52:23PM +0800, Qu Wenruo wrote: >> [BUG] >> For certains crafted image, whose csum root leaf has missing backref, if >> we try to trigger write with data csum, it could cause deadlock with the >&g

Re: [PATCH 1/1] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-01 Thread Qu Wenruo
On 2018年08月01日 20:12, Nikolay Borisov wrote: > > > On 1.08.2018 14:13, Qu Wenruo wrote: >> >> >> On 2018年08月01日 18:08, Nikolay Borisov wrote: >>> >>> >>> On 1.08.2018 11:08, Qu Wenruo wrote: >>>> [BUG] >>>> Whe

<    1   2   3   4   5   6   7   8   9   10   >