[PATCH 2/2] btrfs-progs: qgroup limit: add a check for invalid input of 'T/G/M/K'
Add a check to error out in the following case:
# ./btrfs qgroup limit T /mnt/
Invalid size argument given
Without this patch, btrfs-progs would parse the input as 0
and continue.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 5ea4021..9545a20 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -121,6 +121,9 @@ static int parse_limit(const char *p, unsigned long long *s)
return 0;
size = strtoull(p, endptr, 10);
+ if (p == endptr)
+ return 0;
+
switch (*endptr) {
case 'T':
case 't':
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] btrfs-progs: qgroup: show 'none' when we did not limit it on this qgroup
There are two understanding of the '0' value in btrfs qgroup show.
(1) is no-limitation on this qgroup. (2) is the max-limitation is 0.
This patch make it showing in different way.
(1). max-limitation for 0 is still showing '0'.
(2). no-limitation will show 'none'.
qgroupid rfer excl max_rfer max_excl parent
--
0/5 2.19GiB 2.19GiB none none ---
0/257 100.02MiB100.02MiB none none ---
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
qgroup.c | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/qgroup.c b/qgroup.c
index 53815b5..dc04b03 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -237,10 +237,16 @@ static void print_qgroup_column(struct btrfs_qgroup
*qgroup,
print_qgroup_column_add_blank(BTRFS_QGROUP_PARENT, len);
break;
case BTRFS_QGROUP_MAX_RFER:
- len = printf(%*s, max_len, pretty_size_mode(qgroup-max_rfer,
unit_mode));
+ if (qgroup-flags BTRFS_QGROUP_LIMIT_MAX_RFER)
+ len = printf(%*s, max_len,
pretty_size_mode(qgroup-max_rfer, unit_mode));
+ else
+ len = printf(%*s, max_len, none);
break;
case BTRFS_QGROUP_MAX_EXCL:
- len = printf(%*s, max_len, pretty_size_mode(qgroup-max_excl,
unit_mode));
+ if (qgroup-flags BTRFS_QGROUP_LIMIT_MAX_EXCL)
+ len = printf(%*s, max_len,
pretty_size_mode(qgroup-max_excl, unit_mode));
+ else
+ len = printf(%*s, max_len, none);
break;
case BTRFS_QGROUP_CHILD:
len = print_child_column(qgroup);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] btrfs-progs: qgroup: allow user to clear some limitation on qgroup.
Currently, we can not clear a limitation on a qgroup. Although
there is a 'none' choice provided to user to do it, it does not
work well.
It does not set the flag which user want to clear, then kernel
will never know what the user want to do at all.
*Without this commit*
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
# ./btrfs qgroup limit none /mnt
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
This patch will set the flag user want to clear and pass a
size=-1 to kernel. Then kernel will clear it correctly.
*With this commit*
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
# ./btrfs qgroup limit none /mnt
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB none none
0/257 100.02MiB100.02MiB none none
Reported-by: Tsutomu Itoh t-i...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 23 +++
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index b073250..00cc089 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -110,9 +110,10 @@ static int parse_limit(const char *p, unsigned long long
*s)
{
char *endptr;
unsigned long long size;
+ unsigned long long CLEAR_VALUE = -1;
if (strcasecmp(p, none) == 0) {
- *s = 0;
+ *s = CLEAR_VALUE;
return 1;
}
size = strtoull(p, endptr, 10);
@@ -406,17 +407,15 @@ static int cmd_qgroup_limit(int argc, char **argv)
}
memset(args, 0, sizeof(args));
- if (size) {
- if (compressed)
- args.lim.flags |= BTRFS_QGROUP_LIMIT_RFER_CMPR |
- BTRFS_QGROUP_LIMIT_EXCL_CMPR;
- if (exclusive) {
- args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_EXCL;
- args.lim.max_exclusive = size;
- } else {
- args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_RFER;
- args.lim.max_referenced = size;
- }
+ if (compressed)
+ args.lim.flags |= BTRFS_QGROUP_LIMIT_RFER_CMPR |
+ BTRFS_QGROUP_LIMIT_EXCL_CMPR;
+ if (exclusive) {
+ args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_EXCL;
+ args.lim.max_exclusive = size;
+ } else {
+ args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_RFER;
+ args.lim.max_referenced = size;
}
if (argc - optind == 2) {
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: qgroup: allow user to clear the limitation on qgroup
Currently, we can only set a limitation on a qgroup, but we
can not clear it.
This patch provide a choice to user to clear a limitation on
qgroup by passing a value of CLEAR_VALUE(-1) to kernel.
Reported-by: Tsutomu Itoh t-i...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 49 +
1 file changed, 41 insertions(+), 8 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 3d65465..0412d5b 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1317,6 +1317,11 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
struct btrfs_root *quota_root;
struct btrfs_qgroup *qgroup;
int ret = 0;
+ /* Sometimes we would want to clear the limit on this qgroup.
+* To meet this requirement, we treat the -1 as a special value
+* which tell kernel to clear the limit on this qgroup.
+*/
+ const u64 CLEAR_VALUE = -1;
mutex_lock(fs_info-qgroup_ioctl_lock);
quota_root = fs_info-quota_root;
@@ -1332,14 +1337,42 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
}
spin_lock(fs_info-qgroup_lock);
- if (limit-flags BTRFS_QGROUP_LIMIT_MAX_RFER)
- qgroup-max_rfer = limit-max_rfer;
- if (limit-flags BTRFS_QGROUP_LIMIT_MAX_EXCL)
- qgroup-max_excl = limit-max_excl;
- if (limit-flags BTRFS_QGROUP_LIMIT_RSV_RFER)
- qgroup-rsv_rfer = limit-rsv_rfer;
- if (limit-flags BTRFS_QGROUP_LIMIT_RSV_EXCL)
- qgroup-rsv_excl = limit-rsv_excl;
+ if (limit-flags BTRFS_QGROUP_LIMIT_MAX_RFER) {
+ if (limit-max_rfer == CLEAR_VALUE) {
+ qgroup-lim_flags = ~BTRFS_QGROUP_LIMIT_MAX_RFER;
+ limit-flags = ~BTRFS_QGROUP_LIMIT_MAX_RFER;
+ qgroup-max_rfer = 0;
+ } else {
+ qgroup-max_rfer = limit-max_rfer;
+ }
+ }
+ if (limit-flags BTRFS_QGROUP_LIMIT_MAX_EXCL) {
+ if (limit-max_excl == CLEAR_VALUE) {
+ qgroup-lim_flags = ~BTRFS_QGROUP_LIMIT_MAX_EXCL;
+ limit-flags = ~BTRFS_QGROUP_LIMIT_MAX_EXCL;
+ qgroup-max_excl = 0;
+ } else {
+ qgroup-max_excl = limit-max_excl;
+ }
+ }
+ if (limit-flags BTRFS_QGROUP_LIMIT_RSV_RFER) {
+ if (limit-rsv_rfer == CLEAR_VALUE) {
+ qgroup-lim_flags = ~BTRFS_QGROUP_LIMIT_RSV_RFER;
+ limit-flags = ~BTRFS_QGROUP_LIMIT_RSV_RFER;
+ qgroup-rsv_rfer = 0;
+ } else {
+ qgroup-rsv_rfer = limit-rsv_rfer;
+ }
+ }
+ if (limit-flags BTRFS_QGROUP_LIMIT_RSV_EXCL) {
+ if (limit-rsv_excl == CLEAR_VALUE) {
+ qgroup-lim_flags = ~BTRFS_QGROUP_LIMIT_RSV_EXCL;
+ limit-flags = ~BTRFS_QGROUP_LIMIT_RSV_EXCL;
+ qgroup-rsv_excl = 0;
+ } else {
+ qgroup-rsv_excl = limit-rsv_excl;
+ }
+ }
qgroup-lim_flags |= limit-flags;
spin_unlock(fs_info-qgroup_lock);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] btrfs-progs: qgroup limit: error out if input value is negative
If we pass a negative value to command qgroup limit, btrfs-progs
would convert it to unsigned long long silently. That's a little
confusing to user, why I can limit my quota to a negative value.
This patch add a check in parse_limit, if the input value is negative,
error out to user.
Reported-by: Tsutomu Itoh t-i...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 4
1 file changed, 4 insertions(+)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 00cc089..5ea4021 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -116,6 +116,10 @@ static int parse_limit(const char *p, unsigned long long
*s)
*s = CLEAR_VALUE;
return 1;
}
+
+ if (p[0] == '-')
+ return 0;
+
size = strtoull(p, endptr, 10);
switch (*endptr) {
case 'T':
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.
When we enable quota, btrfs will rescan quota numbers. We need
to wait the rescan finished before any more operations on btrfs
qgroups. Otherwith, the new btrfs-progs would WARN out:
WARNING: Rescan is running, qgroup data may be incorrect.
It would make btrfs/022 failed.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
tests/btrfs/022 | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/btrfs/022 b/tests/btrfs/022
index 5c1a82d..56d4f3d 100755
--- a/tests/btrfs/022
+++ b/tests/btrfs/022
@@ -51,6 +51,7 @@ _basic_test()
{
_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
+ _run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid \
$seqres.full 21
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.
Hi Filip,
Qu is off duty this week, would you please try this patch here?
Thanx
Yang
On 06/03/2015 03:10 PM, Dongsheng Yang wrote:
When we enable quota, btrfs will rescan quota numbers. We need
to wait the rescan finished before any more operations on btrfs
qgroups. Otherwith, the new btrfs-progs would WARN out:
WARNING: Rescan is running, qgroup data may be incorrect.
It would make btrfs/022 failed.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
tests/btrfs/022 | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/btrfs/022 b/tests/btrfs/022
index 5c1a82d..56d4f3d 100755
--- a/tests/btrfs/022
+++ b/tests/btrfs/022
@@ -51,6 +51,7 @@ _basic_test()
{
_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
+ _run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid \
$seqres.full 21
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] btrfs-progs: qgroup: allow user to clear some limitation on qgroup.
On 06/03/2015 04:40 PM, Tsutomu Itoh wrote:
On 2015/06/03 15:57, Dongsheng Yang wrote:
Currently, we can not clear a limitation on a qgroup. Although
there is a 'none' choice provided to user to do it, it does not
work well.
It does not set the flag which user want to clear, then kernel
will never know what the user want to do at all.
*Without this commit*
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
# ./btrfs qgroup limit none /mnt
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
This patch will set the flag user want to clear and pass a
size=-1 to kernel. Then kernel will clear it correctly.
*With this commit*
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB 5.00GiB none
0/257 100.02MiB100.02MiB none none
# ./btrfs qgroup limit none /mnt
# ./btrfs qgroup show -re /mnt
qgroupid rfer excl max_rfer max_excl
0/5 2.19GiB 2.19GiB none none
0/257 100.02MiB100.02MiB none none
Reported-by: Tsutomu Itoh t-i...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 23 +++
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index b073250..00cc089 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -110,9 +110,10 @@ static int parse_limit(const char *p, unsigned long
long *s)
{
char *endptr;
unsigned long long size;
+unsigned long long CLEAR_VALUE = -1;
Even if a negative value is specified, it doesn't become an error...
So, it doesn't make an error of the following command.
# btrfs qg lim -- -1 sub1
# btrfs qg show -prce /test1
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB none none --- ---
0/259 8.86GiB 7.88GiB none none --- ---
# btrfs qg lim -- -2 sub1
# btrfs qg show -prce /test1
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB none none --- ---
0/259 8.86GiB 7.88GiB 16.00EiB none --- ---
Agreed, I understand your point here.
If we pass a negative value to limit command, we should Error out. But
currently, btrfs-progs are treating it as a unsigned value in parsing
it. Yes, that's a bit of confusing.
We need to cover it in parsing steps with another patch I think.
Otherwise OK,
Tested-by: Tsutomu Itoh t-i...@jp.fujitsu.com
Thanx
Yang
if (strcasecmp(p, none) == 0) {
-*s = 0;
+*s = CLEAR_VALUE;
return 1;
}
size = strtoull(p, endptr, 10);
@@ -406,17 +407,15 @@ static int cmd_qgroup_limit(int argc, char **argv)
}
memset(args, 0, sizeof(args));
-if (size) {
-if (compressed)
-args.lim.flags |= BTRFS_QGROUP_LIMIT_RFER_CMPR |
- BTRFS_QGROUP_LIMIT_EXCL_CMPR;
-if (exclusive) {
-args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_EXCL;
-args.lim.max_exclusive = size;
-} else {
-args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_RFER;
-args.lim.max_referenced = size;
-}
+if (compressed)
+args.lim.flags |= BTRFS_QGROUP_LIMIT_RFER_CMPR |
+ BTRFS_QGROUP_LIMIT_EXCL_CMPR;
+if (exclusive) {
+args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_EXCL;
+args.lim.max_exclusive = size;
+} else {
+args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_RFER;
+args.lim.max_referenced = size;
}
if (argc - optind == 2) {
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs: fill -last_trans for delayed inode in btrfs_fill_inode.
We need to fill inode when we found a node for it in delayed_nodes_tree. But we did not fill the -last_trans currently, it will cause the test of xfstest/generic/311 fail. Scenario of the 311 is shown as below: Problem: (1). test_fd = open(fname, O_RDWR|O_DIRECT) (2). pwrite(test_fd, buf, 4096, 0) (3). close(test_fd) (4). drop_all_caches() echo 3 /proc/sys/vm/drop_caches (5). test_fd = open(fname, O_RDWR|O_DIRECT) (6). fsync(test_fd); we did not get the correct log entry for the file Reason: When we re-open this file in (5), we would find a node in delayed_nodes_tree and fill the inode we are lookup with the information. But the -last_trans is not filled, then the fsync() will check the -last_trans and found it's 0 then say this inode is already in our tree which is commited, not recording the extents for it. Fix: This patch fill the -last_trans properly and set the runtime_flags if needed in this situation. Then we can get the log entries we expected after (6) and generic/311 passed. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/delayed-inode.c | 2 ++ fs/btrfs/inode.c | 21 - 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/delayed-inode.c b/fs/btrfs/delayed-inode.c index 82f0c7c..9e8b435 100644 --- a/fs/btrfs/delayed-inode.c +++ b/fs/btrfs/delayed-inode.c @@ -1801,6 +1801,8 @@ int btrfs_fill_inode(struct inode *inode, u32 *rdev) set_nlink(inode, btrfs_stack_inode_nlink(inode_item)); inode_set_bytes(inode, btrfs_stack_inode_nbytes(inode_item)); BTRFS_I(inode)-generation = btrfs_stack_inode_generation(inode_item); +BTRFS_I(inode)-last_trans = btrfs_stack_inode_transid(inode_item); + inode-i_version = btrfs_stack_inode_sequence(inode_item); inode-i_rdev = 0; *rdev = btrfs_stack_inode_rdev(inode_item); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index d2e732d..b132936 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -3628,25 +3628,28 @@ static void btrfs_read_locked_inode(struct inode *inode) BTRFS_I(inode)-generation = btrfs_inode_generation(leaf, inode_item); BTRFS_I(inode)-last_trans = btrfs_inode_transid(leaf, inode_item); + inode-i_version = btrfs_inode_sequence(leaf, inode_item); + inode-i_generation = BTRFS_I(inode)-generation; + inode-i_rdev = 0; + rdev = btrfs_inode_rdev(leaf, inode_item); + + BTRFS_I(inode)-index_cnt = (u64)-1; + BTRFS_I(inode)-flags = btrfs_inode_flags(leaf, inode_item); + +cache_index: /* * If we were modified in the current generation and evicted from memory * and then re-read we need to do a full sync since we don't have any * idea about which extents were modified before we were evicted from * cache. +* +* This is required for both inode re-read from disk and delayed inode +* in delayed_nodes_tree. */ if (BTRFS_I(inode)-last_trans == root-fs_info-generation) set_bit(BTRFS_INODE_NEEDS_FULL_SYNC, BTRFS_I(inode)-runtime_flags); - inode-i_version = btrfs_inode_sequence(leaf, inode_item); - inode-i_generation = BTRFS_I(inode)-generation; - inode-i_rdev = 0; - rdev = btrfs_inode_rdev(leaf, inode_item); - - BTRFS_I(inode)-index_cnt = (u64)-1; - BTRFS_I(inode)-flags = btrfs_inode_flags(leaf, inode_item); - -cache_index: path-slots[0]++; if (inode-i_nlink != 1 || path-slots[0] = btrfs_header_nritems(leaf)) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/4] Btrfs-progs: qgroup: show specified quota data.
This patch allow user to choose the information type
in command of qgroup show.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 24 +++-
qgroup.c | 44 +++-
qgroup.h | 7 ++-
3 files changed, 64 insertions(+), 11 deletions(-)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 1c42fc8..86b6d2d 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -204,6 +204,21 @@ static int cmd_qgroup_destroy(int argc, char **argv)
return ret;
}
+static int parse_show_type(u8 *qgroup_type, char *arg)
+{
+ if (strcmp(arg, data) == 0)
+ *qgroup_type = BTRFS_QGROUP_SHOW_TYPE_DATA;
+ else if (strcmp(arg, metadata) == 0)
+ *qgroup_type = BTRFS_QGROUP_SHOW_TYPE_METADATA;
+ else if (strcmp(arg, mixed) == 0) {
+ *qgroup_type = BTRFS_QGROUP_SHOW_TYPE_MIXED;
+ } else {
+ fprintf(stderr, ERROR: Invalid qgroup type arguments given\n);
+ return -EINVAL;
+ }
+ return 0;
+}
+
static const char * const cmd_qgroup_show_usage[] = {
btrfs qgroup show -pcreFf
[--sort=qgroupid,rfer,excl,max_rfer,max_excl] path,
@@ -238,6 +253,7 @@ static int cmd_qgroup_show(int argc, char **argv)
int ret = 0;
int fd;
int e;
+ u8 type = BTRFS_QGROUP_SHOW_TYPE_MIXED;
DIR *dirstream = NULL;
u64 qgroupid;
int filter_flag = 0;
@@ -253,6 +269,7 @@ static int cmd_qgroup_show(int argc, char **argv)
int c;
int option_index = 0;
static const struct option long_options[] = {
+ {type, required_argument, NULL, 't'},
{sort, 1, NULL, 'S'},
{raw, no_argument, NULL, GETOPT_VAL_RAW},
{kbytes, no_argument, NULL, GETOPT_VAL_KBYTES},
@@ -299,6 +316,11 @@ static int cmd_qgroup_show(int argc, char **argv)
if (ret)
usage(cmd_qgroup_show_usage);
break;
+ case 't':
+ ret = parse_show_type(type, optarg);
+ if (ret)
+ usage(cmd_qgroup_show_usage);
+ break;
case GETOPT_VAL_RAW:
unit_mode = UNITS_RAW;
break;
@@ -350,7 +372,7 @@ static int cmd_qgroup_show(int argc, char **argv)
BTRFS_QGROUP_FILTER_PARENT,
qgroupid);
}
- ret = btrfs_show_qgroups(fd, filter_set, comparer_set);
+ ret = btrfs_show_qgroups(fd, filter_set, comparer_set, type);
e = errno;
close_file_or_dir(fd, dirstream);
if (ret 0)
diff --git a/qgroup.c b/qgroup.c
index 91e25ea..0e32284 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -226,14 +226,36 @@ static void print_qgroup_column_add_blank(enum
btrfs_qgroup_column_enum column,
}
static void print_qgroup_column(struct btrfs_qgroup *qgroup,
- enum btrfs_qgroup_column_enum column)
+ enum btrfs_qgroup_column_enum column,
+ u8 type)
{
BUG_ON(column = BTRFS_QGROUP_ALL || column 0);
int len;
int unit_mode = btrfs_qgroup_columns[column].unit_mode;
int max_len = btrfs_qgroup_columns[column].max_len;
- struct btrfs_qgroup_info *info = qgroup-data_info;
- struct btrfs_qgroup_limits *limits = qgroup-mixed_limits;
+ struct btrfs_qgroup_limits *limits = NULL;
+ struct btrfs_qgroup_info *info = NULL;
+ int need_free = 0;
+
+
+ if (type == BTRFS_QGROUP_SHOW_TYPE_DATA) {
+ info = qgroup-data_info;
+ limits = qgroup-data_limits;
+ } else if (type == BTRFS_QGROUP_SHOW_TYPE_METADATA) {
+ info = qgroup-metadata_info;
+ limits = qgroup-metadata_limits;
+ } else {
+ /*
+* For mixed type show.
+*/
+ info = malloc(sizeof(*info));
+ info-rfer = qgroup-data_info.rfer +
+ qgroup-metadata_info.rfer;
+ info-excl = qgroup-data_info.excl +
+ qgroup-metadata_info.excl;
+ limits = qgroup-mixed_limits;
+ need_free = 1;
+ }
switch (column) {
@@ -265,16 +287,19 @@ static void print_qgroup_column(struct btrfs_qgroup
*qgroup,
default:
break;
}
+
+ if (need_free)
+ free(info);
}
-static void print_single_qgroup_table(struct btrfs_qgroup *qgroup)
+static void print_single_qgroup_table(struct btrfs_qgroup *qgroup, u8 type)
{
int i;
for (i = 0; i BTRFS_QGROUP_ALL; i++) {
if (!btrfs_qgroup_columns[i].need_print
[PATCH 3/7] Btrfs: qgroup: record and account ref for qgroup in different type.
This patch make the qgroup account the information of data or metadata
by different infos.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c| 48 +
fs/btrfs/qgroup.c | 239 ++
fs/btrfs/qgroup.h | 25 -
fs/btrfs/tests/qgroup-tests.c | 15 ++-
fs/btrfs/transaction.c| 11 +-
5 files changed, 217 insertions(+), 121 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 3f49316..cb27da6 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -83,7 +83,7 @@ static int __btrfs_free_extent(struct btrfs_trans_handle
*trans,
u64 root_objectid, u64 owner_objectid,
u64 owner_offset, int refs_to_drop,
struct btrfs_delayed_extent_op *extra_op,
- int no_quota);
+ int no_quota, unsigned int quota_type);
static void __run_delayed_extent_op(struct btrfs_delayed_extent_op *extent_op,
struct extent_buffer *leaf,
struct btrfs_extent_item *ei);
@@ -1971,7 +1971,8 @@ static int __btrfs_inc_extent_ref(struct
btrfs_trans_handle *trans,
u64 parent, u64 root_objectid,
u64 owner, u64 offset, int refs_to_add,
int no_quota,
- struct btrfs_delayed_extent_op *extent_op)
+ struct btrfs_delayed_extent_op *extent_op,
+ unsigned int quota_type)
{
struct btrfs_fs_info *fs_info = root-fs_info;
struct btrfs_path *path;
@@ -2013,7 +2014,8 @@ static int __btrfs_inc_extent_ref(struct
btrfs_trans_handle *trans,
btrfs_release_path(path);
ret = btrfs_qgroup_record_ref(trans, fs_info, root_objectid,
- bytenr, num_bytes, type, 0);
+ bytenr, num_bytes, type,
+ quota_type, 0);
goto out;
}
@@ -2037,7 +2039,8 @@ static int __btrfs_inc_extent_ref(struct
btrfs_trans_handle *trans,
if (!no_quota) {
ret = btrfs_qgroup_record_ref(trans, fs_info, root_objectid,
- bytenr, num_bytes, type, 0);
+ bytenr, num_bytes, type,
+ quota_type, 0);
if (ret)
goto out;
}
@@ -2091,13 +2094,15 @@ static int run_delayed_data_ref(struct
btrfs_trans_handle *trans,
node-num_bytes, parent,
ref_root, ref-objectid,
ref-offset, node-ref_mod,
-node-no_quota, extent_op);
+node-no_quota, extent_op,
+BTRFS_QGROUP_REF_TYPE_DATA);
} else if (node-action == BTRFS_DROP_DELAYED_REF) {
ret = __btrfs_free_extent(trans, root, node-bytenr,
node-num_bytes, parent,
ref_root, ref-objectid,
ref-offset, node-ref_mod,
- extent_op, node-no_quota);
+ extent_op, node-no_quota,
+ BTRFS_QGROUP_REF_TYPE_DATA);
} else {
BUG();
}
@@ -2258,12 +2263,12 @@ static int run_delayed_tree_ref(struct
btrfs_trans_handle *trans,
ret = __btrfs_inc_extent_ref(trans, root, node-bytenr,
node-num_bytes, parent, ref_root,
ref-level, 0, 1, node-no_quota,
-extent_op);
+extent_op,
BTRFS_QGROUP_REF_TYPE_METADATA);
} else if (node-action == BTRFS_DROP_DELAYED_REF) {
ret = __btrfs_free_extent(trans, root, node-bytenr,
node-num_bytes, parent, ref_root,
ref-level, 0, 1, extent_op,
- node-no_quota);
+ node-no_quota,
BTRFS_QGROUP_REF_TYPE_METADATA);
} else {
BUG();
}
@@ -3694,7 +3699,7 @@ commit_trans:
data_sinfo-flags, bytes, 1);
return -ENOSPC;
}
- ret = btrfs_qgroup_reserve(root, write_bytes
[PATCH 5/7] Btrfs: qgroup: update quota numbers in btrfs_qgroup_inherit.
Original, the all quota numbers are stored in data_info, but
now, we store them separately in data_info and metadata_info.
Then, when we create a snapshot and update the quota number,
we need to update both of the data_info and metadata_info.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 57 ++-
1 file changed, 40 insertions(+), 17 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 5df8527..9363fd0 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2525,34 +2525,57 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
srcgroup = find_qgroup_rb(fs_info, srcid);
if (!srcgroup)
goto unlock;
- /*
-* FIXME: use the data_info to store all information currently.
-* will seperate the information into data and metadata later.
-**/
dstinfo = dstgroup-data_info;
srcinfo = srcgroup-data_info;
- /*
-* We call inherit after we clone the root in order to make sure
-* our counts don't go crazy, so at this point the only
-* difference between the two roots should be the root node.
-*/
- dstinfo-rfer = srcinfo-rfer;
- dstinfo-rfer_cmpr = srcinfo-rfer_cmpr;
- dstinfo-excl = level_size;
- dstinfo-excl_cmpr = level_size;
- srcinfo-excl = level_size;
- srcinfo-excl_cmpr = level_size;
+ if (!btrfs_fs_incompat(fs_info, QGROUP_TYPE)) {
+ /*
+* We call inherit after we clone the root in order to
make sure
+* our counts don't go crazy, so at this point the only
+* difference between the two roots should be the root
node.
+*/
+ dstinfo-rfer = srcinfo-rfer;
+ dstinfo-rfer_cmpr = srcinfo-rfer_cmpr;
+ dstinfo-excl = level_size;
+ dstinfo-excl_cmpr = level_size;
+ srcinfo-excl = level_size;
+ srcinfo-excl_cmpr = level_size;
+ } else {
+ dstinfo-rfer = srcinfo-rfer;
+ dstinfo-rfer_cmpr = srcinfo-rfer_cmpr;
+ /*
+* add the metadata for dstqgroup.
+*/
+ dstinfo = dstgroup-metadata_info;
+ dstinfo-rfer = level_size;
+ dstinfo-rfer_cmpr = level_size;
+ dstinfo-excl = level_size;
+ dstinfo-excl_cmpr = level_size;
+ srcinfo-excl = 0;
+ srcinfo-excl_cmpr = 0;
+ }
- dstlimits = dstgroup-mixed_limits;
- srclimits = srcgroup-mixed_limits;
+ if (!btrfs_fs_incompat(fs_info, QGROUP_TYPE)) {
+ dstlimits = dstgroup-mixed_limits;
+ srclimits = srcgroup-mixed_limits;
+ } else {
+ dstlimits = dstgroup-data_limits;
+ srclimits = srcgroup-data_limits;
+ }
/* inherit the limit info */
+again:
dstlimits-lim_flags = srclimits-lim_flags;
dstlimits-max_rfer = srclimits-max_rfer;
dstlimits-max_excl = srclimits-max_excl;
dstlimits-rsv_rfer = srclimits-rsv_rfer;
dstlimits-rsv_excl = srclimits-rsv_excl;
+ if (dstlimits != dstgroup-mixed_limits) {
+ dstlimits++;
+ srclimits++;
+ goto again;
+ }
+
qgroup_dirty(fs_info, dstgroup);
qgroup_dirty(fs_info, srcgroup);
}
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/7] Btrfs: qgroup: split information and limits in qgroup to other structures.
Restruct the btrfs_qgroup structure:
btrfs_qgroup: |-|
- |rfer |
|data_info |---btrfs_qgroup_info|rfer_cmpr|
|metadata_info | |excl |
| | |excl_cmpr|
|data_limits| ---
|metadata_limits|
|mixed_limits |---btrfs_qgroup_limits|-
|-- | |lim_flags |
|max_rfer |
|max_excl |
|rsv_rfer |
|rsv_excl |
Then we can account data and metadata separately. And we can limit them
separately. mixed_limits is provided to limit the sum of data and metadata.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 308 +-
1 file changed, 214 insertions(+), 94 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 28b0aa5..dd99908 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -46,33 +46,44 @@
* - check all ioctl parameters
*/
-/*
- * one struct for each qgroup, organized in fs_info-qgroup_tree.
- */
-struct btrfs_qgroup {
- u64 qgroupid;
-
- /*
-* state
-*/
+struct btrfs_qgroup_info {
u64 rfer; /* referenced */
u64 rfer_cmpr; /* referenced compressed */
u64 excl; /* exclusive */
u64 excl_cmpr; /* exclusive compressed */
/*
-* limits
+* reservation tracking
*/
+ u64 reserved;
+};
+
+struct btrfs_qgroup_limits {
u64 lim_flags; /* which limits are set */
u64 max_rfer;
u64 max_excl;
u64 rsv_rfer;
u64 rsv_excl;
+};
+
+/*
+ * one struct for each qgroup, organized in fs_info-qgroup_tree.
+ */
+struct btrfs_qgroup {
+ u64 qgroupid;
/*
-* reservation tracking
+* infos
*/
- u64 reserved;
+ struct btrfs_qgroup_infodata_info;
+ struct btrfs_qgroup_infometadata_info;
+
+ /*
+* limits
+*/
+ struct btrfs_qgroup_limits data_limits;
+ struct btrfs_qgroup_limits metadata_limits;
+ struct btrfs_qgroup_limits mixed_limits;
/*
* lists
@@ -251,11 +262,13 @@ int btrfs_verify_qgroup_counts(struct btrfs_fs_info
*fs_info, u64 qgroupid,
u64 rfer, u64 excl)
{
struct btrfs_qgroup *qgroup;
+ struct btrfs_qgroup_info *info;
qgroup = find_qgroup_rb(fs_info, qgroupid);
if (!qgroup)
return -EINVAL;
- if (qgroup-rfer != rfer || qgroup-excl != excl)
+ info = qgroup-data_info;
+ if (info-rfer != rfer || info-excl != excl)
return -EINVAL;
return 0;
}
@@ -357,26 +370,30 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info
*fs_info)
switch (found_key.type) {
case BTRFS_QGROUP_INFO_KEY: {
struct btrfs_qgroup_info_item *ptr;
+ struct btrfs_qgroup_info *info;
+ info = qgroup-data_info;
ptr = btrfs_item_ptr(l, slot,
struct btrfs_qgroup_info_item);
- qgroup-rfer = btrfs_qgroup_info_rfer(l, ptr);
- qgroup-rfer_cmpr = btrfs_qgroup_info_rfer_cmpr(l, ptr);
- qgroup-excl = btrfs_qgroup_info_excl(l, ptr);
- qgroup-excl_cmpr = btrfs_qgroup_info_excl_cmpr(l, ptr);
+ info-rfer = btrfs_qgroup_info_rfer(l, ptr);
+ info-rfer_cmpr = btrfs_qgroup_info_rfer_cmpr(l, ptr);
+ info-excl = btrfs_qgroup_info_excl(l, ptr);
+ info-excl_cmpr = btrfs_qgroup_info_excl_cmpr(l, ptr);
/* generation currently unused */
break;
}
case BTRFS_QGROUP_LIMIT_KEY: {
struct btrfs_qgroup_limit_item *ptr;
+ struct btrfs_qgroup_limits *limits;
+ limits = qgroup-mixed_limits;
ptr = btrfs_item_ptr(l, slot,
struct btrfs_qgroup_limit_item);
- qgroup-lim_flags = btrfs_qgroup_limit_flags(l, ptr);
- qgroup-max_rfer = btrfs_qgroup_limit_max_rfer(l, ptr);
- qgroup-max_excl = btrfs_qgroup_limit_max_excl(l, ptr);
- qgroup-rsv_rfer
[PATCH 4/7] Btrfs: qgroup: update all infos and limits to disk.
There are not only one info for a qgroup, then we need
to store the all infos and limits to disk.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 52
1 file changed, 40 insertions(+), 12 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 9dd2627..5df8527 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -624,6 +624,8 @@ info_again:
if (btrfs_fs_incompat(quota_root-fs_info, QGROUP_TYPE))
key.objectid = BTRFS_QGROUP_DATA_LIMIT_OBJECTID;
+ else
+ key.objectid = 0;
key.type = BTRFS_QGROUP_LIMIT_KEY;
@@ -700,6 +702,8 @@ info_again:
if (btrfs_fs_incompat(quota_root-fs_info, QGROUP_TYPE))
key.objectid = BTRFS_QGROUP_DATA_LIMIT_OBJECTID;
+ else
+ key.objectid = 0;
key.type = BTRFS_QGROUP_LIMIT_KEY;
@@ -739,14 +743,21 @@ static int update_qgroup_limit_item(struct
btrfs_trans_handle *trans,
int ret;
int slot;
- key.objectid = 0;
+ if (!btrfs_fs_incompat(root-fs_info, QGROUP_TYPE)) {
+ key.objectid = 0;
+ limits = qgroup-mixed_limits;
+ } else {
+ key.objectid = BTRFS_QGROUP_DATA_LIMIT_OBJECTID;
+ limits = qgroup-data_limits;
+ }
+
key.type = BTRFS_QGROUP_LIMIT_KEY;
key.offset = qgroup-qgroupid;
path = btrfs_alloc_path();
if (!path)
return -ENOMEM;
-
+again:
ret = btrfs_search_slot(trans, root, key, path, 0, 1);
if (ret 0)
ret = -ENOENT;
@@ -754,8 +765,6 @@ static int update_qgroup_limit_item(struct
btrfs_trans_handle *trans,
if (ret)
goto out;
- limits = qgroup-mixed_limits;
-
l = path-nodes[0];
slot = path-slots[0];
qgroup_limit = btrfs_item_ptr(l, slot, struct btrfs_qgroup_limit_item);
@@ -766,6 +775,14 @@ static int update_qgroup_limit_item(struct
btrfs_trans_handle *trans,
btrfs_set_qgroup_limit_rsv_excl(l, qgroup_limit, limits-rsv_excl);
btrfs_mark_buffer_dirty(l);
+ btrfs_release_path(path);
+
+ if (btrfs_fs_incompat(root-fs_info, QGROUP_TYPE)
+ key.objectid != BTRFS_QGROUP_MIXED_LIMIT_OBJECTID) {
+ key.objectid++;
+ limits++;
+ goto again;
+ }
out:
btrfs_free_path(path);
@@ -780,14 +797,18 @@ static int update_qgroup_info_item(struct
btrfs_trans_handle *trans,
struct btrfs_key key;
struct extent_buffer *l;
struct btrfs_qgroup_info_item *qgroup_info;
- struct btrfs_qgroup_info *data_info;
+ struct btrfs_qgroup_info *info;
int ret;
int slot;
if (btrfs_test_is_dummy_root(root))
return 0;
- key.objectid = 0;
+ if (!btrfs_fs_incompat(root-fs_info, QGROUP_TYPE))
+ key.objectid = 0;
+ else
+ key.objectid = BTRFS_QGROUP_DATA_INFO_OBJECTID;
+
key.type = BTRFS_QGROUP_INFO_KEY;
key.offset = qgroup-qgroupid;
@@ -795,6 +816,8 @@ static int update_qgroup_info_item(struct
btrfs_trans_handle *trans,
if (!path)
return -ENOMEM;
+ info = qgroup-data_info;
+again:
ret = btrfs_search_slot(trans, root, key, path, 0, 1);
if (ret 0)
ret = -ENOENT;
@@ -802,19 +825,24 @@ static int update_qgroup_info_item(struct
btrfs_trans_handle *trans,
if (ret)
goto out;
- data_info = qgroup-data_info;
-
l = path-nodes[0];
slot = path-slots[0];
qgroup_info = btrfs_item_ptr(l, slot, struct btrfs_qgroup_info_item);
btrfs_set_qgroup_info_generation(l, qgroup_info, trans-transid);
- btrfs_set_qgroup_info_rfer(l, qgroup_info, data_info-rfer);
- btrfs_set_qgroup_info_rfer_cmpr(l, qgroup_info, data_info-rfer_cmpr);
- btrfs_set_qgroup_info_excl(l, qgroup_info, data_info-excl);
- btrfs_set_qgroup_info_excl_cmpr(l, qgroup_info, data_info-excl_cmpr);
+ btrfs_set_qgroup_info_rfer(l, qgroup_info, info-rfer);
+ btrfs_set_qgroup_info_rfer_cmpr(l, qgroup_info, info-rfer_cmpr);
+ btrfs_set_qgroup_info_excl(l, qgroup_info, info-excl);
+ btrfs_set_qgroup_info_excl_cmpr(l, qgroup_info, info-excl_cmpr);
btrfs_mark_buffer_dirty(l);
+ btrfs_release_path(path);
+ if (btrfs_fs_incompat(root-fs_info, QGROUP_TYPE)
+ key.objectid != BTRFS_QGROUP_METADATA_INFO_OBJECTID) {
+ key.objectid++;
+ info++;
+ goto again;
+ }
out:
btrfs_free_path(path);
return ret;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 7/7] Btrfs: qgroup: allow user to limit qgroup in different type.
until now, we can account the quota number for data and metadata
separately, but we can not limit only one of them. This patch
add the support to only limit one of the data or metadata.
Also you can limit them at the same time.
Even you can limit data quota, metadata quota and mixed quota
at the same time.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 10 +++---
fs/btrfs/qgroup.h | 26 ++
2 files changed, 33 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index bb98bdd..793c7bb 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1322,6 +1322,7 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
struct btrfs_root *quota_root;
struct btrfs_qgroup *qgroup;
struct btrfs_qgroup_limits *limits;
+ u8 limits_type = get_limit_type(limit-flags);
int ret = 0;
mutex_lock(fs_info-qgroup_ioctl_lock);
@@ -1337,9 +1338,12 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
goto out;
}
- /* To the compatibility, treat the mixed limits as the
-* default limits now. will change it later. */
- limits = qgroup-mixed_limits;
+ if (limits_type == BTRFS_QGROUP_LIMIT_DATA)
+ limits = qgroup-data_limits;
+ else if (limits_type == BTRFS_QGROUP_LIMIT_METADATA)
+ limits = qgroup-metadata_limits;
+ else
+ limits = qgroup-mixed_limits;
spin_lock(fs_info-qgroup_lock);
if (limit-flags BTRFS_QGROUP_LIMIT_MAX_RFER)
diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h
index 11b73f3..6e4565f 100644
--- a/fs/btrfs/qgroup.h
+++ b/fs/btrfs/qgroup.h
@@ -36,6 +36,32 @@
#define BTRFS_QGROUP_REF_TYPE_DEFAULT BTRFS_QGROUP_REF_TYPE_DATA
/*
+ * As we can account and limit data and metadata in
+ * a qgroup separately. We use the first 8 bits in
+ * btrfs_qgroup_limit-flags to specify the type we
+ * want to limit. Currently, there are three choice:
+ * DATA, METADATA, MIXED.
+ */
+#define BTRFS_QGROUP_LIMIT_TYPE_SHIFT 56
+
+#define BTRFS_QGROUP_LIMIT_DATA 1
+#define BTRFS_QGROUP_LIMIT_METADATA 2
+#define BTRFS_QGROUP_LIMIT_MIXED4
+
+static inline u8 get_limit_type(u64 flag)
+{
+u8 type = (flag BTRFS_QGROUP_LIMIT_TYPE_SHIFT);
+ /*
+* If user did not set the type for limit, default
+* for mixed_limits;
+*/
+ if (!type)
+ type = BTRFS_QGROUP_LIMIT_MIXED;
+ return type;
+}
+
+
+/*
* A description of the operations, all of these operations only happen when we
* are adding the 1st reference for that subvolume in the case of adding space
* or on the last reference delete in the case of subtraction. The only
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/7 V2] Btrfs: qgroup: part-4: Add type to btrfs qgroup.
0 10485760 ---
--- --DATA info and limits
qgroupid rfer excl max_rfer max_excl parent
child
--
-
0/5 163841638400 ---
---
0/257 163841638400 ---
--- --METADATA info and limits
qgroupid rfer excl max_rfer max_excl parent
child
--
-
0/5 163841638400 ---
---
0/25710502144 1050214400 ---
--- --MIXED info and limits
After all, you can use btrfs-debug-tree tool to show the details for each items.
Any comment or test are welcome !!!
Thanx
Yang
Dongsheng Yang (7):
Btrfs: qgroup: split information and limits in qgroup to other
structures.
Btrfs: qgroup: add incompatability feature for QGROUP_TYPE.
Btrfs: qgroup: record and account ref for qgroup in different type.
Btrfs: qgroup: update all infos and limits to disk.
Btrfs: qgroup: update quota numbers in btrfs_qgroup_inherit.
Btrfs: qgroup: account data and metadata separately in rescan.
Btrfs: qgroup: allow user to limit qgroup in different type.
fs/btrfs/ctree.h | 20 +-
fs/btrfs/extent-tree.c| 48 ++--
fs/btrfs/qgroup.c | 536 +-
fs/btrfs/qgroup.h | 52 +++-
fs/btrfs/tests/qgroup-tests.c | 15 +-
fs/btrfs/transaction.c| 11 +-
6 files changed, 534 insertions(+), 148 deletions(-)
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 6/7] Btrfs: qgroup: account data and metadata separately in rescan.
So far, we can account quota number for data and metadata separately,
but in quota resan, we have not implemented this feature. This patch
make the rescan smart to distingush data and metadata and account them
separately.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 21 ++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 9363fd0..bb98bdd 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2872,6 +2872,8 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct
btrfs_path *path,
int new_roots;
int slot;
int ret;
+ bool skinny_metadata = btrfs_fs_incompat(fs_info,
+SKINNY_METADATA);
path-leave_spinning = 1;
mutex_lock(fs_info-qgroup_rescan_lock);
@@ -2910,14 +2912,27 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info,
struct btrfs_path *path,
mutex_unlock(fs_info-qgroup_rescan_lock);
for (; slot btrfs_header_nritems(scratch_leaf); ++slot) {
+ u8 quota_type = BTRFS_QGROUP_REF_TYPE_DATA;
+ struct btrfs_extent_item *ei;
+ u64 flags;
+
+ ei = btrfs_item_ptr(scratch_leaf, slot, struct
btrfs_extent_item);
+ flags = btrfs_extent_flags(scratch_leaf, ei);
+
btrfs_item_key_to_cpu(scratch_leaf, found, slot);
if (found.type != BTRFS_EXTENT_ITEM_KEY
found.type != BTRFS_METADATA_ITEM_KEY)
continue;
- if (found.type == BTRFS_METADATA_ITEM_KEY)
+ if (found.type == BTRFS_METADATA_ITEM_KEY) {
num_bytes = fs_info-extent_root-nodesize;
- else
+ if (btrfs_fs_incompat(fs_info, QGROUP_TYPE))
+ quota_type = BTRFS_QGROUP_REF_TYPE_METADATA;
+ } else {
num_bytes = found.offset;
+ if (!skinny_metadata (flags
BTRFS_EXTENT_FLAG_TREE_BLOCK)
+ btrfs_fs_incompat(fs_info, QGROUP_TYPE))
+ quota_type = BTRFS_QGROUP_REF_TYPE_METADATA;
+ }
ulist_reinit(qgroups);
ret = btrfs_find_all_roots(NULL, fs_info, found.objectid, 0,
@@ -2937,7 +2952,7 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct
btrfs_path *path,
goto out;
}
- ret = qgroup_adjust_counters(fs_info, 0, num_bytes, 0, qgroups,
+ ret = qgroup_adjust_counters(fs_info, 0, num_bytes, quota_type,
qgroups,
seq, 0, new_roots, 1);
if (ret 0) {
spin_unlock(fs_info-qgroup_lock);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/7] Btrfs: qgroup: add incompatability feature for QGROUP_TYPE.
As we need to change the structure in disk for qgroup, we have
to introduce a incompatability feature for it.
|0 |BTRFS_QGROUP_INFO_KEY |qgroupid| |TYPE_OBJECTID
|BTRFS_QGROUP_INFO_KEY |qgroupid|
similar for the limits.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ctree.h | 20 ++-
fs/btrfs/qgroup.c | 77 +++
2 files changed, 91 insertions(+), 6 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index f00eacd..d029119 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -150,6 +150,22 @@ struct btrfs_ordered_sum;
#define BTRFS_DEV_REPLACE_DEVID 0ULL
/*
+ * the items for qgroup info.
+ */
+#define BTRFS_QGROUP_DATA_INFO_OBJECTID 1ULL
+
+#define BTRFS_QGROUP_METADATA_INFO_OBJECTID 2ULL
+
+/*
+ * the items for qgroup limits.
+ */
+#define BTRFS_QGROUP_DATA_LIMIT_OBJECTID 1ULL
+
+#define BTRFS_QGROUP_METADATA_LIMIT_OBJECTID 2ULL
+
+#define BTRFS_QGROUP_MIXED_LIMIT_OBJECTID 3ULL
+
+/*
* the max metadata block size. This limit is somewhat artificial,
* but the memmove costs go through the roof for larger blocks.
*/
@@ -522,6 +538,7 @@ struct btrfs_super_block {
#define BTRFS_FEATURE_INCOMPAT_RAID56 (1ULL 7)
#define BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA (1ULL 8)
#define BTRFS_FEATURE_INCOMPAT_NO_HOLES(1ULL 9)
+#define BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE (1ULL 10)
#define BTRFS_FEATURE_COMPAT_SUPP 0ULL
#define BTRFS_FEATURE_COMPAT_SAFE_SET 0ULL
@@ -539,7 +556,8 @@ struct btrfs_super_block {
BTRFS_FEATURE_INCOMPAT_RAID56 |\
BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \
BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \
-BTRFS_FEATURE_INCOMPAT_NO_HOLES)
+BTRFS_FEATURE_INCOMPAT_NO_HOLES | \
+BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE)
#define BTRFS_FEATURE_INCOMPAT_SAFE_SET\
(BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index dd99908..34eb4f5 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -372,7 +372,19 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)
struct btrfs_qgroup_info_item *ptr;
struct btrfs_qgroup_info *info;
- info = qgroup-data_info;
+ /*
+* In newer qgroup, we store the quota data in
+* different info_items.
+*/
+ if (!btrfs_fs_incompat(fs_info, QGROUP_TYPE)) {
+ info = qgroup-data_info;
+ } else {
+ if (found_key.objectid ==
BTRFS_QGROUP_DATA_INFO_OBJECTID)
+ info = qgroup-data_info;
+ else
+ info = qgroup-metadata_info;
+ }
+
ptr = btrfs_item_ptr(l, slot,
struct btrfs_qgroup_info_item);
info-rfer = btrfs_qgroup_info_rfer(l, ptr);
@@ -386,7 +398,17 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)
struct btrfs_qgroup_limit_item *ptr;
struct btrfs_qgroup_limits *limits;
- limits = qgroup-mixed_limits;
+ if (!btrfs_fs_incompat(fs_info, QGROUP_TYPE)) {
+ limits = qgroup-mixed_limits;
+ } else {
+ if (found_key.objectid ==
BTRFS_QGROUP_DATA_LIMIT_OBJECTID)
+ limits = qgroup-data_limits;
+ else if (found_key.objectid ==
BTRFS_QGROUP_METADATA_LIMIT_OBJECTID)
+ limits = qgroup-metadata_limits;
+ else
+ limits = qgroup-mixed_limits;
+ }
+
ptr = btrfs_item_ptr(l, slot,
struct btrfs_qgroup_limit_item);
limits-lim_flags = btrfs_qgroup_limit_flags(l, ptr);
@@ -563,7 +585,10 @@ static int add_qgroup_item(struct btrfs_trans_handle
*trans,
if (!path)
return -ENOMEM;
- key.objectid = 0;
+ if (!btrfs_fs_incompat(quota_root-fs_info, QGROUP_TYPE))
+ key.objectid = 0;
+ else
+ key.objectid = BTRFS_QGROUP_DATA_INFO_OBJECTID;
key.type = BTRFS_QGROUP_INFO_KEY;
key.offset = qgroupid;
@@ -573,6 +598,7 @@ static int
[PATCH 2/4] Btrfs-progs: qgroup: print info and limits type in btrfs-debug-tree.
We have restructured the btrfs_qgroup, and we are using objectid for btrfs_qgroup_info_item now. Then we need to print the objectid in btrfs-debug-tree. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- print-tree.c | 16 1 file changed, 16 insertions(+) diff --git a/print-tree.c b/print-tree.c index 3a7c13c..7cb9563 100644 --- a/print-tree.c +++ b/print-tree.c @@ -644,6 +644,22 @@ static void print_objectid(u64 objectid, u8 type) printf(%llu/%llu, objectid 48, objectid ((1ll 48) - 1)); return; + case BTRFS_QGROUP_INFO_KEY: + if (objectid == 0) + printf(MIXED); + else if (objectid == BTRFS_QGROUP_DATA_INFO_OBJECTID) + printf(DATA); + else + printf(METADATA); + return; + case BTRFS_QGROUP_LIMIT_KEY: + if (objectid == BTRFS_QGROUP_DATA_LIMIT_OBJECTID) + printf(DATA); + else if (objectid == BTRFS_QGROUP_METADATA_LIMIT_OBJECTID) + printf(METADATA); + else + printf(MIXED); + return; case BTRFS_UUID_KEY_SUBVOL: case BTRFS_UUID_KEY_RECEIVED_SUBVOL: printf(0x%016llx, (unsigned long long)objectid); -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/4] Btrfs-progs: qgroup: add a opt for type of qgroup limit.
This patch add a support for user to limit the data, metadata,
or mixed of a qgroup. Even you can set these three limits at
one time for a qgroup.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
cmds-qgroup.c | 48 +++-
ioctl.h | 13 +
2 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index c5082f7..1c42fc8 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -367,14 +367,46 @@ static const char * const cmd_qgroup_limit_usage[] = {
-c limit amount of data after compression. This is the default,,
it is currently not possible to turn off this option.,
-e limit space exclusively assigned to this qgroup,
+ -t limit space for data, metadata or mixed,
NULL
};
+inline u8 get_limit_type(u64 flag)
+{
+ return (flag BTRFS_QGROUP_LIMIT_TYPE_SHIFT);
+}
+
+inline u64 set_limit_type(u64 flag, u8 type)
+{
+ u64 temp = (flag BTRFS_QGROUP_LIMIT_TYPE_SHIFT);
+
+ temp |= type;
+ flag |= (temp BTRFS_QGROUP_LIMIT_TYPE_SHIFT);
+
+ return flag;
+}
+
+static int parse_limit_type(u8 *qgroup_type, char *arg)
+{
+ if (strcmp(arg, data) == 0)
+ *qgroup_type = BTRFS_QGROUP_LIMIT_DATA;
+ else if (strcmp(arg, metadata) == 0)
+ *qgroup_type = BTRFS_QGROUP_LIMIT_METADATA;
+ else if (strcmp(arg, mixed) == 0) {
+ *qgroup_type = BTRFS_QGROUP_LIMIT_MIXED;
+ } else {
+ fprintf(stderr, ERROR: Invalid qgroup type arguments given\n);
+ return -EINVAL;
+ }
+ return 0;
+}
+
static int cmd_qgroup_limit(int argc, char **argv)
{
int ret = 0;
int fd;
int e;
+ u8 type = BTRFS_QGROUP_LIMIT_MIXED;
char *path = NULL;
struct btrfs_ioctl_qgroup_limit_args args;
unsigned long long size;
@@ -384,7 +416,14 @@ static int cmd_qgroup_limit(int argc, char **argv)
optind = 1;
while (1) {
- int c = getopt(argc, argv, ce);
+ int c;
+ static const struct option long_options[] = {
+ {type, required_argument, NULL, 't'},
+ {NULL, 0, NULL, 0}
+ };
+
+ c = getopt_long(argc, argv, ce,
+ long_options, NULL);
if (c 0)
break;
switch (c) {
@@ -394,6 +433,11 @@ static int cmd_qgroup_limit(int argc, char **argv)
case 'e':
exclusive = 1;
break;
+ case 't':
+ ret = parse_limit_type(type, optarg);
+ if (ret)
+ return -EINVAL;
+ break;
default:
usage(cmd_qgroup_limit_usage);
}
@@ -419,6 +463,8 @@ static int cmd_qgroup_limit(int argc, char **argv)
args.lim.flags |= BTRFS_QGROUP_LIMIT_MAX_RFER;
args.lim.max_referenced = size;
}
+
+ args.lim.flags = set_limit_type(args.lim.flags, type);
}
if (argc - optind == 2) {
diff --git a/ioctl.h b/ioctl.h
index d550ca6..a1ebbb5 100644
--- a/ioctl.h
+++ b/ioctl.h
@@ -45,6 +45,19 @@ struct btrfs_ioctl_vol_args {
#define BTRFS_QGROUP_INHERIT_SET_LIMITS(1ULL 0)
+/*
+ * As we can account and limit data and metadata in
+ * a qgroup separately. We use the first 8 bits in
+ * btrfs_qgroup_limit-flags to specify the type we
+ * want to limit. Currently, there are three choice:
+ * DATA, METADATA, MIXED.
+ */
+#define BTRFS_QGROUP_LIMIT_TYPE_SHIFT 56
+
+#define BTRFS_QGROUP_LIMIT_DATA1
+#define BTRFS_QGROUP_LIMIT_METADATA2
+#define BTRFS_QGROUP_LIMIT_MIXED 4
+
struct btrfs_qgroup_limit {
__u64 flags;
__u64 max_referenced;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/4] Btrfs-progs: qgroup: add incompatability feature for QGROUP_TYPE.
This patch splits info and limits from qgroup.
btrfs_qgroup: |-|
- |rfer |
|data_info |---btrfs_qgroup_info|rfer_cmpr|
|metadata_info | |excl |
| | |excl_cmpr|
|data_limits| ---
|metadata_limits|
|mixed_limits |---btrfs_qgroup_limits|-
|-- | |lim_flags |
|max_rfer |
|max_excl |
|rsv_rfer |
|rsv_excl |
And the objectid for each items are changed:
|0 |BTRFS_QGROUP_INFO_KEY |qgroupid| |TYPE_OBJECTID
|BTRFS_QGROUP_INFO_KEY |qgroupid|
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
ctree.h | 20 ++-
mkfs.c | 2 +
qgroup.c | 184 +++
3 files changed, 147 insertions(+), 59 deletions(-)
diff --git a/ctree.h b/ctree.h
index 2d2988b..85d4992 100644
--- a/ctree.h
+++ b/ctree.h
@@ -125,6 +125,22 @@ struct btrfs_free_space_ctl;
#define BTRFS_DEV_ITEMS_OBJECTID 1ULL
/*
+ * * the items for qgroup info.
+ * */
+#define BTRFS_QGROUP_DATA_INFO_OBJECTID 1ULL
+
+#define BTRFS_QGROUP_METADATA_INFO_OBJECTID 2ULL
+
+/*
+ * * the items for qgroup limits.
+ * */
+#define BTRFS_QGROUP_DATA_LIMIT_OBJECTID 1ULL
+
+#define BTRFS_QGROUP_METADATA_LIMIT_OBJECTID 2ULL
+
+#define BTRFS_QGROUP_MIXED_LIMIT_OBJECTID 3ULL
+
+/*
* the max metadata block size. This limit is somewhat artificial,
* but the memmove costs go through the roof for larger blocks.
*/
@@ -473,6 +489,7 @@ struct btrfs_super_block {
#define BTRFS_FEATURE_INCOMPAT_RAID56 (1ULL 7)
#define BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA (1ULL 8)
#define BTRFS_FEATURE_INCOMPAT_NO_HOLES(1ULL 9)
+#define BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE (1ULL 10)
#define BTRFS_FEATURE_COMPAT_SUPP 0ULL
@@ -486,7 +503,8 @@ struct btrfs_super_block {
BTRFS_FEATURE_INCOMPAT_RAID56 |\
BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS | \
BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \
-BTRFS_FEATURE_INCOMPAT_NO_HOLES)
+BTRFS_FEATURE_INCOMPAT_NO_HOLES | \
+BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE)
/*
* A leaf is full of items. offset and size tell us where to find
diff --git a/mkfs.c b/mkfs.c
index f83554d..de7612b 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1143,6 +1143,8 @@ static const struct btrfs_fs_feature {
reduced-size metadata extent refs },
{ no-holes, BTRFS_FEATURE_INCOMPAT_NO_HOLES,
no explicit hole extents for files },
+ { qgroup-type, BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE,
+ account data and metadata separately },
/* Keep this one last */
{ list-all, BTRFS_FEATURE_LIST_ALL, NULL }
};
diff --git a/qgroup.c b/qgroup.c
index 5a4e393..91e25ea 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -30,6 +30,26 @@ struct qgroup_lookup {
struct rb_root root;
};
+struct btrfs_qgroup_info {
+u64 rfer; /* referenced */
+u64 rfer_cmpr; /* referenced compressed */
+u64 excl; /* exclusive */
+u64 excl_cmpr; /* exclusive compressed */
+
+/*
+* reservation tracking
+*/
+u64 reserved;
+};
+
+struct btrfs_qgroup_limits {
+u64 flags; /* which limits are set */
+u64 max_rfer;
+u64 max_excl;
+u64 rsv_rfer;
+u64 rsv_excl;
+};
+
struct btrfs_qgroup {
struct rb_node rb_node;
struct rb_node sort_node;
@@ -40,23 +60,20 @@ struct btrfs_qgroup {
struct rb_node all_parent_node;
u64 qgroupid;
+ u64 generation;
+
/*
* info_item
*/
- u64 generation;
- u64 rfer; /*referenced*/
- u64 rfer_cmpr; /*referenced compressed*/
- u64 excl; /*exclusive*/
- u64 excl_cmpr; /*exclusive compressed*/
+ struct btrfs_qgroup_infodata_info;
+ struct btrfs_qgroup_infometadata_info;
/*
*limit_item
*/
- u64 flags; /*which limits are set*/
- u64 max_rfer;
- u64 max_excl;
- u64 rsv_rfer;
- u64 rsv_excl;
+ struct btrfs_qgroup_limits data_limits
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
Okey, I found a patch from Mark 35 weeks ago which was not merged into xfstest: [PATCH] xfstests/btrfs: add test for quota groups and drop snapshot It looks can answer my question here. Thanx Yang On Mon, Mar 16, 2015 at 12:59 PM, Dongsheng Yang yangds.f...@cn.fujitsu.com wrote: On 03/04/2015 04:20 AM, Mark Fasheh wrote: On Tue, Mar 03, 2015 at 09:00:36AM -0500, Josef Bacik wrote: On 03/03/2015 06:18 AM, Dongsheng Yang wrote: On 02/26/2015 02:05 PM, Dongsheng Yang wrote: Wait a minute, this patch seems not working well in accounting quota number when deleting data shared by different subvolumes. I will investigate more about it and send a V2 out. I have sent a fstest [PATCH] fstest: btrfs: add a test for quota number when deleting a subvol. for this problem I was trying to solve in this patch. Please consider reverting the two commits introduced the problem: e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota) 1152651a (btrfs: qgroup: account shared subtrees during snapshot delete) We aren't reverting these two commits, what we had was worse before than it is now. We need to figure out why this is resulting in problems and fix that rather than throwing out a whole bunch of work. Thanks, Agreed, reverting these would re-introduce far more problems than it would solve. Hi Josef and Mark, Sorry for the late. Thanx for your reply and sorry I am not clear about the far more problems. From the result below, I saw there are some diff reports in V4.0-rc1 and when I revert these two commits. It's consistent now. But I think there must be something I am missing. I am glad to investigate more about it and solve it. But at first I need to know what's the problem SUBTREE is trying to solve. Example: mkfs.btrfs /dev/sdc -f mount /dev/sdc /mnt btrfs quota enable /mnt btrfs sub create /mnt/sub dd if=/dev/zero of=/mnt/sub/data bs=1024 count=1000 btrfs sub snapshot /mnt/sub /mnt/snap btrfs qgroup create 1/1 /mnt btrfs qgroup assign 0/257 1/1 /mnt btrfs quota rescan -w /mnt sync btrfs sub delete /mnt/sub --- -wait for cleaner_kthread. btrfs qgroup show /mnt umount /mnt btrfs check --qgroup-report /dev/sdc |grep diff Result: (1). V4.0-rc1 # btrfs qgroup show /mnt qgroupid rfer excl 0/5 16.00KiB 16.00KiB 0/257 1000.00KiB0.00B 0/258 1016.00KiB 1016.00KiB 1/11000.00KiB0.00B # btrfs check --qgroup-report /dev/sdc |grep diff Counts for qgroup id: 257 are different diff:referenced -1024000 referenced compressed -1024000 Counts for qgroup id: 281474976710657 are different diff:referenced -1024000 referenced compressed -1024000 (2). V4.0-rc1 with reverts of 96c92f and c04cad # btrfs qgroup show /mnt qgroupid rfer excl 0/5 16.00KiB 16.00KiB 0/257 0.00B0.00B 0/258 1016.00KiB 1016.00KiB 1/1 0.00B0.00B # btrfs check --qgroup-report /dev/sdc |grep diff # *No diff in btrfs check --qgroup-report* Thanx --Mark -- Mark Fasheh . -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/7] Btrfs: qgroup: part-4: Add type to btrfs qgroup.
Hi all,
I am cooking a V2 for this patchset, it will make user to limit
data, metadata or mixed of a qgroup.
Even, you can limit all of the three quota number at one time. I believe
it's much better.
Thanx.
On 02/10/2015 06:24 PM, Dongsheng Yang wrote:
Hi all,
This patchset is based on [Btrfs: qgroup: part-3: bug fixes.]
I am introducing a type to qgroup, then we can get the numbers what we
only care about.
Easy way to get the code for testing:
btrfs: https://yangdongsh...@github.com/yangdongsheng/linux.git
qgroup_type
btrfs-progs: https://yangdongsh...@github.com/yangdongsheng/btrfs-progs.git
qgroup_type
with the patches applied in this series, both in btrfs and btrfs-progs:
[root@atest-guest linux_btrfs]# mkfs.btrfs -O qgroup-type /dev/sdc -f
Btrfs v3.18-76-ga900a61
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Turning ON incompat feature 'skinny-metadata': reduced-size metadata extent refs
Turning ON incompat feature 'qgroup-type': create qgroup in different type
fs created label (null) on /dev/sdc
nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB
[root@atest-guest linux_btrfs]# mount /dev/sdc /mnt/
[root@atest-guest linux_btrfs]# btrfs quota enable /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data --- default type is mixed.
[root@atest-guest linux_btrfs]# btrfs sub create --qgroup-type data /mnt/sub1
Create subvolume '/mnt/sub1'
Set qgroup arguments:
qgroup type: data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B0.00B --- --- data
--- create a data qgroup
[root@atest-guest linux_btrfs]# btrfs qgroup limit -e 5M 0/258 /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B 5.00MiB --- --- data
[root@atest-guest linux_btrfs]# dd if=/dev/zero of=/mnt/sub1/data bs=1024
dd: error writing \u2018/mnt/sub1/data\u2019: Disk quota exceeded
5121+0 records in
5120+0 records out
5242880 bytes (5.2 MB) copied, 0.0218646 s, 240 MB/s
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# ll /mnt/sub1/data
-rw-r--r--. 1 root root 5242880 Feb 10 16:17 /mnt/sub1/data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 163841638400 --- ---
metadata,data
0/257 163841638400 --- ---
metadata,data
0/258 5242880 52428800 5242880 --- --- data
--- excl == max_excl == data_size == 5M == 5242880
Dongsheng Yang (7):
Btrfs: qgroup: Add type to btrfs_qgroup.
btrfs: qgroup: apply type to the recording and accounting.
btrfs: qgroup: Apply type to btrfs_qgroup_inherit().
btrfs: qgroup: Apply qgroup type to qgroup creating.
btrfs: qgroup: Apply qgroup type to subvol creating.
btrfs: qgroup: apply type to quota rescan.
btrfs: qgroup: fix a bug when type of parent is different with
child's.
fs/btrfs/ctree.h | 10 ++-
fs/btrfs/extent-tree.c| 48 ++-
fs/btrfs/ioctl.c | 49 ---
fs/btrfs/qgroup.c | 186 --
fs/btrfs/qgroup.h | 29 ++-
fs/btrfs/tests/qgroup-tests.c | 19 +++--
fs/btrfs/transaction.c| 13 +--
include/uapi
[PATCH v2] fstest: btrfs: add a test for quota number when deleting a subvol.
This regression is introduced by two commits:
e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
changlog:
v1:
1. Fix lots of coding style problems pointed by Eryu.
2. Add more description for the regression.
tests/btrfs/083 | 118
tests/btrfs/083.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 122 insertions(+)
create mode 100755 tests/btrfs/083
create mode 100644 tests/btrfs/083.out
diff --git a/tests/btrfs/083 b/tests/btrfs/083
new file mode 100755
index 000..a5e85e3
--- /dev/null
+++ b/tests/btrfs/083
@@ -0,0 +1,118 @@
+#! /bin/bash
+# FS QA Test No. 083
+#
+# This is a case for the regression introduced by
+#
+# e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
+# 1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
+#
+# The problem is shown as below:
+# $ btrfs sub create /mnt/sub
+# Create subvolume '/mnt/sub'
+# $ fallocate -l 1M /mnt/sub/data
+# $ btrfs quota enable /mnt
+# $ sync
+# $ btrfs qgroup show --raw /mnt
+# qgroupid rfer excl
+#
+# 0/5 1638416384
+# 0/257 1064960 1064960
+# $ btrfs sub dele /mnt/sub
+# Delete subvolume (no-commit): '/mnt/sub'
+#
+# ... ... Wait for the
cleaner_kthread, about 30s.
+# $ btrfs qgroup show --raw /mnt
+# qgroupid rfer excl
+#
+# 0/5 1638416384
+# 0/257 1048576 1048576 ---sub 0/257 is
deleted, but the quota numbers are not 0.
+#
+#
+# What we are expecting is:
+# $ btrfs qgroup show --raw /mnt
+# $ qgroupid rfer excl
+#
+# 0/5 1638416384
+# 0/257 00
+#
+# Currently, just revert these two commits mentioned above, we
+# can get the expected result. But there are some more things
+# we need to consider.
+#
+#---
+# Copyright (c) 2015 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo QA output created by $seq
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap _cleanup; exit \$status 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+rm -f $seqres.full
+
+_scratch_mkfs $seqres.full 21
+_scratch_mount
+
+$XFS_IO_PROG -f -d -c pwrite 0 8K $SCRATCH_MNT/foo 21 /dev/null
+
+_run_btrfs_util_prog subvolume snapshot $SCRATCH_MNT $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog quota enable $SCRATCH_MNT
+_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+
+units=`_btrfs_qgroup_units`
+orig_result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+
+_run_btrfs_util_prog subvolume delete $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog filesystem sync $SCRATCH_MNT
+
+timeout=100
+# There is a background thread doing the clean work
+for ((i=0; i$timeout; i++)); do
+ result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+ if (($orig_result != $result)); then
+ break
+ fi
+ sleep 1
+done
+
+$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG '/[0-9]/ {print
$2 $3}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/083.out b/tests/btrfs/083.out
new file mode 100644
index 000..e73797e
--- /dev/null
+++ b/tests/btrfs/083.out
@@ -0,0 +1,3 @@
+QA output created by 083
+24576 24576
+0 0
diff --git a/tests/btrfs/group b/tests/btrfs/group
index fd2fa76..e98a154 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -85,3 +85,4 @@
080 auto snapshot
081 auto quick clone
082 auto quick remount
+083 auto qgroup
--
1.8.4.2
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
On 03/04/2015 04:20 AM, Mark Fasheh wrote: On Tue, Mar 03, 2015 at 09:00:36AM -0500, Josef Bacik wrote: On 03/03/2015 06:18 AM, Dongsheng Yang wrote: On 02/26/2015 02:05 PM, Dongsheng Yang wrote: Wait a minute, this patch seems not working well in accounting quota number when deleting data shared by different subvolumes. I will investigate more about it and send a V2 out. I have sent a fstest [PATCH] fstest: btrfs: add a test for quota number when deleting a subvol. for this problem I was trying to solve in this patch. Please consider reverting the two commits introduced the problem: e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota) 1152651a (btrfs: qgroup: account shared subtrees during snapshot delete) We aren't reverting these two commits, what we had was worse before than it is now. We need to figure out why this is resulting in problems and fix that rather than throwing out a whole bunch of work. Thanks, Agreed, reverting these would re-introduce far more problems than it would solve. Hi Josef and Mark, Sorry for the late. Thanx for your reply and sorry I am not clear about the far more problems. From the result below, I saw there are some diff reports in V4.0-rc1 and when I revert these two commits. It's consistent now. But I think there must be something I am missing. I am glad to investigate more about it and solve it. But at first I need to know what's the problem SUBTREE is trying to solve. Example: mkfs.btrfs /dev/sdc -f mount /dev/sdc /mnt btrfs quota enable /mnt btrfs sub create /mnt/sub dd if=/dev/zero of=/mnt/sub/data bs=1024 count=1000 btrfs sub snapshot /mnt/sub /mnt/snap btrfs qgroup create 1/1 /mnt btrfs qgroup assign 0/257 1/1 /mnt btrfs quota rescan -w /mnt sync btrfs sub delete /mnt/sub --- -wait for cleaner_kthread. btrfs qgroup show /mnt umount /mnt btrfs check --qgroup-report /dev/sdc |grep diff Result: (1). V4.0-rc1 # btrfs qgroup show /mnt qgroupid rfer excl 0/5 16.00KiB 16.00KiB 0/257 1000.00KiB0.00B 0/258 1016.00KiB 1016.00KiB 1/11000.00KiB0.00B # btrfs check --qgroup-report /dev/sdc |grep diff Counts for qgroup id: 257 are different diff:referenced -1024000 referenced compressed -1024000 Counts for qgroup id: 281474976710657 are different diff:referenced -1024000 referenced compressed -1024000 (2). V4.0-rc1 with reverts of 96c92f and c04cad # btrfs qgroup show /mnt qgroupid rfer excl 0/5 16.00KiB 16.00KiB 0/257 0.00B0.00B 0/258 1016.00KiB 1016.00KiB 1/1 0.00B0.00B # btrfs check --qgroup-report /dev/sdc |grep diff # *No diff in btrfs check --qgroup-report* Thanx --Mark -- Mark Fasheh . -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] fstest: btrfs: add a test for quota number when deleting a subvol.
Hi Guan, sorry for the late.
On 03/06/2015 01:06 PM, Eryu Guan wrote:
On Tue, Mar 03, 2015 at 07:13:30PM +0800, Dongsheng Yang wrote:
This regression is introduced by two commits:
e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
I think you should add more description about the issue you're going to
test. I see only failures like
[root@dhcp-66-86-11 xfstests]# diff -u tests/btrfs/084.out
/root/xfstests/results//btrfs/084.out.bad
--- tests/btrfs/084.out 2015-03-06 12:47:02.31900 +0800
+++ /root/xfstests/results//btrfs/084.out.bad 2015-03-06
12:48:03.70700 +0800
@@ -1,3 +1,3 @@
QA output created by 084
24576 24576
-0 0
+8192 0
but from the test I don't know why 8192 0 is expected result not 0 0
Sure, I will add more description in V2 to explain why we are expecting
0 0 here.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
tests/btrfs/084 | 84 +
tests/btrfs/084.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 88 insertions(+)
create mode 100755 tests/btrfs/084
create mode 100644 tests/btrfs/084.out
diff --git a/tests/btrfs/084 b/tests/btrfs/084
new file mode 100755
index 000..7b7c7ac
--- /dev/null
+++ b/tests/btrfs/084
@@ -0,0 +1,84 @@
+#! /bin/bash
+# FS QA Test No. 084
+#
+# This is a case for the regression introduced by
+#
+# e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
+# 1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
+#
+#---
+# Copyright (c) 2015 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo QA output created by $seq
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap _cleanup; exit \$status 0 1 2 3 15
+
+_cleanup()
+{
+cd /
+rm -f $tmp.*
Better to use a tab not 4 spaces, maybe new should be updated too (in
another patch)
Thanx, will send another patch for ./new.
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+_scratch_mount
_scratch_mkfs first before you mount it.
And you need to remove $seqres.full before test, since you called
_run_btrfs_util_prog and it will dump commands and outputs to
$seqres.full, the file will keep growing over runs if you don't remove
it first.
rm -f $seqres.full
Great! thanx for your review.
+
+$XFS_IO_PROG -f -d -c pwrite 0 8K $SCRATCH_MNT/foo 21 /dev/null
+
+_run_btrfs_util_prog subvolume snapshot $SCRATCH_MNT $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog quota enable $SCRATCH_MNT
+_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+
+units=`_btrfs_qgroup_units`
+orig_result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+
+_run_btrfs_util_prog subvolume delete $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog filesystem sync $SCRATCH_MNT
+
+timeout=100
+# There is a background thread doing the clean work
+for ((i=0; i$timeout; i++))
+do
+ result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+ if (($orig_result != $result))
+ then
+ break
+ fi
+ sleep 1
I'm not sure if we need 100 iterations and sleeping 1 sec after each
iteration, that means we need 100s for a PASS run.
The 100sec is the limit for the loop, it means if the clean work is done
in 100sec, we will
break immediately. We don't have to wait for 100s in each test.
+done
Please follow this code style, as Dave pointed before
for ; do
# code
done
if ; then
# code
fi
Great,
+
+$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG '/[0-9]/ {print $2
$3}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/084.out b/tests/btrfs/084.out
new file mode 100644
index 000..7bf6dbf
--- /dev/null
+++ b/tests/btrfs/084.out
@@ -0,0 +1,3 @@
+QA output created by 084
+24576 24576
+0 0
diff --git a/tests/btrfs/group b/tests/btrfs/group
index fe82a9c..1be7392 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -86,3 +86,4
Re: [PATCH] fstest: btrfs: add a test for quota number when deleting a subvol.
On 03/16/2015 01:33 PM, Eryu Guan wrote:
On Mon, Mar 16, 2015 at 01:06:52PM +0800, Dongsheng Yang wrote:
Hi Guan, sorry for the late.
On 03/06/2015 01:06 PM, Eryu Guan wrote:
On Tue, Mar 03, 2015 at 07:13:30PM +0800, Dongsheng Yang wrote:
This regression is introduced by two commits:
e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
[snip]
+_cleanup()
+{
+cd /
+rm -f $tmp.*
Better to use a tab not 4 spaces, maybe new should be updated too (in
another patch)
Thanx, will send another patch for ./new.
Just FYI, I've already sent out the fix, please see
http://www.spinics.net/lists/fstests/msg01073.html
Great!! Thanx for your information. But I found some other 4 spaces in
new, I am not sure
whether we also need to replace them with tab or not as shown below.
Thanx
commit 931340c0c5599ae2c6714df16c796ea24240a5a7
Author: Dongsheng Yang yangds.f...@cn.fujitsu.com
Date: Mon Mar 16 01:15:53 2015 -0400
new: replace 4 spaces with a tab.
Sugguested-by: Eryu Guan eg...@redhat.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
diff --git a/new b/new
index 86f9075..f94daed 100755
--- a/new
+++ b/new
@@ -29,15 +29,15 @@ trap rm -f /tmp/$$.; exit 0 1 2 3 15
_cleanup()
{
-:
+ :
}
SRC_GROUPS=`find tests -not -path tests -type d -printf %f `
usage()
{
-echo Usage $0 test_dir
-echo Available dirs are: $SRC_GROUPS
-exit
+ echo Usage $0 test_dir
+ echo Available dirs are: $SRC_GROUPS
+ exit
}
[ $# -eq 0 ] usage
@@ -46,8 +46,8 @@ shift
if [ ! -f $tdir/group ]
then
-echo Creating the $tdir/group index ...
-cat 'End-of-File' $tdir/group
+ echo Creating the $tdir/group index ...
+ cat 'End-of-File' $tdir/group
# QA groups control
#
# define groups and default group owners
@@ -65,15 +65,15 @@ fi
if [ ! -w $tdir/group ]
then
-chmod u+w $tdir/group
-echo Warning: making the index file \$tdir/group\ writeable
+ chmod u+w $tdir/group
+ echo Warning: making the index file \$tdir/group\ writeable
fi
if make
then
-:
+ :
else
-echo Warning: make failed -- some tests may be missing
+ echo Warning: make failed -- some tests may be missing
fi
i=0
@@ -83,16 +83,16 @@ eof=1
for found in `cat $tdir/group | $AWK_PROG '{ print $1 }'`
do
-line=$((line+1))
-if [ -z $found ] || [ $found == # ];then
+ line=$((line+1))
+ if [ -z $found ] || [ $found == # ];then
continue
-fi
-i=$((i+1))
-id=`printf %03d $i`
-if [ $id != $found ];then
+ fi
+ i=$((i+1))
+ id=`printf %03d $i`
+ if [ $id != $found ];then
eof=0
break
-fi
+ fi
done
if [ $eof -eq 1 ]; then
line=$((line+1))
@@ -104,9 +104,9 @@ echo Next test is $id
if [ -f $tdir/$id ]
then
-echo Error: test $id already exists!
-_cleanup
-exit 1
+ echo Error: test $id already exists!
+ _cleanup
+ exit 1
fi
echo -n Creating skeletal script for you to edit ...
@@ -148,8 +148,8 @@ trap _cleanup; exit \\\$status 0 1 2 3 15
_cleanup()
{
-cd /
-rm -f \$tmp.*
+ cd /
+ rm -f \$tmp.*
}
# get standard environment, filters and checks
@@ -184,39 +184,39 @@ ${EDITOR-vi} $tdir/$id
if [ $# -eq 0 ]
then
-while true
-do
+ while true
+ do
echo -n Add to group(s) [other] (? for list):
read ans
[ -z $ans ] ans=other
if [ X$ans = X? ]
then
- for d in $SRC_GROUPS; do
+ for d in $SRC_GROUPS; do
l=$(sed -n tests/$d/group \
- -e 's/#.*//' \
- -e 's/$/ /' \
- -e 's;\(^[0-9][0-9][0-9]\)\(.*$\);\2;p')
+ -e 's/#.*//' \
+ -e 's/$/ /' \
+ -e 's;\(^[0-9][0-9][0-9]\)\(.*$\);\2;p')
grpl=$grpl $l
- done
- lst=`for word in $grpl; do echo $word; done | sort| uniq `
- echo $lst
+ done
+ lst=`for word in $grpl; do echo $word; done | sort| uniq `
+ echo $lst
else
- break
+ break
fi
-done
+ done
else
-# expert mode, groups are on the command line
-#
-for g in $*
-do
+ # expert mode, groups are on the command line
+ #
+ for g in $*
+ do
if grep ^$g[ ] $tdir/group /dev/null
then
- :
+ :
else
- echo Warning: group \$g\ not defined in $tdir/group
+ echo Warning: group \$g\ not defined in $tdir/group
fi
-done
-ans=$*
+ done
+ ans=$*
fi
Thanks,
Eryu Guan
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org
Re: Quota limit question
Sorry about the late reply!!
On 03/10/2015 04:41 PM, Qu Wenruo wrote:
Original Message
Subject: Re: Quota limit question
From: Christian Robottom Reis k...@canonical.com
To: Duncan 1i5t5.dun...@cox.net
Date: 2015年03月07日 05:44
Just as a follow-up, I upgraded btrfs-tools and the kernel again. I
currently have a filesystem which reports 1G exclusive use:
root@riff# btrfs qg show -r -e /var -p -c
qgroupid rfer excl max_rfer max_excl parent
child
--
-
0/261 1.52GiB 1.01GiB0.00B100.00GiB --- ---
It's recommended to sync the btrfs before 'qg show' command, since
quota tree is not updated until commit transaction.
Yes, sync is required be fore qg show now. There is no ioctl to get
qgroup info from kernel space,
so we have to read the data from disk then we have to do a sync before
we getting the correct information.
This filesystem reports over quota, and removing the quota fixes that:
root@riff# touch x
touch: cannot touch ‘x’: Disk quota exceeded
root@riff# btrfs qg limit -e
none 261 /var
root@riff# touch x
root@riff#
So at the moment quotas are pretty much unusable in kernel 3.18.6/tools
3.18.2, at least for my use case, and that's a bit surprising since
there isn't anything very interesting about it (other than it contains a
bunch of lxc-cloned rootfs).
Yes, Btrfs really has some problems, from yours to other annoying
incorrect values.
[Some guess]
For your case, I'm afraid your reserved space in your qgroup takes too
much space, and it's definitely a bug. Maybe some
btrfs_qgroup_reserve() leaking?
Agreed, there must be a reserve leaking in your use case. And I have posted
some patches about qgroup.reserved. some of them is about reserved leaking.
Could you try the tree :
btrfs:https://yangdongsh...@github.com/yangdongsheng/linux.git
qgroup_type
btrfs-progs:https://yangdongsh...@github.com/yangdongsheng/btrfs-progs.git
qgroup_type
It could be working better but I can not guarantee your problem here
will be solved by it.
[Reproducer]
Would you please describe your workload? Or some easy-to-reproduce
scripts?
[Image dump/debug tree]
Another one which may help is your btrfs-debug-tree output on your
quota tree.
You can dump it by btrfs-debug-tree -t 8 DEVICE.
Or full disk metadata dump by btrfs-image -c9 DEVICE
A reproducer is great for investigation. btrfs-debug-tree is important
to us.
Thanx
Yang
Thanks,
Qu
I've proactively added Yang who has submitted a few patches on quota
checking recently just to let me know if he thinks that this should be
fixed with a trunk kernel, or if he'd like to investigate or consider
this further. Thanks!
On Wed, Dec 24, 2014 at 03:52:41AM +, Duncan wrote:
Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as
excerpted:
On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis
wrote:
# btrfs qgroup limit 2000m 0/261 . touch x touch: cannot
touch
‘x’: Disk quota exceeded
The strange thing is that it doesn't seem to be actually out of
space:
# btrfs qgroup show -p -r -e /var | grep 261
0/261810048 391114752 2097152000 0 ---
Replying to myself as I had not yet been subscribed in time to
receive a
reply; I just upgraded to 3.18.1 and am seeing the same issue on the
same subvolume (and on no others).
Looking at the thread here on gmane.org (list2news and list2web
gateway),
it appears my reply was the only reply in any case, and it was
general as
I don't run quotas myself.
Basically I suggested upgrading, as the quota code as some rather huge
bugs in it (quotas could go seriously negative!) with the old versions
you were running. But you've upgraded at least the kernel now
(userspace
you didn't say).
Here's a link to the thread on the gmane web interface for
completeness,
but the above about covers my reply, as I said the only one until your
thread bump and my reply here, so there's not much new there unless
someone posts further followups to this thread...
http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
--
Duncan - List replies preferred. No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master. Richard Stallman
--
To unsubscribe from this list: send the line unsubscribe
linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4 1/3] Btrfs: get more accurate output in df command.
When function btrfs_statfs() calculate the tatol size of fs, it is calculating
the total size of disks and then dividing it by a factor. But in some usecase,
the result is not good to user.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 3.0G 1018M 1.3G 45% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a. df -h should report Size as 2GiB rather than as 3GiB.
Because this is 2 device raid1, the limiting factor is devid 1 @2GiB.
b. df -h should report Avail as 0.97GiB or less, rather than as 1.3GiB.
1.85 (the capacity of the allocated chunk)
-1.018 (the file stored)
+(2-1.85=0.15) (the residual capacity of the disks
considering a raid1 fs)
---
= 0.97
This patch drops the factor at all and calculate the size observable to
user without considering which raid level the data is in and what's the
size exactly in disk.
After this patch applied:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 2.0G 1.3G 713M 66% /mnt
# df /mnt
Filesystem 1K-blocksUsed Available Use% Mounted on
/dev/vdf12097152 1359424729536 66% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: e98c1321-645f-4457-b20d-4f41dc1cf2f4
Total devices 2 FS bytes used 1001.55MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a). The @Size is 2G as we expected.
b). @Available is 700M = 1.85G - 1.3G + (2G - 1.85G).
c). @Used is changed to 1.3G rather than 1018M as above. Because
this patch do not treat the free space in metadata chunk
and system chunk as available to user. It's true, user can
not use these space to store data, then it should not be
thought as available. At the same time, it will make the
@Used + @Available == @Size as possible to user.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
v3-v4:
Fix a trailing whitespace error found by checkpatch.
fs/btrfs/ctree.h | 1 -
fs/btrfs/extent-tree.c | 41
fs/btrfs/super.c | 72 --
3 files changed, 41 insertions(+), 73 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 84c3b00..d1ae425 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3460,7 +3460,6 @@ int btrfs_set_block_group_ro(struct btrfs_root *root,
void btrfs_set_block_group_rw(struct btrfs_root *root,
struct btrfs_block_group_cache *cache);
void btrfs_put_block_group_cache(struct btrfs_fs_info *info);
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo);
int btrfs_error_unpin_extent_range(struct btrfs_root *root,
u64 start, u64 end);
int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr,
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 571f402..857eb0e3 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -8515,47 +8515,6 @@ int btrfs_force_chunk_alloc(struct btrfs_trans_handle
*trans,
CHUNK_ALLOC_FORCE);
}
-/*
- * helper to account the unused space of all the readonly block group in the
- * space_info. takes mirrors into account.
- */
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo)
-{
- struct btrfs_block_group_cache *block_group;
- u64 free_bytes = 0;
- int factor;
-
- /* It's df, we don't care if it's racey */
- if (list_empty(sinfo-ro_bgs))
- return 0;
-
- spin_lock(sinfo-lock);
- list_for_each_entry(block_group, sinfo-ro_bgs, ro_list) {
- spin_lock(block_group-lock);
-
- if (!block_group-ro) {
- spin_unlock(block_group-lock);
- continue;
- }
-
- if (block_group-flags (BTRFS_BLOCK_GROUP_RAID1 |
- BTRFS_BLOCK_GROUP_RAID10 |
- BTRFS_BLOCK_GROUP_DUP))
- factor = 2;
- else
- factor = 1;
-
- free_bytes += (block_group-key.offset -
- btrfs_block_group_used(block_group-item)) *
- factor;
-
- spin_unlock(block_group-lock);
- }
- spin_unlock
[PATCH v3 1/3] Btrfs: get more accurate output in df command.
When function btrfs_statfs() calculate the tatol size of fs, it is calculating
the total size of disks and then dividing it by a factor. But in some usecase,
the result is not good to user.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 3.0G 1018M 1.3G 45% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a. df -h should report Size as 2GiB rather than as 3GiB.
Because this is 2 device raid1, the limiting factor is devid 1 @2GiB.
b. df -h should report Avail as 0.97GiB or less, rather than as 1.3GiB.
1.85 (the capacity of the allocated chunk)
-1.018 (the file stored)
+(2-1.85=0.15) (the residual capacity of the disks
considering a raid1 fs)
---
= 0.97
This patch drops the factor at all and calculate the size observable to
user without considering which raid level the data is in and what's the
size exactly in disk.
After this patch applied:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 2.0G 1.3G 713M 66% /mnt
# df /mnt
Filesystem 1K-blocksUsed Available Use% Mounted on
/dev/vdf12097152 1359424729536 66% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: e98c1321-645f-4457-b20d-4f41dc1cf2f4
Total devices 2 FS bytes used 1001.55MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a). The @Size is 2G as we expected.
b). @Available is 700M = 1.85G - 1.3G + (2G - 1.85G).
c). @Used is changed to 1.3G rather than 1018M as above. Because
this patch do not treat the free space in metadata chunk
and system chunk as available to user. It's true, user can
not use these space to store data, then it should not be
thought as available. At the same time, it will make the
@Used + @Available == @Size as possible to user.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ctree.h | 1 -
fs/btrfs/extent-tree.c | 41
fs/btrfs/super.c | 74 --
3 files changed, 42 insertions(+), 74 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 84c3b00..d1ae425 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3460,7 +3460,6 @@ int btrfs_set_block_group_ro(struct btrfs_root *root,
void btrfs_set_block_group_rw(struct btrfs_root *root,
struct btrfs_block_group_cache *cache);
void btrfs_put_block_group_cache(struct btrfs_fs_info *info);
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo);
int btrfs_error_unpin_extent_range(struct btrfs_root *root,
u64 start, u64 end);
int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr,
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 571f402..857eb0e3 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -8515,47 +8515,6 @@ int btrfs_force_chunk_alloc(struct btrfs_trans_handle
*trans,
CHUNK_ALLOC_FORCE);
}
-/*
- * helper to account the unused space of all the readonly block group in the
- * space_info. takes mirrors into account.
- */
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo)
-{
- struct btrfs_block_group_cache *block_group;
- u64 free_bytes = 0;
- int factor;
-
- /* It's df, we don't care if it's racey */
- if (list_empty(sinfo-ro_bgs))
- return 0;
-
- spin_lock(sinfo-lock);
- list_for_each_entry(block_group, sinfo-ro_bgs, ro_list) {
- spin_lock(block_group-lock);
-
- if (!block_group-ro) {
- spin_unlock(block_group-lock);
- continue;
- }
-
- if (block_group-flags (BTRFS_BLOCK_GROUP_RAID1 |
- BTRFS_BLOCK_GROUP_RAID10 |
- BTRFS_BLOCK_GROUP_DUP))
- factor = 2;
- else
- factor = 1;
-
- free_bytes += (block_group-key.offset -
- btrfs_block_group_used(block_group-item)) *
- factor;
-
- spin_unlock(block_group-lock);
- }
- spin_unlock(sinfo-lock);
-
- return free_bytes;
-}
-
void btrfs_set_block_group_rw(struct
[PATCH v3 2/3] Btrfs: raid56: simplify the parameter of nr_parity_stripes().
We just need the type of a chunk to calculate the number of parity stripes,
but we have to pass a structure of lookup_map to it. This will prevent some
callers to use it where there is no a convenient lookup_map to be passed.
This patch replace the parameter of struct map_lookup * with a profile type.
Then we can use it more easily.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
Reviewed-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com
---
fs/btrfs/raid56.h | 8
fs/btrfs/volumes.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
index 2b5d797..751fcd5 100644
--- a/fs/btrfs/raid56.h
+++ b/fs/btrfs/raid56.h
@@ -19,11 +19,11 @@
#ifndef __BTRFS_RAID56__
#define __BTRFS_RAID56__
-static inline int nr_parity_stripes(struct map_lookup *map)
+static inline int nr_parity_stripes(u64 type)
{
- if (map-type BTRFS_BLOCK_GROUP_RAID5)
+ if (type BTRFS_BLOCK_GROUP_RAID5)
return 1;
- else if (map-type BTRFS_BLOCK_GROUP_RAID6)
+ else if (type BTRFS_BLOCK_GROUP_RAID6)
return 2;
else
return 0;
@@ -31,7 +31,7 @@ static inline int nr_parity_stripes(struct map_lookup *map)
static inline int nr_data_stripes(struct map_lookup *map)
{
- return map-num_stripes - nr_parity_stripes(map);
+ return map-num_stripes - nr_parity_stripes(map-type);
}
#define RAID5_P_STRIPE ((u64)-2)
#define RAID6_Q_STRIPE ((u64)-1)
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index cd4d131..35043c9 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -5196,7 +5196,7 @@ static int __btrfs_map_block(struct btrfs_fs_info
*fs_info, int rw,
/* RAID[56] write or recovery. Return all stripes */
num_stripes = map-num_stripes;
- max_errors = nr_parity_stripes(map);
+ max_errors = nr_parity_stripes(map-type);
*length = map-stripe_len;
stripe_index = 0;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/3] Enhancement for df command.
Hi all, This is V3, just rebase it against 4.0-rc1. Any comment is welcome!! Thanx Dongsheng Yang (3): Btrfs: get more accurate output in df command. Btrfs: raid56: simplify the parameter of nr_parity_stripes(). Btrfs: adapt df command to RAID5/6. fs/btrfs/ctree.h | 1 - fs/btrfs/extent-tree.c | 41 fs/btrfs/raid56.h | 8 ++--- fs/btrfs/super.c | 86 +++--- fs/btrfs/volumes.c | 2 +- 5 files changed, 59 insertions(+), 79 deletions(-) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 3/3] Btrfs: adapt df command to RAID5/6.
When we use btrfs with raid5/6, the output of df is unexpected as below.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid5
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 4.2G 1.3G 2.9G 32% /mnt
[root@atest-guest linux_btrfs]# btrfs fi show /mnt
Label: none uuid: f7fac7f2-3898-482e-9cf2-fbcd7fdd7084
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
The @size should be 2G rather than 4.2G.
This patch makes the btrfs_calc_avail_data_space() consider raid5/6,
then we can get the correct result of it.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid5
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 2.0G 1.3G 713M 66% /mnt
[root@atest-guest linux_btrfs]# btrfs fi show /mnt
Label: none uuid: ea3a6e6e-fbe1-47aa-b4b5-bc37b98565d9
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/super.c | 16 ++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 25b0f79..7cac6b4 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -52,6 +52,7 @@
#include props.h
#include xattr.h
#include volumes.h
+#include raid56.h
#include export.h
#include compression.h
#include rcu-string.h
@@ -1698,6 +1699,14 @@ static int btrfs_calc_avail_data_space(struct btrfs_root
*root, u64 *free_bytes)
min_stripes = 4;
num_stripes = 4;
data_stripes = 2;
+ } else if (type BTRFS_BLOCK_GROUP_RAID5) {
+ min_stripes = 2;
+ num_stripes = nr_devices;
+ data_stripes = num_stripes - nr_parity_stripes(type);
+ } else if (type BTRFS_BLOCK_GROUP_RAID6) {
+ min_stripes = 3;
+ num_stripes = nr_devices;
+ data_stripes = num_stripes - nr_parity_stripes(type);
}
if (type BTRFS_BLOCK_GROUP_DUP)
@@ -1787,8 +1796,11 @@ static int btrfs_calc_avail_data_space(struct btrfs_root
*root, u64 *free_bytes)
while (nr_devices = min_stripes) {
if (num_stripes nr_devices) {
num_stripes = nr_devices;
- if (type BTRFS_BLOCK_GROUP_RAID0)
- data_stripes = num_stripes;
+ /* Only RAID0, RAID5 and RAID6 will get here.
+* And we can use the following calculation
+* for all the three cases.
+**/
+ data_stripes = num_stripes - nr_parity_stripes(type);
}
if (devices_info[i].max_avail = min_stripe_size) {
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] fstest: btrfs: add a test for quota number when deleting a subvol.
This regression is introduced by two commits:
e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
tests/btrfs/084 | 84 +
tests/btrfs/084.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 88 insertions(+)
create mode 100755 tests/btrfs/084
create mode 100644 tests/btrfs/084.out
diff --git a/tests/btrfs/084 b/tests/btrfs/084
new file mode 100755
index 000..7b7c7ac
--- /dev/null
+++ b/tests/btrfs/084
@@ -0,0 +1,84 @@
+#! /bin/bash
+# FS QA Test No. 084
+#
+# This is a case for the regression introduced by
+#
+# e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
+# 1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
+#
+#---
+# Copyright (c) 2015 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo QA output created by $seq
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap _cleanup; exit \$status 0 1 2 3 15
+
+_cleanup()
+{
+cd /
+rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+_scratch_mount
+
+$XFS_IO_PROG -f -d -c pwrite 0 8K $SCRATCH_MNT/foo 21 /dev/null
+
+_run_btrfs_util_prog subvolume snapshot $SCRATCH_MNT $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog quota enable $SCRATCH_MNT
+_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+
+units=`_btrfs_qgroup_units`
+orig_result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+
+_run_btrfs_util_prog subvolume delete $SCRATCH_MNT/snap
+
+_run_btrfs_util_prog filesystem sync $SCRATCH_MNT
+
+timeout=100
+# There is a background thread doing the clean work
+for ((i=0; i$timeout; i++))
+do
+ result=`$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG
'NR==NF {print $3}'`
+ if (($orig_result != $result))
+ then
+ break
+ fi
+ sleep 1
+done
+
+$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | $AWK_PROG '/[0-9]/ {print
$2 $3}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/084.out b/tests/btrfs/084.out
new file mode 100644
index 000..7bf6dbf
--- /dev/null
+++ b/tests/btrfs/084.out
@@ -0,0 +1,3 @@
+QA output created by 084
+24576 24576
+0 0
diff --git a/tests/btrfs/group b/tests/btrfs/group
index fe82a9c..1be7392 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -86,3 +86,4 @@
081 auto quick clone
082 auto quick remount
083 auto quick send
+084 qgroup
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] Btrfs: qgroup: part-2: bug fixes for qgroup reservation.
On 02/10/2015 06:23 PM, Dongsheng Yang wrote: Hi all, This is part2 for qgroup. It is based on [PATCH 0/9] Btrfs: qgroup: part-1: bug fixes. [1/4] is reviewed-by Josef. [2/4] - [4/4] are trying to solve a problem in qgroup reservation. It would be weired I introduce a may_use in 2/4 and delete it in 4/4. Yes, I am glad to rebase and squash them, here I am keeping this history of the problem for reviewing. I have explained the problem and sent the [2/4] out. Josef give me a looks good at that time. But in the later, I found another problem and solved it with [4/4]. So they are trying to solve different problems at different timing. When the all patches are reviewed by you guys, I am okey to squash them if you request. Hi, any comment on this patchset? Thanx Thanx. Dongsheng Yang (4): Btrfs: qgroup: free reserved in exceeding quota. Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use. Btrfs: qgroup, Account data space in more proper timings. btrfs: qgroup: do a reservation in a higher level. fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 25 +++-- fs/btrfs/file.c| 13 ++--- fs/btrfs/inode.c | 3 ++- fs/btrfs/qgroup.c | 10 -- fs/btrfs/relocation.c | 2 +- 6 files changed, 21 insertions(+), 34 deletions(-) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/7] Btrfs: qgroup: part-4: Add type to btrfs qgroup.
On 02/10/2015 06:24 PM, Dongsheng Yang wrote:
Hi all,
This patchset is based on [Btrfs: qgroup: part-3: bug fixes.]
I am introducing a type to qgroup, then we can get the numbers what we
only care about.
Easy way to get the code for testing:
btrfs: https://yangdongsh...@github.com/yangdongsheng/linux.git
qgroup_type
btrfs-progs: https://yangdongsh...@github.com/yangdongsheng/btrfs-progs.git
qgroup_type
with the patches applied in this series, both in btrfs and btrfs-progs:
Any comment on this patchset?
I think it's a good enhancement for qgroup.
Thanx
[root@atest-guest linux_btrfs]# mkfs.btrfs -O qgroup-type /dev/sdc -f
Btrfs v3.18-76-ga900a61
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Turning ON incompat feature 'skinny-metadata': reduced-size metadata extent refs
Turning ON incompat feature 'qgroup-type': create qgroup in different type
fs created label (null) on /dev/sdc
nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB
[root@atest-guest linux_btrfs]# mount /dev/sdc /mnt/
[root@atest-guest linux_btrfs]# btrfs quota enable /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data --- default type is mixed.
[root@atest-guest linux_btrfs]# btrfs sub create --qgroup-type data /mnt/sub1
Create subvolume '/mnt/sub1'
Set qgroup arguments:
qgroup type: data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B0.00B --- --- data
--- create a data qgroup
[root@atest-guest linux_btrfs]# btrfs qgroup limit -e 5M 0/258 /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B 5.00MiB --- --- data
[root@atest-guest linux_btrfs]# dd if=/dev/zero of=/mnt/sub1/data bs=1024
dd: error writing \u2018/mnt/sub1/data\u2019: Disk quota exceeded
5121+0 records in
5120+0 records out
5242880 bytes (5.2 MB) copied, 0.0218646 s, 240 MB/s
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# ll /mnt/sub1/data
-rw-r--r--. 1 root root 5242880 Feb 10 16:17 /mnt/sub1/data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 163841638400 --- ---
metadata,data
0/257 163841638400 --- ---
metadata,data
0/258 5242880 52428800 5242880 --- --- data
--- excl == max_excl == data_size == 5M == 5242880
Dongsheng Yang (7):
Btrfs: qgroup: Add type to btrfs_qgroup.
btrfs: qgroup: apply type to the recording and accounting.
btrfs: qgroup: Apply type to btrfs_qgroup_inherit().
btrfs: qgroup: Apply qgroup type to qgroup creating.
btrfs: qgroup: Apply qgroup type to subvol creating.
btrfs: qgroup: apply type to quota rescan.
btrfs: qgroup: fix a bug when type of parent is different with
child's.
fs/btrfs/ctree.h | 10 ++-
fs/btrfs/extent-tree.c| 48 ++-
fs/btrfs/ioctl.c | 49 ---
fs/btrfs/qgroup.c | 186 --
fs/btrfs/qgroup.h | 29 ++-
fs/btrfs/tests/qgroup-tests.c | 19 +++--
fs/btrfs/transaction.c| 13 +--
include/uapi/linux/btrfs.h| 1 +
8 files changed, 246 insertions(+), 109 deletions(-)
--
To unsubscribe from this list: send the line
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
On 02/26/2015 02:05 PM, Dongsheng Yang wrote:
Wait a minute, this patch seems not working well in accounting quota
number when
deleting data shared by different subvolumes.
I will investigate more about it and send a V2 out.
I have sent a fstest
[PATCH] fstest: btrfs: add a test for quota number when deleting a subvol.
for this problem I was trying to solve in this patch.
Please consider reverting the two commits introduced the problem:
e339a6b0 (Btrfs: __btrfs_mod_ref should always use no_quota)
1152651a (btrfs: qgroup: account shared subtrees during snapshot delete)
as what I attached.
Thanx
Thanx
Yang
On 02/13/2015 05:38 PM, Dongsheng Yang wrote:hen
Hi Josef and Mark,
I saw the commit message of 1152651a:
[btrfs: qgroup: account shared subtrees during snapshot delete]
that SUBTREE operation was introduced to account the quota number
in subvolume deleting.
But, I found it does not work well currently：
Create subvolume '/mnt/sub'
+ btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 16.00KiB 16.00KiB 0.00B 0.00B --- ---
+ dd if=/dev/zero of=/mnt/sub/data bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000876526 s, 117 MB/s
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 118784 0 0 --- ---
+ btrfs sub snap /mnt/sub /mnt/snap
Create a snapshot of '/mnt/sub' in '/mnt/snap'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
+ btrfs sub delete /mnt/sub
Delete subvolume (no-commit): '/mnt/sub'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
---
--- WAIT for cleaner
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 100.00KiB 0.00B 0.00B 0.00B --- --- --- the data size was
not cleaned from qgroup.
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- --- but the excl
in the snapshot qgroup is increased to 116KB by SUBTREE operation
My question is:
(1), SUBTREE is not working well.
(2), why we need a rule-breaker in qgroup operations SUBTREE to do
this work?
We can do it via the delayed-ref routine like the other qgroup
operations.
with my patch here applied and 1152651a reverted, I got the result:
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 0.00B 0.00B 0.00B 0.00B --- --- --- data and tree
block both are cleaned
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- -- quota numbers
in snapshot qgroup is correct.
(1), quota is working well now in subvolume deleting.
(2), all operations are inserted by delayed-ref.
So, the conclusion seems like that:
(1), commit 1152651a was introducing a SUBTREE operation to account
the number in subvolume deleted.
(2). SUBTREE is not working well.
(3). we can solve the same problem in a better way without
introducing a new operation of SUBTREE.
To be honest, I am suspecting myself *very much*. I trust you guys
and believe there must be something I am missing.
Then I send this mail to ask your help.
Thanx in advance.
Yang
On 02/10/2015 06:24 PM, Dongsheng Yang wrote:
In function of __btrfs_mod_ref(), we are passing the no_quota=1
to process_func() anyway. It will make the numbers in qgroup be
wrong when deleting a subvolume. The data size in a deleted subvolume
will never be decressed from all related qgroups.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 652be74..f59809c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3056,6 +3056,7 @@ static int __btrfs_mod_ref(struct
btrfs_trans_handle *trans,
int i;
int level;
int ret = 0;
+int no_quota = 0;
int (*process_func)(struct btrfs_trans_handle *, struct
btrfs_root *,
u64, u64, u64, u64, u64, u64, int);
@@ -3080,6 +3081,9 @@ static int __btrfs_mod_ref(struct
btrfs_trans_handle *trans,
else
parent = 0;
+if (!root-fs_info-quota_enabled || !is_fstree(ref_root))
+no_quota = 1;
+
for (i = 0; i nritems; i
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
Wait a minute, this patch seems not working well in accounting quota
number when
deleting data shared by different subvolumes.
I will investigate more about it and send a V2 out.
Thanx
Yang
On 02/13/2015 05:38 PM, Dongsheng Yang wrote:hen
Hi Josef and Mark,
I saw the commit message of 1152651a:
[btrfs: qgroup: account shared subtrees during snapshot delete]
that SUBTREE operation was introduced to account the quota number
in subvolume deleting.
But, I found it does not work well currently：
Create subvolume '/mnt/sub'
+ btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 16.00KiB 16.00KiB 0.00B 0.00B --- ---
+ dd if=/dev/zero of=/mnt/sub/data bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000876526 s, 117 MB/s
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 118784 0 0 --- ---
+ btrfs sub snap /mnt/sub /mnt/snap
Create a snapshot of '/mnt/sub' in '/mnt/snap'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
+ btrfs sub delete /mnt/sub
Delete subvolume (no-commit): '/mnt/sub'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
---
--- WAIT for cleaner
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 100.00KiB 0.00B 0.00B 0.00B --- --- --- the data size was
not cleaned from qgroup.
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- --- but the excl in
the snapshot qgroup is increased to 116KB by SUBTREE operation
My question is:
(1), SUBTREE is not working well.
(2), why we need a rule-breaker in qgroup operations SUBTREE to do
this work?
We can do it via the delayed-ref routine like the other qgroup
operations.
with my patch here applied and 1152651a reverted, I got the result:
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 0.00B 0.00B 0.00B 0.00B --- --- --- data and tree block
both are cleaned
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- -- quota numbers in
snapshot qgroup is correct.
(1), quota is working well now in subvolume deleting.
(2), all operations are inserted by delayed-ref.
So, the conclusion seems like that:
(1), commit 1152651a was introducing a SUBTREE operation to account
the number in subvolume deleted.
(2). SUBTREE is not working well.
(3). we can solve the same problem in a better way without introducing
a new operation of SUBTREE.
To be honest, I am suspecting myself *very much*. I trust you guys and
believe there must be something I am missing.
Then I send this mail to ask your help.
Thanx in advance.
Yang
On 02/10/2015 06:24 PM, Dongsheng Yang wrote:
In function of __btrfs_mod_ref(), we are passing the no_quota=1
to process_func() anyway. It will make the numbers in qgroup be
wrong when deleting a subvolume. The data size in a deleted subvolume
will never be decressed from all related qgroups.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 652be74..f59809c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3056,6 +3056,7 @@ static int __btrfs_mod_ref(struct
btrfs_trans_handle *trans,
int i;
int level;
int ret = 0;
+int no_quota = 0;
int (*process_func)(struct btrfs_trans_handle *, struct
btrfs_root *,
u64, u64, u64, u64, u64, u64, int);
@@ -3080,6 +3081,9 @@ static int __btrfs_mod_ref(struct
btrfs_trans_handle *trans,
else
parent = 0;
+if (!root-fs_info-quota_enabled || !is_fstree(ref_root))
+no_quota = 1;
+
for (i = 0; i nritems; i++) {
if (level == 0) {
btrfs_item_key_to_cpu(buf, key, i);
@@ -3098,7 +3102,7 @@ static int __btrfs_mod_ref(struct
btrfs_trans_handle *trans,
key.offset -= btrfs_file_extent_offset(buf, fi);
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, key.objectid,
- key.offset, 1);
+ key.offset, no_quota
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
Hi Josef and Mark,
I saw the commit message of 1152651a:
[btrfs: qgroup: account shared subtrees during snapshot delete]
that SUBTREE operation was introduced to account the quota number
in subvolume deleting.
But, I found it does not work well currently：
Create subvolume '/mnt/sub'
+ btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 16.00KiB 16.00KiB 0.00B 0.00B --- ---
+ dd if=/dev/zero of=/mnt/sub/data bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000876526 s, 117 MB/s
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 118784 0 0 --- ---
+ btrfs sub snap /mnt/sub /mnt/snap
Create a snapshot of '/mnt/sub' in '/mnt/snap'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
+ btrfs sub delete /mnt/sub
Delete subvolume (no-commit): '/mnt/sub'
+ sync
+ btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0 --- ---
0/257 118784 16384 0 0 --- ---
0/258 118784 16384 0 0 --- ---
---
--- WAIT for cleaner
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 100.00KiB 0.00B 0.00B 0.00B --- --- --- the data size was not
cleaned from qgroup.
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- --- but the excl in
the snapshot qgroup is increased to 116KB by SUBTREE operation
My question is:
(1), SUBTREE is not working well.
(2), why we need a rule-breaker in qgroup operations SUBTREE to do this
work?
We can do it via the delayed-ref routine like the other qgroup operations.
with my patch here applied and 1152651a reverted, I got the result:
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB 0.00B 0.00B --- ---
0/257 0.00B 0.00B 0.00B 0.00B --- --- --- data and tree block
both are cleaned
0/258 116.00KiB 116.00KiB 0.00B 0.00B --- --- -- quota numbers in
snapshot qgroup is correct.
(1), quota is working well now in subvolume deleting.
(2), all operations are inserted by delayed-ref.
So, the conclusion seems like that:
(1), commit 1152651a was introducing a SUBTREE operation to account the
number in subvolume deleted.
(2). SUBTREE is not working well.
(3). we can solve the same problem in a better way without introducing a
new operation of SUBTREE.
To be honest, I am suspecting myself *very much*. I trust you guys and
believe there must be something I am missing.
Then I send this mail to ask your help.
Thanx in advance.
Yang
On 02/10/2015 06:24 PM, Dongsheng Yang wrote:
In function of __btrfs_mod_ref(), we are passing the no_quota=1
to process_func() anyway. It will make the numbers in qgroup be
wrong when deleting a subvolume. The data size in a deleted subvolume
will never be decressed from all related qgroups.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 652be74..f59809c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3056,6 +3056,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
int i;
int level;
int ret = 0;
+ int no_quota = 0;
int (*process_func)(struct btrfs_trans_handle *, struct btrfs_root *,
u64, u64, u64, u64, u64, u64, int);
@@ -3080,6 +3081,9 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle *trans,
else
parent = 0;
+ if (!root-fs_info-quota_enabled || !is_fstree(ref_root))
+ no_quota = 1;
+
for (i = 0; i nritems; i++) {
if (level == 0) {
btrfs_item_key_to_cpu(buf, key, i);
@@ -3098,7 +3102,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
key.offset -= btrfs_file_extent_offset(buf, fi);
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, key.objectid,
- key.offset, 1);
+ key.offset, no_quota);
if (ret)
goto fail;
} else {
@@ -3106,7 +3110,7
Re: [PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
On 02/10/2015 07:24 PM, Filipe David Manana wrote:
On Tue, Feb 10, 2015 at 10:24 AM, Dongsheng Yang
yangds.f...@cn.fujitsu.com wrote:
In function of __btrfs_mod_ref(), we are passing the no_quota=1
to process_func() anyway. It will make the numbers in qgroup be
wrong when deleting a subvolume. The data size in a deleted subvolume
will never be decressed from all related qgroups.
How about getting a test case for xfstests for this?
Good idea!!
Will do it.
Thanx
thanks
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 652be74..f59809c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3056,6 +3056,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
int i;
int level;
int ret = 0;
+ int no_quota = 0;
int (*process_func)(struct btrfs_trans_handle *, struct btrfs_root *,
u64, u64, u64, u64, u64, u64, int);
@@ -3080,6 +3081,9 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
else
parent = 0;
+ if (!root-fs_info-quota_enabled || !is_fstree(ref_root))
+ no_quota = 1;
+
for (i = 0; i nritems; i++) {
if (level == 0) {
btrfs_item_key_to_cpu(buf, key, i);
@@ -3098,7 +3102,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
key.offset -= btrfs_file_extent_offset(buf, fi);
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, key.objectid,
- key.offset, 1);
+ key.offset, no_quota);
if (ret)
goto fail;
} else {
@@ -3106,7 +3110,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
num_bytes = root-nodesize;
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, level - 1, 0,
- 1);
+ no_quota);
if (ret)
goto fail;
}
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/9] Btrfs: qgroup: part-1: bug fixes for qgroup inherit.
Hi all,
[1/9] is a RESEND with a reviewed-by.
[2/9]-[6/9] are bug fixes about qgroup_inherit(), these patch make
user can set the limits when creating a subvolume with the related
patch in userspace applied.
[7/9]-[8/9] are a cleanup for qgroup.
[9/9] provides a way for btrfs-progs to get the quota status by ioctl(),
then btrfs-progs can warn user when they are using qgroup with quota_disabled.
With this series patch applied, both btrfs and btrfs-progs:
[root@atest-guest linux_btrfs]# btrfs quota enable /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
[root@atest-guest linux_btrfs]# btrfs sub create /mnt/sub1
Create subvolume '/mnt/sub1'
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
[root@atest-guest linux_btrfs]# btrfs sub create /mnt/sub2 -r 10M -e 5M
Create subvolume '/mnt/sub2'
Set qgroup arguments:
max reference: 10485760
max exclusive: 5242880
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
0/25816.00KiB 16.00KiB 10.00MiB 5.00MiB --- ---
Dongsheng Yang (8):
btrfs: qgroup: move WARN_ON() to the correct location.
btrfs: qgroup: inherit limit info from srcgroup in creating snapshot.
btrfs: qgroup: update qgroup in memory at the same time when we update
it in btree.
btrfs: qgroup: consolidate the parameter of fucntion
update_qgroup_limit_item().
btrfs: qgroup: update limit info in function btrfs_run_qgroups().
btrfs: qgroup: fix limit args override whole limit struct
Btrfs: qgroup: make the btree for qgroup increase from left to right.
Btrfs: qgroup: cleanup, remove an unsued parameter in
btrfs_create_qgroup().
Fan Chengniang (1):
btrfs: qgroup: obtain quota status
fs/btrfs/ioctl.c | 9 +++-
fs/btrfs/qgroup.c | 109 +++---
fs/btrfs/qgroup.h | 6 ++-
fs/btrfs/tests/qgroup-tests.c | 4 +-
include/uapi/linux/btrfs.h| 3 ++
5 files changed, 87 insertions(+), 44 deletions(-)
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 9/9] btrfs: qgroup: obtain quota status
From: Fan Chengniang fancn.f...@cn.fujitsu.com
add a function to obtain quota status. Now it can only get
whether quota is enabled. We can extend the function to get
more quota status
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
---
fs/btrfs/ioctl.c | 6 ++
fs/btrfs/qgroup.c | 14 ++
fs/btrfs/qgroup.h | 3 +++
include/uapi/linux/btrfs.h | 3 +++
4 files changed, 26 insertions(+)
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 658d83d..f6e730b 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -4572,6 +4572,12 @@ static long btrfs_ioctl_quota_ctl(struct file *file,
void __user *arg)
case BTRFS_QUOTA_CTL_DISABLE:
ret = btrfs_quota_disable(trans, root-fs_info);
break;
+ case BTRFS_QUOTA_CTL_STATUS:
+ ret = btrfs_quota_status(trans, root-fs_info, sa);
+ if (!ret)
+ if (copy_to_user(arg, sa, sizeof(*sa)))
+ ret = -EFAULT;
+ break;
default:
ret = -EINVAL;
break;
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 24c5bb1..b57fe45 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -993,6 +993,20 @@ out:
return ret;
}
+int btrfs_quota_status(struct btrfs_trans_handle *trans,
+ struct btrfs_fs_info *fs_info,
+ struct btrfs_ioctl_quota_ctl_args *sa)
+{
+ int ret = 0;
+
+ if (fs_info-quota_enabled)
+ sa-status |= BTRFS_QUOTA_STATUS_QUOTA_ENABLED;
+ else
+ sa-status = ~BTRFS_QUOTA_STATUS_QUOTA_ENABLED;
+
+ return ret;
+}
+
static void qgroup_dirty(struct btrfs_fs_info *fs_info,
struct btrfs_qgroup *qgroup)
{
diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h
index c5242aa..52d8b19 100644
--- a/fs/btrfs/qgroup.h
+++ b/fs/btrfs/qgroup.h
@@ -62,6 +62,9 @@ int btrfs_quota_enable(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info);
int btrfs_quota_disable(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info);
+int btrfs_quota_status(struct btrfs_trans_handle *trans,
+ struct btrfs_fs_info *fs_info,
+ struct btrfs_ioctl_quota_ctl_args *sa);
int btrfs_qgroup_rescan(struct btrfs_fs_info *fs_info);
void btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info);
int btrfs_qgroup_wait_for_completion(struct btrfs_fs_info *fs_info);
diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h
index 611e1c5..d4014e6 100644
--- a/include/uapi/linux/btrfs.h
+++ b/include/uapi/linux/btrfs.h
@@ -423,6 +423,9 @@ struct btrfs_ioctl_get_dev_stats {
#define BTRFS_QUOTA_CTL_ENABLE 1
#define BTRFS_QUOTA_CTL_DISABLE2
#define BTRFS_QUOTA_CTL_RESCAN__NOTUSED3
+#define BTRFS_QUOTA_CTL_STATUS 4
+
+#define BTRFS_QUOTA_STATUS_QUOTA_ENABLED 1
struct btrfs_ioctl_quota_ctl_args {
__u64 cmd;
__u64 status;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 5/9] btrfs: qgroup: update limit info in function btrfs_run_qgroups().
When we commit_transaction(), qgroups in btree should be updated. But, limit info is not considered currently. It will cause a problem when a qgroup of a snapshot inherit the limit info from srcqgroup, then there is an inconsistency. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/qgroup.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index a7ffebb..953befd 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -2154,6 +2154,10 @@ int btrfs_run_qgroups(struct btrfs_trans_handle *trans, if (ret) fs_info-qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT; + ret = update_qgroup_limit_item(trans, quota_root, qgroup); + if (ret) + fs_info-qgroup_flags |= + BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT; spin_lock(fs_info-qgroup_lock); } if (fs_info-quota_enabled) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/4] btrfs: qgroup: do a reservation in a higher level.
There are two problems in qgroup:
a). The PAGE_CACHE is 4K, even when we are writing a data of 1K,
qgroup will reserve a 4K size. It will cause the last 3K in a qgroup
is not available to user.
b). When user is writing a inline data, qgroup will not reserve it,
it means this is a window we can exceed the limit of a qgroup.
The main idea of this patch is reserving the data size of write_bytes
rather than the reserve_bytes. It means qgroup will not care about
the data size btrfs will reserve for user, but only care about the
data size user is going to write. Then reserve it when user want to
write and release it in transaction committed.
In this way, qgroup can be released from the complex procedure in
btrfs and only do the reserve when user want to write and account
when the data is written in commit_transaction().
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ctree.h | 2 +-
fs/btrfs/extent-tree.c | 38 +--
fs/btrfs/file.c| 4 +--
fs/btrfs/inode.c | 15 ---
fs/btrfs/qgroup.c | 70 +++---
fs/btrfs/qgroup.h | 4 ---
fs/btrfs/relocation.c | 2 +-
7 files changed, 14 insertions(+), 121 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 7e60741..1d17e7a 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3433,7 +3433,7 @@ enum btrfs_reserve_flush_enum {
BTRFS_RESERVE_FLUSH_ALL,
};
-int btrfs_check_data_free_space(struct inode *inode, u64 bytes);
+int btrfs_check_data_free_space(struct inode *inode, u64 bytes, u64
write_bytes);
void btrfs_free_reserved_data_space(struct inode *inode, u64 bytes);
void btrfs_trans_release_metadata(struct btrfs_trans_handle *trans,
struct btrfs_root *root);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 67c2e28..652be74 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3286,7 +3286,7 @@ again:
num_pages *= 16;
num_pages *= PAGE_CACHE_SIZE;
- ret = btrfs_check_data_free_space(inode, num_pages);
+ ret = btrfs_check_data_free_space(inode, num_pages, num_pages);
if (ret)
goto out_put;
@@ -3673,7 +3673,7 @@ u64 btrfs_get_alloc_profile(struct btrfs_root *root, int
data)
* This will check the space that the inode allocates from to make sure we have
* enough space for bytes.
*/
-int btrfs_check_data_free_space(struct inode *inode, u64 bytes)
+int btrfs_check_data_free_space(struct inode *inode, u64 bytes, u64
write_bytes)
{
struct btrfs_space_info *data_sinfo;
struct btrfs_root *root = BTRFS_I(inode)-root;
@@ -3774,7 +3774,7 @@ commit_trans:
data_sinfo-flags, bytes, 1);
return -ENOSPC;
}
- ret = btrfs_qgroup_reserve(root, bytes);
+ ret = btrfs_qgroup_reserve(root, write_bytes);
if (ret)
goto out;
data_sinfo-bytes_may_use += bytes;
@@ -3800,7 +3800,6 @@ void btrfs_free_reserved_data_space(struct inode *inode,
u64 bytes)
data_sinfo = root-fs_info-data_sinfo;
spin_lock(data_sinfo-lock);
WARN_ON(data_sinfo-bytes_may_use bytes);
- btrfs_qgroup_free(root, bytes);
data_sinfo-bytes_may_use -= bytes;
trace_btrfs_space_reservation(root-fs_info, space_info,
data_sinfo-flags, bytes, 0);
@@ -5041,8 +5040,6 @@ void btrfs_subvolume_release_metadata(struct btrfs_root
*root,
u64 qgroup_reserved)
{
btrfs_block_rsv_release(root, rsv, (u64)-1);
- if (qgroup_reserved)
- btrfs_qgroup_free(root, qgroup_reserved);
}
/**
@@ -5278,11 +5275,8 @@ out_fail:
to_free = 0;
}
spin_unlock(BTRFS_I(inode)-lock);
- if (dropped) {
- if (root-fs_info-quota_enabled)
- btrfs_qgroup_free(root, dropped * root-nodesize);
+ if (dropped)
to_free += btrfs_calc_trans_metadata_size(root, dropped);
- }
if (to_free) {
btrfs_block_rsv_release(root, block_rsv, to_free);
@@ -5321,9 +5315,6 @@ void btrfs_delalloc_release_metadata(struct inode *inode,
u64 num_bytes)
trace_btrfs_space_reservation(root-fs_info, delalloc,
btrfs_ino(inode), to_free, 0);
- if (root-fs_info-quota_enabled) {
- btrfs_qgroup_free(root, dropped * root-nodesize);
- }
btrfs_block_rsv_release(root, root-fs_info-delalloc_block_rsv,
to_free);
@@ -5348,7 +5339,7 @@ int btrfs_delalloc_reserve_space(struct inode *inode, u64
num_bytes)
{
int ret;
- ret = btrfs_check_data_free_space(inode, num_bytes);
+ ret = btrfs_check_data_free_space(inode, num_bytes, num_bytes);
if (ret)
return ret
[PATCH 4/9] btrfs: qgroup: consolidate the parameter of fucntion update_qgroup_limit_item().
Cleanup: Change the parameter of update_qgroup_limit_item() to the family of
update_qgroup_xxx_item().
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 51 ---
1 file changed, 24 insertions(+), 27 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 1aae0c3..a7ffebb 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -644,9 +644,8 @@ out:
}
static int update_qgroup_limit_item(struct btrfs_trans_handle *trans,
- struct btrfs_root *root, u64 qgroupid,
- u64 flags, u64 max_rfer, u64 max_excl,
- u64 rsv_rfer, u64 rsv_excl)
+ struct btrfs_root *root,
+ struct btrfs_qgroup *qgroup)
{
struct btrfs_path *path;
struct btrfs_key key;
@@ -657,7 +656,7 @@ static int update_qgroup_limit_item(struct
btrfs_trans_handle *trans,
key.objectid = 0;
key.type = BTRFS_QGROUP_LIMIT_KEY;
- key.offset = qgroupid;
+ key.offset = qgroup-qgroupid;
path = btrfs_alloc_path();
if (!path)
@@ -673,11 +672,11 @@ static int update_qgroup_limit_item(struct
btrfs_trans_handle *trans,
l = path-nodes[0];
slot = path-slots[0];
qgroup_limit = btrfs_item_ptr(l, slot, struct btrfs_qgroup_limit_item);
- btrfs_set_qgroup_limit_flags(l, qgroup_limit, flags);
- btrfs_set_qgroup_limit_max_rfer(l, qgroup_limit, max_rfer);
- btrfs_set_qgroup_limit_max_excl(l, qgroup_limit, max_excl);
- btrfs_set_qgroup_limit_rsv_rfer(l, qgroup_limit, rsv_rfer);
- btrfs_set_qgroup_limit_rsv_excl(l, qgroup_limit, rsv_excl);
+ btrfs_set_qgroup_limit_flags(l, qgroup_limit, qgroup-lim_flags);
+ btrfs_set_qgroup_limit_max_rfer(l, qgroup_limit, qgroup-max_rfer);
+ btrfs_set_qgroup_limit_max_excl(l, qgroup_limit, qgroup-max_excl);
+ btrfs_set_qgroup_limit_rsv_rfer(l, qgroup_limit, qgroup-rsv_rfer);
+ btrfs_set_qgroup_limit_rsv_excl(l, qgroup_limit, qgroup-rsv_excl);
btrfs_mark_buffer_dirty(l);
@@ -1184,15 +1183,6 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
ret = -ENOENT;
goto out;
}
- ret = update_qgroup_limit_item(trans, quota_root, qgroupid,
- limit-flags, limit-max_rfer,
- limit-max_excl, limit-rsv_rfer,
- limit-rsv_excl);
- if (ret) {
- fs_info-qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
- btrfs_info(fs_info, unable to update quota limit for %llu,
- qgroupid);
- }
spin_lock(fs_info-qgroup_lock);
qgroup-lim_flags = limit-flags;
@@ -1201,6 +1191,14 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
qgroup-rsv_rfer = limit-rsv_rfer;
qgroup-rsv_excl = limit-rsv_excl;
spin_unlock(fs_info-qgroup_lock);
+
+ ret = update_qgroup_limit_item(trans, quota_root, qgroup);
+ if (ret) {
+ fs_info-qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
+ btrfs_info(fs_info, unable to update quota limit for %llu,
+ qgroupid);
+ }
+
out:
mutex_unlock(fs_info-qgroup_ioctl_lock);
return ret;
@@ -2276,20 +2274,19 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
}
if (inherit inherit-flags BTRFS_QGROUP_INHERIT_SET_LIMITS) {
- ret = update_qgroup_limit_item(trans, quota_root, objectid,
- inherit-lim.flags,
- inherit-lim.max_rfer,
- inherit-lim.max_excl,
- inherit-lim.rsv_rfer,
- inherit-lim.rsv_excl);
- if (ret)
- goto unlock;
-
dstgroup-lim_flags = inherit-lim.flags;
dstgroup-max_rfer = inherit-lim.max_rfer;
dstgroup-max_excl = inherit-lim.max_excl;
dstgroup-rsv_rfer = inherit-lim.rsv_rfer;
dstgroup-rsv_excl = inherit-lim.rsv_excl;
+
+ ret = update_qgroup_limit_item(trans, quota_root, dstgroup);
+ if (ret) {
+ fs_info-qgroup_flags |=
BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
+ btrfs_info(fs_info, unable to update quota limit for
%llu,
+ dstgroup-qgroupid);
+ goto unlock;
+ }
}
if (srcid) {
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http
[PATCH 1/2] btrfs-progs:correct the return value
From: Fan Chengniang fancn.f...@cn.fujitsu.com
the return values 12 and 13 are not used spectially except as
return value. No description and definition about them. so I
change them to generic errno
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
---
qgroup.c | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/qgroup.c b/qgroup.c
index d59f4bb..5a4e393 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -21,6 +21,7 @@
#include ctree.h
#include ioctl.h
#include utils.h
+#include errno.h
#define BTRFS_QGROUP_NFILTERS_INCREASE (2 * BTRFS_QGROUP_FILTER_MAX)
#define BTRFS_QGROUP_NCOMPS_INCREASE (2 * BTRFS_QGROUP_COMP_MAX)
@@ -1294,7 +1295,7 @@ qgroup_inherit_realloc(struct btrfs_qgroup_inherit
**inherit, int n, int pos)
out = calloc(sizeof(*out) + sizeof(out-qgroups[0]) * (nitems + n), 1);
if (out == NULL) {
fprintf(stderr, ERROR: Not enough memory\n);
- return 13;
+ return -ENOMEM;
}
if (*inherit) {
@@ -1322,7 +1323,7 @@ int qgroup_inherit_add_group(struct btrfs_qgroup_inherit
**inherit, char *arg)
if (qgroupid == 0) {
fprintf(stderr, ERROR: bad qgroup specification\n);
- return 12;
+ return -EINVAL;
}
if (*inherit)
@@ -1349,7 +1350,7 @@ int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit
**inherit, char *arg,
if (!p) {
bad:
fprintf(stderr, ERROR: bad copy specification\n);
- return 12;
+ return -EINVAL;
}
*p = 0;
qgroup_src = parse_qgroupid(arg);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] btrfs: qgroup: return EINVAL if level of parent is not higher than child's.
When we create a subvol inheriting a qgroup, we need to check the level
of them. Otherwise, there is a chance a qgroup can inherit another qgroup
at the same level.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 5a1463d..d519055 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2244,6 +2244,11 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
ret = -EINVAL;
goto out;
}
+
+ if ((srcgroup-qgroupid 48) = (objectid 48)) {
+ ret = -EINVAL;
+ goto out;
+ }
++i_qgroups;
}
}
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] btrfs: qgroup: fix a wrong parameter of no_quota.
In function of __btrfs_mod_ref(), we are passing the no_quota=1
to process_func() anyway. It will make the numbers in qgroup be
wrong when deleting a subvolume. The data size in a deleted subvolume
will never be decressed from all related qgroups.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 652be74..f59809c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3056,6 +3056,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
int i;
int level;
int ret = 0;
+ int no_quota = 0;
int (*process_func)(struct btrfs_trans_handle *, struct btrfs_root *,
u64, u64, u64, u64, u64, u64, int);
@@ -3080,6 +3081,9 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
else
parent = 0;
+ if (!root-fs_info-quota_enabled || !is_fstree(ref_root))
+ no_quota = 1;
+
for (i = 0; i nritems; i++) {
if (level == 0) {
btrfs_item_key_to_cpu(buf, key, i);
@@ -3098,7 +3102,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
key.offset -= btrfs_file_extent_offset(buf, fi);
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, key.objectid,
- key.offset, 1);
+ key.offset, no_quota);
if (ret)
goto fail;
} else {
@@ -3106,7 +3110,7 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle
*trans,
num_bytes = root-nodesize;
ret = process_func(trans, root, bytenr, num_bytes,
parent, ref_root, level - 1, 0,
- 1);
+ no_quota);
if (ret)
goto fail;
}
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] btrfs-progs:set max_rfer and max_excl on creating new subvolume
From: Fan Chengniang fancn.f...@cn.fujitsu.com
add -r and -e options to enable setting max reference size
and max exclusive on creating new subvolumes.
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
---
Documentation/btrfs-subvolume.txt | 8 +++-
cmds-qgroup.c | 42 --
cmds-subvolume.c | 57 +++-
ioctl.h | 3 ++
qgroup.c | 92 +++
qgroup.h | 3 ++
6 files changed, 160 insertions(+), 45 deletions(-)
diff --git a/Documentation/btrfs-subvolume.txt
b/Documentation/btrfs-subvolume.txt
index b0ae030..fe13943 100644
--- a/Documentation/btrfs-subvolume.txt
+++ b/Documentation/btrfs-subvolume.txt
@@ -43,7 +43,7 @@ from normal directories.
SUBCOMMAND
---
-*create* [-i qgroupid] [dest]name::
+*create* [-i qgroupid] [-r max_rfer] [-e max_excl] [dest]name::
Create a subvolume name in dest.
+
If dest is not given, subvolume name will be created in the currently
@@ -54,6 +54,12 @@ directory.
-i qgroupid
Add the newly created subvolume to a qgroup. This option can be given multiple
times.
++
+-r max_rfer
+set the newly created subvolume's max reference size
++
+-e max_excl
+set the newly created subvolume's max exclusive size
*delete* [options] subvolume [subvolume...]::
Delete the subvolume(s) from the filesystem.
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 2172944..30f0851 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -106,48 +106,6 @@ static int qgroup_create(int create, int argc, char **argv)
return 0;
}
-static int parse_limit(const char *p, unsigned long long *s)
-{
- char *endptr;
- unsigned long long size;
-
- if (strcasecmp(p, none) == 0) {
- *s = 0;
- return 1;
- }
- size = strtoull(p, endptr, 10);
- switch (*endptr) {
- case 'T':
- case 't':
- size *= 1024;
- /* fallthrough */
- case 'G':
- case 'g':
- size *= 1024;
- /* fallthrough */
- case 'M':
- case 'm':
- size *= 1024;
- /* fallthrough */
- case 'K':
- case 'k':
- size *= 1024;
- ++endptr;
- break;
- case 0:
- break;
- default:
- return 0;
- }
-
- if (*endptr)
- return 0;
-
- *s = size;
-
- return 1;
-}
-
static const char * const cmd_qgroup_assign_usage[] = {
btrfs qgroup assign src dst path,
Enable subvolume qgroup support for a filesystem.,
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index a31cb1b..f0e22ac 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -42,13 +42,15 @@ static const char * const subvolume_cmd_group_usage[] = {
};
static const char * const cmd_subvol_create_usage[] = {
- btrfs subvolume create [-i qgroupid] [dest/]name,
+ btrfs subvolume create [-i qgroupid] [-r max_rfer] [-e max_excl]
[dest/]name,
Create a subvolume,
Create a subvolume name in dest. If dest is not given,
subvolume name will be created in the current directory.,
,
-i qgroupid add the newly created subvolume to a qgroup. This,
option can be given multiple times.,
+ -r max_rfer set the newly created subvolume's max reference size,
+ -e max_excl set the newly created subvolume's max exclusive size,
NULL
};
@@ -66,7 +68,7 @@ static int cmd_subvol_create(int argc, char **argv)
optind = 1;
while (1) {
- int c = getopt(argc, argv, c:i:v);
+ int c = getopt(argc, argv, c:i:r:e:v);
if (c 0)
break;
@@ -85,6 +87,20 @@ static int cmd_subvol_create(int argc, char **argv)
goto out;
}
break;
+ case 'r':
+ res = qgroup_inherit_set_maxrfer(inherit, optarg);
+ if (res) {
+ retval = res;
+ goto out;
+ }
+ break;
+ case 'e':
+ res = qgroup_inherit_set_maxexcl(inherit, optarg);
+ if (res) {
+ retval = res;
+ goto out;
+ }
+ break;
default:
usage(cmd_subvol_create_usage);
}
@@ -129,6 +145,43 @@ static int cmd_subvol_create(int argc, char **argv)
printf(Create subvolume '%s/%s'\n, dstdir, newname);
if (inherit) {
struct btrfs_ioctl_vol_args_v2 args;
+ struct btrfs_ioctl_quota_ctl_args sa;
+
+
[PATCH 0/4] Btrfs: qgroup: part-2: bug fixes for qgroup reservation.
Hi all, This is part2 for qgroup. It is based on [PATCH 0/9] Btrfs: qgroup: part-1: bug fixes. [1/4] is reviewed-by Josef. [2/4] - [4/4] are trying to solve a problem in qgroup reservation. It would be weired I introduce a may_use in 2/4 and delete it in 4/4. Yes, I am glad to rebase and squash them, here I am keeping this history of the problem for reviewing. I have explained the problem and sent the [2/4] out. Josef give me a looks good at that time. But in the later, I found another problem and solved it with [4/4]. So they are trying to solve different problems at different timing. When the all patches are reviewed by you guys, I am okey to squash them if you request. Thanx. Dongsheng Yang (4): Btrfs: qgroup: free reserved in exceeding quota. Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use. Btrfs: qgroup, Account data space in more proper timings. btrfs: qgroup: do a reservation in a higher level. fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 25 +++-- fs/btrfs/file.c| 13 ++--- fs/btrfs/inode.c | 3 ++- fs/btrfs/qgroup.c | 10 -- fs/btrfs/relocation.c | 2 +- 6 files changed, 21 insertions(+), 34 deletions(-) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/9] btrfs: qgroup: update qgroup in memory at the same time when we update it in btree.
When we call btrfs_qgroup_inherit() with BTRFS_QGROUP_INHERIT_SET_LIMITS,
btrfs will update the limit info of qgroup in btree but forget to update
the qgroup in rbtree at the same time. It obviousely will cause an
inconsistency.
This patch fix it by updating the rbtree at the same time.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 28 +---
1 file changed, 17 insertions(+), 11 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index b2cee33..1aae0c3 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2230,17 +2230,6 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
if (ret)
goto out;
- if (inherit inherit-flags BTRFS_QGROUP_INHERIT_SET_LIMITS) {
- ret = update_qgroup_limit_item(trans, quota_root, objectid,
- inherit-lim.flags,
- inherit-lim.max_rfer,
- inherit-lim.max_excl,
- inherit-lim.rsv_rfer,
- inherit-lim.rsv_excl);
- if (ret)
- goto out;
- }
-
if (srcid) {
struct btrfs_root *srcroot;
struct btrfs_key srckey;
@@ -2286,6 +2275,23 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
goto unlock;
}
+ if (inherit inherit-flags BTRFS_QGROUP_INHERIT_SET_LIMITS) {
+ ret = update_qgroup_limit_item(trans, quota_root, objectid,
+ inherit-lim.flags,
+ inherit-lim.max_rfer,
+ inherit-lim.max_excl,
+ inherit-lim.rsv_rfer,
+ inherit-lim.rsv_excl);
+ if (ret)
+ goto unlock;
+
+ dstgroup-lim_flags = inherit-lim.flags;
+ dstgroup-max_rfer = inherit-lim.max_rfer;
+ dstgroup-max_excl = inherit-lim.max_excl;
+ dstgroup-rsv_rfer = inherit-lim.rsv_rfer;
+ dstgroup-rsv_excl = inherit-lim.rsv_excl;
+ }
+
if (srcid) {
srcgroup = find_qgroup_rb(fs_info, srcid);
if (!srcgroup)
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 8/9] Btrfs: qgroup: cleanup, remove an unsued parameter in btrfs_create_qgroup().
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ioctl.c | 3 +--
fs/btrfs/qgroup.c | 2 +-
fs/btrfs/qgroup.h | 3 +--
fs/btrfs/tests/qgroup-tests.c | 4 ++--
4 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index d49fe8a..658d83d 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -4669,8 +4669,7 @@ static long btrfs_ioctl_qgroup_create(struct file *file,
void __user *arg)
/* FIXME: check if the IDs really exist */
if (sa-create) {
- ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid,
- NULL);
+ ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid);
} else {
ret = btrfs_remove_qgroup(trans, root-fs_info, sa-qgroupid);
}
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index d3c261b..24c5bb1 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1094,7 +1094,7 @@ out:
}
int btrfs_create_qgroup(struct btrfs_trans_handle *trans,
- struct btrfs_fs_info *fs_info, u64 qgroupid, char *name)
+ struct btrfs_fs_info *fs_info, u64 qgroupid)
{
struct btrfs_root *quota_root;
struct btrfs_qgroup *qgroup;
diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h
index 18cc68c..c5242aa 100644
--- a/fs/btrfs/qgroup.h
+++ b/fs/btrfs/qgroup.h
@@ -70,8 +70,7 @@ int btrfs_add_qgroup_relation(struct btrfs_trans_handle
*trans,
int btrfs_del_qgroup_relation(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 src, u64 dst);
int btrfs_create_qgroup(struct btrfs_trans_handle *trans,
- struct btrfs_fs_info *fs_info, u64 qgroupid,
- char *name);
+ struct btrfs_fs_info *fs_info, u64 qgroupid);
int btrfs_remove_qgroup(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 qgroupid);
int btrfs_limit_qgroup(struct btrfs_trans_handle *trans,
diff --git a/fs/btrfs/tests/qgroup-tests.c b/fs/btrfs/tests/qgroup-tests.c
index ec3dcb2..bcc1c7a 100644
--- a/fs/btrfs/tests/qgroup-tests.c
+++ b/fs/btrfs/tests/qgroup-tests.c
@@ -232,7 +232,7 @@ static int test_no_shared_qgroup(struct btrfs_root *root)
init_dummy_trans(trans);
test_msg(Qgroup basic add\n);
- ret = btrfs_create_qgroup(NULL, fs_info, 5, NULL);
+ ret = btrfs_create_qgroup(NULL, fs_info, 5);
if (ret) {
test_msg(Couldn't create a qgroup %d\n, ret);
return ret;
@@ -301,7 +301,7 @@ static int test_multiple_refs(struct btrfs_root *root)
test_msg(Qgroup multiple refs test\n);
/* We have 5 created already from the previous test */
- ret = btrfs_create_qgroup(NULL, fs_info, 256, NULL);
+ ret = btrfs_create_qgroup(NULL, fs_info, 256);
if (ret) {
test_msg(Couldn't create a qgroup %d\n, ret);
return ret;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] Btrfs: qgroup: part-3: bug fixes for deleting subvolume.
Hi all, This is a part3 for qgroup, it is based on Btrfs: qgroup: part-2: bug fixes. Dongsheng Yang (3): btrfs: qgroup: return EINVAL if level of parent is not higher than child's. btrfs: qgroup: allow to remove qgroup which has parent but no child. btrfs: qgroup: fix a wrong parameter of no_quota. fs/btrfs/extent-tree.c | 8 ++-- fs/btrfs/qgroup.c | 35 ++- 2 files changed, 36 insertions(+), 7 deletions(-) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/9] btrfs: qgroup: inherit limit info from srcgroup in creating snapshot.
Currently, when we snapshot a subvol, snapshot will not copy the limits from srcqgroup. This patch make the qgroup in snapshot inherit the limit info when create a snapshot. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/qgroup.c | 8 1 file changed, 8 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 97159a8..b2cee33 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -2302,6 +2302,14 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle *trans, dstgroup-excl_cmpr = level_size; srcgroup-excl = level_size; srcgroup-excl_cmpr = level_size; + + /* inherit the limit info */ + dstgroup-lim_flags = srcgroup-lim_flags; + dstgroup-max_rfer = srcgroup-max_rfer; + dstgroup-max_excl = srcgroup-max_excl; + dstgroup-rsv_rfer = srcgroup-rsv_rfer; + dstgroup-rsv_excl = srcgroup-rsv_excl; + qgroup_dirty(fs_info, dstgroup); qgroup_dirty(fs_info, srcgroup); } -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/9] btrfs: qgroup: move WARN_ON() to the correct location.
In function qgroup_excl_accounting(), we need to WARN when qg-excl is less than what we want to free, same to child and parents. But currently, for parent qgroup, the WARN_ON() is located after freeing qg-excl. It will WARN out even we free it normally. This patch move this WARN_ON() before freeing qg-excl. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com Reviewed-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com --- fs/btrfs/qgroup.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 48b60db..97159a8 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -1431,9 +1431,8 @@ static int qgroup_excl_accounting(struct btrfs_fs_info *fs_info, qgroup = u64_to_ptr(unode-aux); qgroup-rfer += sign * oper-num_bytes; qgroup-rfer_cmpr += sign * oper-num_bytes; + WARN_ON(sign 0 qgroup-excl oper-num_bytes); qgroup-excl += sign * oper-num_bytes; - if (sign 0) - WARN_ON(qgroup-excl oper-num_bytes); qgroup-excl_cmpr += sign * oper-num_bytes; qgroup_dirty(fs_info, qgroup); -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 7/9] Btrfs: qgroup: make the btree for qgroup increase from left to right.
It's not a big deal, but I would rather make the btree follow the
convention to increase from left to right.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 34717ed..d3c261b 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -117,9 +117,9 @@ static struct btrfs_qgroup *find_qgroup_rb(struct
btrfs_fs_info *fs_info,
while (n) {
qgroup = rb_entry(n, struct btrfs_qgroup, node);
if (qgroup-qgroupid qgroupid)
- n = n-rb_left;
- else if (qgroup-qgroupid qgroupid)
n = n-rb_right;
+ else if (qgroup-qgroupid qgroupid)
+ n = n-rb_left;
else
return qgroup;
}
@@ -139,9 +139,9 @@ static struct btrfs_qgroup *add_qgroup_rb(struct
btrfs_fs_info *fs_info,
qgroup = rb_entry(parent, struct btrfs_qgroup, node);
if (qgroup-qgroupid qgroupid)
- p = (*p)-rb_left;
- else if (qgroup-qgroupid qgroupid)
p = (*p)-rb_right;
+ else if (qgroup-qgroupid qgroupid)
+ p = (*p)-rb_left;
else
return qgroup;
}
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 6/9] btrfs: qgroup: fix limit args override whole limit struct
btrfs_limit_group use arg limit to override the old qgroup_limit of corresponding qgroup. However, we should override part of old qgroup_limit according to the bit which has been set in arg limit. Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/qgroup.c | 15 ++- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 953befd..34717ed 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -1185,11 +1185,16 @@ int btrfs_limit_qgroup(struct btrfs_trans_handle *trans, } spin_lock(fs_info-qgroup_lock); - qgroup-lim_flags = limit-flags; - qgroup-max_rfer = limit-max_rfer; - qgroup-max_excl = limit-max_excl; - qgroup-rsv_rfer = limit-rsv_rfer; - qgroup-rsv_excl = limit-rsv_excl; + if (limit-flags BTRFS_QGROUP_LIMIT_MAX_RFER) + qgroup-max_rfer = limit-max_rfer; + if (limit-flags BTRFS_QGROUP_LIMIT_MAX_EXCL) + qgroup-max_excl = limit-max_excl; + if (limit-flags BTRFS_QGROUP_LIMIT_RSV_RFER) + qgroup-rsv_rfer = limit-rsv_rfer; + if (limit-flags BTRFS_QGROUP_LIMIT_RSV_EXCL) + qgroup-rsv_excl = limit-rsv_excl; + qgroup-lim_flags |= limit-flags; + spin_unlock(fs_info-qgroup_lock); ret = update_qgroup_limit_item(trans, quota_root, qgroup); -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 5/7] btrfs: qgroup: Apply qgroup type to subvol creating.
Although we have a type in each qgroup, the all qgroups
are created in a default mode. This patch provide a
method to user to specify the type of qgroup for a sub volume.
It uses an unused byte in btrfs_ioctl_vol_args_v2 to pass
a qgroup type from userspace to kernel. Then kernel can
create a qgroup with requested type to the subvolume.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ioctl.c | 38 --
fs/btrfs/qgroup.c | 9 ++---
fs/btrfs/qgroup.h | 2 +-
fs/btrfs/transaction.c | 2 +-
include/uapi/linux/btrfs.h | 1 +
5 files changed, 37 insertions(+), 15 deletions(-)
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index b1f46e8..a6d83c4 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -432,7 +432,8 @@ static noinline int create_subvol(struct inode *dir,
struct dentry *dentry,
char *name, int namelen,
u64 *async_transid,
- struct btrfs_qgroup_inherit *inherit)
+ struct btrfs_qgroup_inherit *inherit,
+ u8 qgroup_type)
{
struct btrfs_trans_handle *trans;
struct btrfs_key key;
@@ -476,7 +477,7 @@ static noinline int create_subvol(struct inode *dir,
trans-block_rsv = block_rsv;
trans-bytes_reserved = block_rsv.size;
- ret = btrfs_qgroup_inherit(trans, root-fs_info, 0, objectid, inherit);
+ ret = btrfs_qgroup_inherit(trans, root-fs_info, 0, objectid, inherit,
qgroup_type);
if (ret)
goto fail;
@@ -808,7 +809,8 @@ static noinline int btrfs_mksubvol(struct path *parent,
char *name, int namelen,
struct btrfs_root *snap_src,
u64 *async_transid, bool readonly,
- struct btrfs_qgroup_inherit *inherit)
+ struct btrfs_qgroup_inherit *inherit,
+ u8 qgroup_type)
{
struct inode *dir = parent-dentry-d_inode;
struct dentry *dentry;
@@ -851,7 +853,7 @@ static noinline int btrfs_mksubvol(struct path *parent,
async_transid, readonly, inherit);
} else {
error = create_subvol(dir, dentry, name, namelen,
- async_transid, inherit);
+ async_transid, inherit, qgroup_type);
}
if (!error)
fsnotify_mkdir(dir, dentry);
@@ -1594,7 +1596,8 @@ out:
static noinline int btrfs_ioctl_snap_create_transid(struct file *file,
char *name, unsigned long fd, int subvol,
u64 *transid, bool readonly,
- struct btrfs_qgroup_inherit *inherit)
+ struct btrfs_qgroup_inherit *inherit,
+ u8 qgroup_type)
{
int namelen;
int ret = 0;
@@ -1617,7 +1620,8 @@ static noinline int
btrfs_ioctl_snap_create_transid(struct file *file,
if (subvol) {
ret = btrfs_mksubvol(file-f_path, name, namelen,
-NULL, transid, readonly, inherit);
+NULL, transid, readonly, inherit,
+qgroup_type);
} else {
struct fd src = fdget(fd);
struct inode *src_inode;
@@ -1640,7 +1644,8 @@ static noinline int
btrfs_ioctl_snap_create_transid(struct file *file,
} else {
ret = btrfs_mksubvol(file-f_path, name, namelen,
BTRFS_I(src_inode)-root,
-transid, readonly, inherit);
+transid, readonly, inherit,
+qgroup_type);
}
fdput(src);
}
@@ -1663,7 +1668,8 @@ static noinline int btrfs_ioctl_snap_create(struct file
*file,
ret = btrfs_ioctl_snap_create_transid(file, vol_args-name,
vol_args-fd, subvol,
- NULL, false, NULL);
+ NULL, false, NULL,
+ BTRFS_QGROUP_TYPE_DEFAULT);
kfree(vol_args);
return ret;
@@ -1673,11 +1679,13 @@ static noinline int btrfs_ioctl_snap_create_v2(struct
file *file,
void __user *arg, int subvol)
{
struct btrfs_ioctl_vol_args_v2 *vol_args;
+ struct btrfs_root *root = BTRFS_I(file_inode(file))-root;
int ret;
u64 transid = 0;
u64 *ptr = NULL
[PATCH 3/4] Btrfs: qgroup, Account data space in more proper timings.
Currenly, in data writing, -reserved is accounted in
fill_delalloc(), but -may_use is released in clear_bit_hook()
which is called by btrfs_finish_ordered_io(). That's too late,
that said, between fill_delalloc() and btrfs_finish_ordered_io(),
the data is doublely accounted by qgroup. It will cause some
unexpected -EDQUOT.
Example:
# btrfs quota enable /root/btrfs-auto-test/
# btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
# btrfs qgroup limit 1G /root/btrfs-auto-test//sub
dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024 count=150
dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded
681353+0 records in
681352+0 records out
697704448 bytes (698 MB) copied, 8.15563 s, 85.5 MB/s
It's (698 MB) when we got an -EDQUOT, but we limit it by 1G.
This patch move the btrfs_qgroup_reserve/free() for data from
btrfs_delalloc_reserve/release_metadata() to btrfs_check_data_free_space()
and btrfs_free_reserved_data_space(). Then the accounter in qgroup
will be updated at the same time with the accounter in space_info updated.
In this way, the unexpected -EDQUOT will be killed.
Reported-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 16 +---
fs/btrfs/file.c| 9 -
2 files changed, 9 insertions(+), 16 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index d1a7ce0..67c2e28 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3774,12 +3774,16 @@ commit_trans:
data_sinfo-flags, bytes, 1);
return -ENOSPC;
}
+ ret = btrfs_qgroup_reserve(root, bytes);
+ if (ret)
+ goto out;
data_sinfo-bytes_may_use += bytes;
trace_btrfs_space_reservation(root-fs_info, space_info,
data_sinfo-flags, bytes, 1);
+out:
spin_unlock(data_sinfo-lock);
- return 0;
+ return ret;
}
/*
@@ -3796,6 +3800,7 @@ void btrfs_free_reserved_data_space(struct inode *inode,
u64 bytes)
data_sinfo = root-fs_info-data_sinfo;
spin_lock(data_sinfo-lock);
WARN_ON(data_sinfo-bytes_may_use bytes);
+ btrfs_qgroup_free(root, bytes);
data_sinfo-bytes_may_use -= bytes;
trace_btrfs_space_reservation(root-fs_info, space_info,
data_sinfo-flags, bytes, 0);
@@ -5191,8 +5196,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode,
u64 num_bytes)
spin_unlock(BTRFS_I(inode)-lock);
if (root-fs_info-quota_enabled) {
- ret = btrfs_qgroup_reserve(root, num_bytes +
- nr_extents * root-nodesize);
+ ret = btrfs_qgroup_reserve(root, nr_extents * root-nodesize);
if (ret)
goto out_fail;
}
@@ -5200,8 +5204,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode,
u64 num_bytes)
ret = reserve_metadata_bytes(root, block_rsv, to_reserve, flush);
if (unlikely(ret)) {
if (root-fs_info-quota_enabled)
- btrfs_qgroup_free(root, num_bytes +
- nr_extents * root-nodesize);
+ btrfs_qgroup_free(root, nr_extents * root-nodesize);
goto out_fail;
}
@@ -5319,8 +5322,7 @@ void btrfs_delalloc_release_metadata(struct inode *inode,
u64 num_bytes)
trace_btrfs_space_reservation(root-fs_info, delalloc,
btrfs_ino(inode), to_free, 0);
if (root-fs_info-quota_enabled) {
- btrfs_qgroup_free(root, num_bytes +
- dropped * root-nodesize);
+ btrfs_qgroup_free(root, dropped * root-nodesize);
}
btrfs_block_rsv_release(root, root-fs_info-delalloc_block_rsv,
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index e409025..0ab1333 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2527,7 +2527,6 @@ static long btrfs_fallocate(struct file *file, int mode,
{
struct inode *inode = file_inode(file);
struct extent_state *cached_state = NULL;
- struct btrfs_root *root = BTRFS_I(inode)-root;
u64 cur_offset;
u64 last_byte;
u64 alloc_start;
@@ -2555,11 +2554,6 @@ static long btrfs_fallocate(struct file *file, int mode,
ret = btrfs_check_data_free_space(inode, alloc_end - alloc_start);
if (ret)
return ret;
- if (root-fs_info-quota_enabled) {
- ret = btrfs_qgroup_reserve(root, alloc_end - alloc_start);
- if (ret)
- goto out_reserve_fail;
- }
mutex_lock(inode-i_mutex);
ret = inode_newsize_ok(inode
[PATCH 2/4] Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use.
Currently, for pre_alloc or delay_alloc, the bytes will be accounted
in space_info by the three guys.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
But on the other hand, in qgroup, there are only two counters to account the
bytes, qgroup-reserved and qgroup-excl. And qg-reserved accounts
bytes in space_info-bytes_may_use and qg-excl accounts bytes in
space_info-used. So the bytes in space_info-reserved is not accounted
in qgroup. If so, there is a window we can exceed the quota limit when
bytes is in space_info-reserved.
Example:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 20987904 20987904 010485760 --- ---
qg-excl is 20987904 larger than max_excl 10485760.
This patch introduce a new counter named may_use to qgroup, then
there are three counters in qgroup to account bytes in space_info
as below.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
qgroup-may_use --- qgroup-reserved --- qgroup-excl
With this patch applied:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
fallocate: /mnt/data9: fallocate failed: Disk quota exceeded
fallocate: /mnt/data10: fallocate failed: Disk quota exceeded
fallocate: /mnt/data11: fallocate failed: Disk quota exceeded
fallocate: /mnt/data12: fallocate failed: Disk quota exceeded
fallocate: /mnt/data13: fallocate failed: Disk quota exceeded
fallocate: /mnt/data14: fallocate failed: Disk quota exceeded
fallocate: /mnt/data15: fallocate failed: Disk quota exceeded
fallocate: /mnt/data16: fallocate failed: Disk quota exceeded
fallocate: /mnt/data17: fallocate failed: Disk quota exceeded
fallocate: /mnt/data18: fallocate failed: Disk quota exceeded
fallocate: /mnt/data19: fallocate failed: Disk quota exceeded
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rferexclmax_rfer max_excl parent child
-- -
0/5 9453568 9453568 010485760 --- ---
Reported-by: Cyril SCETBON cyril.scet...@free.fr
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 20 ++-
fs/btrfs/inode.c | 18 -
fs/btrfs/qgroup.c | 68 +++---
fs/btrfs/qgroup.h | 4 +++
4 files changed, 104 insertions(+), 6 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 88b4e32..d1a7ce0 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5512,8 +5512,12 @@ static int pin_down_extent(struct btrfs_root *root,
set_extent_dirty(root-fs_info-pinned_extents, bytenr,
bytenr + num_bytes - 1, GFP_NOFS | __GFP_NOFAIL);
- if (reserved)
+ if (reserved) {
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ num_bytes, -1);
trace_btrfs_reserved_extent_free(root, bytenr, num_bytes);
+ }
return 0;
}
@@ -6244,6 +6248,9 @@ void btrfs_free_tree_block(struct btrfs_trans_handle
*trans,
btrfs_update_reserved_bytes(cache, buf-len, RESERVE_FREE, 0);
trace_btrfs_reserved_extent_free(root, buf-start, buf-len);
pin = 0;
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ buf-len, -1);
}
out:
if (pin)
@@ -6978,7 +6985,11 @@ static int __btrfs_free_reserved_extent(struct
btrfs_root *root,
else {
btrfs_add_free_space(cache, start, len);
btrfs_update_reserved_bytes(cache, len, RESERVE_FREE, delalloc);
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ len, -1);
}
+
btrfs_put_block_group(cache);
trace_btrfs_reserved_extent_free(root, start, len);
@@ -7214,6 +7225,9 @@ int btrfs_alloc_logged_file_extent(struct
btrfs_trans_handle *trans,
BUG_ON(ret); /* logic error */
ret = alloc_reserved_file_extent(trans, root, 0, root_objectid,
0, owner, offset, ins, 1);
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ ins-offset, 1);
btrfs_put_block_group(block_group);
return ret
[PATCH 4/4] btrfs-progs:show qgroup type
From: Fan Chengniang fancn.f...@cn.fujitsu.com
make 'btrfs qgroup show' command show qgroup type.
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
---
qgroup.c | 42 +++---
qgroup.h | 1 +
2 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/qgroup.c b/qgroup.c
index a647925..d359659 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -135,6 +135,13 @@ static struct {
.max_len= 5,
},
{
+ .name = type,
+ .column_name= Type,
+ .need_print = 1,
+ .unit_mode = 0,
+ .max_len= 8,
+ },
+ {
.name = NULL,
.column_name= NULL,
.need_print = 0,
@@ -201,6 +208,20 @@ static int print_child_column(struct btrfs_qgroup *qgroup)
return len;
}
+static int print_type_column(struct btrfs_qgroup *qgroup)
+{
+ __u8 type = qgroup-type;
+ int len = 0;
+
+ if (type == BTRFS_QGROUP_TYPE_DATA)
+ len = printf(data);
+ if (type == BTRFS_QGROUP_TYPE_METADATA)
+ len = printf(metadata);
+ if (type == BTRFS_QGROUP_TYPE_MIXED)
+ len = printf(metadata,data);
+ return len;
+}
+
static void print_qgroup_column_add_blank(enum btrfs_qgroup_column_enum column,
int len)
{
@@ -244,6 +265,10 @@ static void print_qgroup_column(struct btrfs_qgroup
*qgroup,
len = print_child_column(qgroup);
print_qgroup_column_add_blank(BTRFS_QGROUP_CHILD, len);
break;
+ case BTRFS_QGROUP_TYPE:
+ len = print_type_column(qgroup);
+ print_qgroup_column_add_blank(BTRFS_QGROUP_TYPE, len);
+ break;
default:
break;
}
@@ -258,7 +283,7 @@ static void print_single_qgroup_table(struct btrfs_qgroup
*qgroup)
continue;
print_qgroup_column(qgroup, i);
- if (i != BTRFS_QGROUP_CHILD)
+ if (i != BTRFS_QGROUP_ALL - 1)
printf( );
}
printf(\n);
@@ -275,7 +300,7 @@ static void print_table_head()
if (!btrfs_qgroup_columns[i].need_print)
continue;
if ((i == BTRFS_QGROUP_QGROUPID) | (i == BTRFS_QGROUP_PARENT) |
- (i == BTRFS_QGROUP_CHILD))
+ (i == BTRFS_QGROUP_CHILD) | (i == BTRFS_QGROUP_TYPE))
printf(%-*s, max_len, btrfs_qgroup_columns[i].name);
else
printf(%*s, max_len, btrfs_qgroup_columns[i].name);
@@ -287,7 +312,7 @@ static void print_table_head()
if (!btrfs_qgroup_columns[i].need_print)
continue;
if ((i == BTRFS_QGROUP_QGROUPID) | (i == BTRFS_QGROUP_PARENT) |
- (i == BTRFS_QGROUP_CHILD)) {
+ (i == BTRFS_QGROUP_CHILD) | (i == BTRFS_QGROUP_TYPE)) {
len = strlen(btrfs_qgroup_columns[i].name);
while (len--)
printf(-);
@@ -976,6 +1001,17 @@ static void __update_columns_max_len(struct btrfs_qgroup
*bq,
if (btrfs_qgroup_columns[column].max_len len)
btrfs_qgroup_columns[column].max_len = len;
break;
+ case BTRFS_QGROUP_TYPE:
+ len = 0;
+ if (bq-type == BTRFS_QGROUP_TYPE_DATA)
+ len = strlen(data);
+ if (bq-type == BTRFS_QGROUP_TYPE_METADATA)
+ len = strlen(metadata);
+ if (bq-type == BTRFS_QGROUP_TYPE_MIXED)
+ len = strlen(metadata,data);
+ if (btrfs_qgroup_columns[column].max_len len)
+ btrfs_qgroup_columns[column].max_len = len;
+ break;
default:
break;
}
diff --git a/qgroup.h b/qgroup.h
index 8423fdf..cc61477 100644
--- a/qgroup.h
+++ b/qgroup.h
@@ -75,6 +75,7 @@ enum btrfs_qgroup_column_enum {
BTRFS_QGROUP_MAX_EXCL,
BTRFS_QGROUP_PARENT,
BTRFS_QGROUP_CHILD,
+BTRFS_QGROUP_TYPE,
BTRFS_QGROUP_ALL,
};
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/4] btrfs-progs:specify qgroup type when creating qgroup
From: Fan Chengniang fancn.f...@cn.fujitsu.com
add --qgroup-type option to specify qgroup type. Type
can be metadata, data or mixed.
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
Documentation/btrfs-qgroup.txt | 5 +
cmds-qgroup.c | 37 ++---
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/Documentation/btrfs-qgroup.txt b/Documentation/btrfs-qgroup.txt
index eadfe1c..7b07e97 100644
--- a/Documentation/btrfs-qgroup.txt
+++ b/Documentation/btrfs-qgroup.txt
@@ -45,6 +45,11 @@ Create a subvolume quota group.
+
For the '0/subvolume id' qgroup, a qgroup can be created even before the
subvolume created.
++
+--qgroup-type=type
+specify the type of qgroup to set qgroup limit.
++
+attr can be one of metadata, data, mixed.
*destroy* qgroupid path::
Destroy a qgroup.
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 30f0851..23accdf 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -78,16 +78,45 @@ static int qgroup_create(int create, int argc, char **argv)
int ret = 0;
int fd;
int e;
- char *path = argv[2];
+ char *path = argv[argc - 1];
struct btrfs_ioctl_qgroup_create_args args;
DIR *dirstream = NULL;
+ __u8 qgroup_type = 0;
+ __u64 create_type;
- if (check_argc_exact(argc, 3))
+ optind = 1;
+ while (1) {
+ int c;
+ static const struct option long_options[] = {
+ {qgroup-type, required_argument, NULL, 'q'},
+ {NULL, 0, NULL, 0}
+ };
+
+ c = getopt_long(argc, argv, , long_options, NULL);
+ if (c 0)
+ break;
+
+ switch (c) {
+ case 'q':
+ ret = btrfs_qgroup_set_qgroup_type(qgroup_type,
optarg);
+ if (ret)
+ return -1;
+ break;
+ default:
+ return -1;
+ }
+ }
+
+ if (check_argc_exact(argc - optind, 2))
return -1;
memset(args, 0, sizeof(args));
args.create = create;
- args.qgroupid = parse_qgroupid(argv[1]);
+ if (create qgroup_type) {
+ create_type = qgroup_type;
+ args.create |= (create_type BTRFS_QGROUP_TYPE_SHIFT);
+ }
+ args.qgroupid = parse_qgroupid(argv[argc - 2]);
fd = open_file_or_dir(path, dirstream);
if (fd 0) {
@@ -137,6 +166,8 @@ static int cmd_qgroup_remove(int argc, char **argv)
static const char * const cmd_qgroup_create_usage[] = {
btrfs qgroup create qgroupid path,
Create a subvolume quota group.,
+ --qgroup-type=metadata,data,mixed,
+ specify which qgroup type to set qgroup limit,
NULL
};
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/7] btrfs: qgroup: Apply type to btrfs_qgroup_inherit().
Now, we have a type in each qgroup, then when we do a inherit(),
we need to consider the type of qgroup to decide account metadata
or not.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 18 ++
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 2fa8497..0129dae 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2382,10 +2382,20 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle
*trans,
dstgroup-type = srcgroup-type;
dstgroup-rfer = srcgroup-rfer;
dstgroup-rfer_cmpr = srcgroup-rfer_cmpr;
- dstgroup-excl = level_size;
- dstgroup-excl_cmpr = level_size;
- srcgroup-excl = level_size;
- srcgroup-excl_cmpr = level_size;
+ if (dstgroup-type BTRFS_QGROUP_TYPE_METADATA) {
+ dstgroup-excl = level_size;
+ dstgroup-excl_cmpr = level_size;
+ } else {
+ dstgroup-excl = 0;
+ dstgroup-excl_cmpr = 0;
+ }
+ if (srcgroup-type BTRFS_QGROUP_TYPE_METADATA) {
+ srcgroup-excl = level_size;
+ srcgroup-excl_cmpr = level_size;
+ } else {
+ srcgroup-excl = 0;
+ srcgroup-excl_cmpr = 0;
+ }
/* inherit the limit info */
dstgroup-lim_flags = srcgroup-lim_flags;
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/7] btrfs: qgroup: Apply qgroup type to qgroup creating.
This patch provide a method to user to create a qgroup
in a specified type.
It use a byte of the -create in btrfs_ioctl_qgroup_create_args
to pass the type of qgroup to kernel. Then there is a macro
named as BTRFS_QGROUP_TYPE_SHIFT to help kernel get the type
of qgroup user want to creat.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ioctl.c | 12 ++--
fs/btrfs/qgroup.h | 6 ++
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index a50f295..b1f46e8 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -4675,8 +4675,16 @@ static long btrfs_ioctl_qgroup_create(struct file *file,
void __user *arg)
/* FIXME: check if the IDs really exist */
if (sa-create) {
- ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid,
- BTRFS_QGROUP_TYPE_DEFAULT);
+ u8 type = (sa-create BTRFS_QGROUP_TYPE_SHIFT);
+ if (type) {
+ if (!btrfs_fs_incompat(root-fs_info, QGROUP_TYPE)) {
+ ret = -EINVAL;
+ goto out;
+ }
+ } else {
+ type = BTRFS_QGROUP_TYPE_DEFAULT;
+ }
+ ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid,
type);
} else {
ret = btrfs_remove_qgroup(trans, root-fs_info, sa-qgroupid);
}
diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h
index 75d0bb6..0be0548 100644
--- a/fs/btrfs/qgroup.h
+++ b/fs/btrfs/qgroup.h
@@ -30,6 +30,12 @@
#define BTRFS_QGROUP_TYPE_DEFAULT BTRFS_QGROUP_TYPE_MIXED
/*
+ * We are using the first byte in -create of btrfs_ioctl_qgroup_create_args
+ * to specify the type of the qgroup to be created.
+ */
+#define BTRFS_QGROUP_TYPE_SHIFT56
+
+/*
* A description of the operations, all of these operations only happen when we
* are adding the 1st reference for that subvolume in the case of adding space
* or on the last reference delete in the case of subtraction. The only
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC PATCH 0/7] Btrfs: qgroup: part-4: Add type to btrfs qgroup.
Hi all,
This patchset is based on [Btrfs: qgroup: part-3: bug fixes.]
I am introducing a type to qgroup, then we can get the numbers what we
only care about.
Easy way to get the code for testing:
btrfs: https://yangdongsh...@github.com/yangdongsheng/linux.git
qgroup_type
btrfs-progs: https://yangdongsh...@github.com/yangdongsheng/btrfs-progs.git
qgroup_type
with the patches applied in this series, both in btrfs and btrfs-progs:
[root@atest-guest linux_btrfs]# mkfs.btrfs -O qgroup-type /dev/sdc -f
Btrfs v3.18-76-ga900a61
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Turning ON incompat feature 'skinny-metadata': reduced-size metadata extent refs
Turning ON incompat feature 'qgroup-type': create qgroup in different type
fs created label (null) on /dev/sdc
nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB
[root@atest-guest linux_btrfs]# mount /dev/sdc /mnt/
[root@atest-guest linux_btrfs]# btrfs quota enable /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data --- default type is mixed.
[root@atest-guest linux_btrfs]# btrfs sub create --qgroup-type data /mnt/sub1
Create subvolume '/mnt/sub1'
Set qgroup arguments:
qgroup type: data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B0.00B --- --- data
--- create a data qgroup
[root@atest-guest linux_btrfs]# btrfs qgroup limit -e 5M 0/258 /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 16.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/25716.00KiB 16.00KiB0.00B0.00B --- ---
metadata,data
0/258 0.00B0.00B0.00B 5.00MiB --- --- data
[root@atest-guest linux_btrfs]# dd if=/dev/zero of=/mnt/sub1/data bs=1024
dd: error writing \u2018/mnt/sub1/data\u2019: Disk quota exceeded
5121+0 records in
5120+0 records out
5242880 bytes (5.2 MB) copied, 0.0218646 s, 240 MB/s
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# ll /mnt/sub1/data
-rw-r--r--. 1 root root 5242880 Feb 10 16:17 /mnt/sub1/data
[root@atest-guest linux_btrfs]# btrfs qgroup show -prce --raw /mnt
qgroupid rfer excl max_rfer max_excl parent child type
-- -
0/5 163841638400 --- ---
metadata,data
0/257 163841638400 --- ---
metadata,data
0/258 5242880 52428800 5242880 --- --- data
--- excl == max_excl == data_size == 5M == 5242880
Dongsheng Yang (7):
Btrfs: qgroup: Add type to btrfs_qgroup.
btrfs: qgroup: apply type to the recording and accounting.
btrfs: qgroup: Apply type to btrfs_qgroup_inherit().
btrfs: qgroup: Apply qgroup type to qgroup creating.
btrfs: qgroup: Apply qgroup type to subvol creating.
btrfs: qgroup: apply type to quota rescan.
btrfs: qgroup: fix a bug when type of parent is different with
child's.
fs/btrfs/ctree.h | 10 ++-
fs/btrfs/extent-tree.c| 48 ++-
fs/btrfs/ioctl.c | 49 ---
fs/btrfs/qgroup.c | 186 --
fs/btrfs/qgroup.h | 29 ++-
fs/btrfs/tests/qgroup-tests.c | 19 +++--
fs/btrfs/transaction.c| 13 +--
include/uapi/linux/btrfs.h| 1 +
8 files changed, 246 insertions(+), 109 deletions(-)
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs
[PATCH 2/4] btrfs-progs: specify qgroup type when creating subvolume
From: Fan Chengniang fancn.f...@cn.fujitsu.com
add --qgroup-type option to specify qgroup type. Type
can be metadata, data or all.
Signed-off-by: Fan Chengniang fancn.f...@cn.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
Documentation/btrfs-subvolume.txt | 4
cmds-subvolume.c | 50 +--
qgroup.c | 15
qgroup.h | 1 +
4 files changed, 63 insertions(+), 7 deletions(-)
diff --git a/Documentation/btrfs-subvolume.txt
b/Documentation/btrfs-subvolume.txt
index fe13943..4ac8cf5 100644
--- a/Documentation/btrfs-subvolume.txt
+++ b/Documentation/btrfs-subvolume.txt
@@ -60,6 +60,10 @@ set the newly created subvolume's max reference size
+
-e max_excl
set the newly created subvolume's max exclusive size
+--qgroup-type=type
+specify the type of qgroup to set qgroup limit.
++
+attr can be one of metadata, data, mixed.
*delete* [options] subvolume [subvolume...]::
Delete the subvolume(s) from the filesystem.
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index f0e22ac..871f749 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -51,6 +51,8 @@ static const char * const cmd_subvol_create_usage[] = {
option can be given multiple times.,
-r max_rfer set the newly created subvolume's max reference size,
-e max_excl set the newly created subvolume's max exclusive size,
+ --qgroup-type=metadata,data,mixed,
+ specify which qgroup type to set qgroup limit,
NULL
};
@@ -65,10 +67,19 @@ static int cmd_subvol_create(int argc, char **argv)
char*dst;
struct btrfs_qgroup_inherit *inherit = NULL;
DIR *dirstream = NULL;
+ __u8qgroup_type = 0;
+ int set_qgroup_type = 0;
optind = 1;
while (1) {
- int c = getopt(argc, argv, c:i:r:e:v);
+ int c;
+ static const struct option long_options[] = {
+ {qgroup-type, required_argument, NULL, 'q'},
+ {NULL, 0, NULL, 0}
+ };
+
+ c = getopt_long(argc, argv, c:i:r:e:v,
+ long_options, NULL);
if (c 0)
break;
@@ -101,6 +112,14 @@ static int cmd_subvol_create(int argc, char **argv)
goto out;
}
break;
+ case 'q':
+ set_qgroup_type = 1;
+ res = btrfs_qgroup_set_qgroup_type(qgroup_type,
optarg);
+ if (res) {
+ retval = res;
+ goto out;
+ }
+ break;
default:
usage(cmd_subvol_create_usage);
}
@@ -143,11 +162,21 @@ static int cmd_subvol_create(int argc, char **argv)
}
printf(Create subvolume '%s/%s'\n, dstdir, newname);
- if (inherit) {
- struct btrfs_ioctl_vol_args_v2 args;
- struct btrfs_ioctl_quota_ctl_args sa;
+ if (set_qgroup_type || inherit)
printf(Set qgroup arguments:\n);
+
+ if (set_qgroup_type) {
+ if (qgroup_type != 0) {
+ printf(\tqgroup type: );
+ if (qgroup_type BTRFS_QGROUP_TYPE_METADATA)
+ printf(metadata );
+ if (qgroup_type BTRFS_QGROUP_TYPE_DATA)
+ printf(data);
+ printf(\n);
+ }
+ }
+ if (inherit) {
if (inherit-num_qgroups 0) {
__u64 index;
__u64 qgroupid;
@@ -168,7 +197,11 @@ static int cmd_subvol_create(int argc, char **argv)
printf(\tmax exclusive: %llu\n,
inherit-lim.max_exclusive);
}
+ }
+ if (inherit || set_qgroup_type) {
+ struct btrfs_ioctl_vol_args_v2 args;
+ struct btrfs_ioctl_quota_ctl_args sa;
sa.cmd = BTRFS_QUOTA_CTL_STATUS;
res = ioctl(fddst, BTRFS_IOC_QUOTA_CTL, sa);
if (res 0) {
@@ -185,9 +218,12 @@ static int cmd_subvol_create(int argc, char **argv)
memset(args, 0, sizeof(args));
strncpy_null(args.name, newname);
- args.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
- args.size = qgroup_inherit_size(inherit);
- args.qgroup_inherit = inherit;
+ if (inherit != NULL) {
+ args.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
+ args.size = qgroup_inherit_size(inherit);
+ args.qgroup_inherit = inherit
[PATCH 6/7] btrfs: qgroup: apply type to quota rescan.
Although we have different type for each qgroup now,
but the rescan is also working as before which is considering
data and metadata both.
This patch fix this problem.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 19 ++-
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index ba1997f..d4ad565 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1759,7 +1759,8 @@ static int qgroup_calc_new_refcnt(struct btrfs_fs_info
*fs_info,
static int qgroup_adjust_counters(struct btrfs_fs_info *fs_info,
u64 root_to_skip, u64 num_bytes,
struct ulist *qgroups, u64 seq,
- int old_roots, int new_roots, int rescan)
+ int old_roots, int new_roots, int rescan,
+ u8 qgroup_type)
{
struct ulist_node *unode;
struct ulist_iterator uiter;
@@ -1771,6 +1772,9 @@ static int qgroup_adjust_counters(struct btrfs_fs_info
*fs_info,
bool dirty = false;
qg = u64_to_ptr(unode-aux);
+
+ if (!(qg-type qgroup_type))
+ continue;
/*
* Wasn't referenced before but is now, add to the reference
* counters.
@@ -2007,7 +2011,7 @@ static int qgroup_shared_accounting(struct
btrfs_trans_handle *trans,
* solution for this.
*/
qgroup_adjust_counters(fs_info, oper-ref_root, oper-num_bytes,
- qgroups, seq, old_roots, new_roots, 0);
+ qgroups, seq, old_roots, new_roots, 0,
oper-quota_type);
out:
spin_unlock(fs_info-qgroup_lock);
ulist_free(qgroups);
@@ -2670,14 +2674,19 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info,
struct btrfs_path *path,
mutex_unlock(fs_info-qgroup_rescan_lock);
for (; slot btrfs_header_nritems(scratch_leaf); ++slot) {
+ u8 type;
+
btrfs_item_key_to_cpu(scratch_leaf, found, slot);
if (found.type != BTRFS_EXTENT_ITEM_KEY
found.type != BTRFS_METADATA_ITEM_KEY)
continue;
- if (found.type == BTRFS_METADATA_ITEM_KEY)
+ if (found.type == BTRFS_METADATA_ITEM_KEY) {
num_bytes = fs_info-extent_root-nodesize;
- else
+ type = BTRFS_QGROUP_TYPE_METADATA;
+ } else {
num_bytes = found.offset;
+ type = BTRFS_QGROUP_TYPE_DATA;
+ }
ulist_reinit(qgroups);
ret = btrfs_find_all_roots(NULL, fs_info, found.objectid, 0,
@@ -2698,7 +2707,7 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct
btrfs_path *path,
}
ret = qgroup_adjust_counters(fs_info, 0, num_bytes, qgroups,
-seq, 0, new_roots, 1);
+seq, 0, new_roots, 1, type);
if (ret 0) {
spin_unlock(fs_info-qgroup_lock);
ulist_free(roots);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/7] Btrfs: qgroup: Add type to btrfs_qgroup.
This patch is introducing a *type* to btrfs_qgroup.
Type could be data, metadata or both. data means this
qgroup only care about the data size, metadata means
this qgroup only care about the metadata size and both
means this qgroup will care about data and metadata like
what it was doing before this patch.
This idea comes from some complain of user that
Why I can not write one byte in the subvolume when
the qgroup does not reach the limit?. That is caused
by the qgroup was accounting the metadata also, but it
is not confusing to a user.
So this patch here allow user to only limit the data size
in a qgroup.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ctree.h | 10 -
fs/btrfs/ioctl.c | 3 ++-
fs/btrfs/qgroup.c | 47 +++
fs/btrfs/qgroup.h | 12 ++-
fs/btrfs/tests/qgroup-tests.c | 4 ++--
5 files changed, 58 insertions(+), 18 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 1d17e7a..24b0d42 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -521,6 +521,11 @@ struct btrfs_super_block {
#define BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA (1ULL 8)
#define BTRFS_FEATURE_INCOMPAT_NO_HOLES(1ULL 9)
+/*
+ * Add a type in btrfs_qgroup_info_item;
+ */
+#define BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE (1ULL 10)
+
#define BTRFS_FEATURE_COMPAT_SUPP 0ULL
#define BTRFS_FEATURE_COMPAT_SAFE_SET 0ULL
#define BTRFS_FEATURE_COMPAT_SAFE_CLEAR0ULL
@@ -537,7 +542,8 @@ struct btrfs_super_block {
BTRFS_FEATURE_INCOMPAT_RAID56 |\
BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \
BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \
-BTRFS_FEATURE_INCOMPAT_NO_HOLES)
+BTRFS_FEATURE_INCOMPAT_NO_HOLES | \
+BTRFS_FEATURE_INCOMPAT_QGROUP_TYPE)
#define BTRFS_FEATURE_INCOMPAT_SAFE_SET\
(BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF)
@@ -1101,6 +1107,7 @@ struct btrfs_qgroup_info_item {
__le64 rfer_cmpr;
__le64 excl;
__le64 excl_cmpr;
+ __u8 type;
} __attribute__ ((__packed__));
/* flags definition for qgroup limits */
@@ -3214,6 +3221,7 @@ BTRFS_SETGET_FUNCS(qgroup_status_rescan, struct
btrfs_qgroup_status_item,
/* btrfs_qgroup_info_item */
BTRFS_SETGET_FUNCS(qgroup_info_generation, struct btrfs_qgroup_info_item,
generation, 64);
+BTRFS_SETGET_FUNCS(qgroup_info_type, struct btrfs_qgroup_info_item, type, 8);
BTRFS_SETGET_FUNCS(qgroup_info_rfer, struct btrfs_qgroup_info_item, rfer, 64);
BTRFS_SETGET_FUNCS(qgroup_info_rfer_cmpr, struct btrfs_qgroup_info_item,
rfer_cmpr, 64);
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index f6e730b..a50f295 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -4675,7 +4675,8 @@ static long btrfs_ioctl_qgroup_create(struct file *file,
void __user *arg)
/* FIXME: check if the IDs really exist */
if (sa-create) {
- ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid);
+ ret = btrfs_create_qgroup(trans, root-fs_info, sa-qgroupid,
+ BTRFS_QGROUP_TYPE_DEFAULT);
} else {
ret = btrfs_remove_qgroup(trans, root-fs_info, sa-qgroupid);
}
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 28b0aa5..8ac785a 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -51,6 +51,7 @@
*/
struct btrfs_qgroup {
u64 qgroupid;
+ u8 type;
/*
* state
@@ -128,7 +129,7 @@ static struct btrfs_qgroup *find_qgroup_rb(struct
btrfs_fs_info *fs_info,
/* must be called with qgroup_lock held */
static struct btrfs_qgroup *add_qgroup_rb(struct btrfs_fs_info *fs_info,
- u64 qgroupid)
+ u64 qgroupid, u8 type)
{
struct rb_node **p = fs_info-qgroup_tree.rb_node;
struct rb_node *parent = NULL;
@@ -151,6 +152,7 @@ static struct btrfs_qgroup *add_qgroup_rb(struct
btrfs_fs_info *fs_info,
return ERR_PTR(-ENOMEM);
qgroup-qgroupid = qgroupid;
+ qgroup-type = type;
INIT_LIST_HEAD(qgroup-groups);
INIT_LIST_HEAD(qgroup-members);
INIT_LIST_HEAD(qgroup-dirty);
@@ -348,7 +350,8 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)
flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
}
if (!qgroup) {
- qgroup = add_qgroup_rb(fs_info, found_key.offset);
+ qgroup = add_qgroup_rb(fs_info, found_key.offset,
+ BTRFS_QGROUP_TYPE_DEFAULT);
if (IS_ERR(qgroup)) {
ret = PTR_ERR(qgroup);
goto out;
@@ -360,6 +363,8 @@ int
Re: [PATCH 3/3] btrfs: qgroup: destroy related qgroup when removing subvolume if needed.
On 01/22/2015 02:16 PM, Wang Shilong wrote:
Hello,
When removing a subvol, if the related qgroup has no child qgroup, we should
destroy
it at the same time. Also remove the TODO entry in qgroup.c.
My two cents here:
Take a quick look at this, i am not sure whether this is right way to do it.
Actually, i think we can only remove a subvolume qgroup safely when both
refer and excl are 0,
because for subvolume removal case, to make qgroup accounting right, we need
walk tree.
At this time, qgroup walking might have not finished, if we remove this
qgroup directly,
accounting for this qgroup will be skipped.
Good point!
Maybe we could remove such qgroup when at qgroup accounting, we find an
qgroup’s refer and excl
are 0, we could remove it, or we do it at the background..
Also, there is another risk, that is to say, if qgroup accounting not work
right, So
we might need gurantee that subvolume it going to be deleted or have been
deleted at the same time,
then we could remove qgroup safely etc.
Yes, you are right. This is not a good timing to remove the qgroup directly.
I need some more investigation for a better solution.
As I have some emergency to deal with in this period, maybe the V2 will
come a little late.
Sorry about it.
Thanx
Yang
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/inode.c | 4
fs/btrfs/qgroup.c | 1 -
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index e687bb0..31de211 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -59,6 +59,7 @@
#include backref.h
#include hash.h
#include props.h
+#include qgroup.h
struct btrfs_iget_args {
struct btrfs_key *location;
@@ -4029,6 +4030,9 @@ int btrfs_unlink_subvol(struct btrfs_trans_handle
*trans,
ret = btrfs_update_inode_fallback(trans, root, dir);
if (ret)
btrfs_abort_transaction(trans, root, ret);
+ret = btrfs_remove_qgroup(trans, root-fs_info, objectid);
+if (ret == -EBUSY)
+ret = 0;
out:
btrfs_free_path(path);
return ret;
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index c3b1e4f..303c078 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -35,7 +35,6 @@
#include qgroup.h
/* TODO XXX FIXME
- * - subvol delete - delete when ref goes to 0? delete limits also?
* - reorganize keys
* - compressed
* - sync
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Best Regards,
Wang Shilong
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] btrfs: qgroup: destroy related qgroup when removing subvolume if needed.
When removing a subvol, if the related qgroup has no child qgroup, we should
destroy
it at the same time. Also remove the TODO entry in qgroup.c.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/inode.c | 4
fs/btrfs/qgroup.c | 1 -
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index e687bb0..31de211 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -59,6 +59,7 @@
#include backref.h
#include hash.h
#include props.h
+#include qgroup.h
struct btrfs_iget_args {
struct btrfs_key *location;
@@ -4029,6 +4030,9 @@ int btrfs_unlink_subvol(struct btrfs_trans_handle *trans,
ret = btrfs_update_inode_fallback(trans, root, dir);
if (ret)
btrfs_abort_transaction(trans, root, ret);
+ ret = btrfs_remove_qgroup(trans, root-fs_info, objectid);
+ if (ret == -EBUSY)
+ ret = 0;
out:
btrfs_free_path(path);
return ret;
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index c3b1e4f..303c078 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -35,7 +35,6 @@
#include qgroup.h
/* TODO XXX FIXME
- * - subvol delete - delete when ref goes to 0? delete limits also?
* - reorganize keys
* - compressed
* - sync
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/3] btrfs: qgroup: allow user to remove qgroup which has parent but no child.
When a qgroup has parents but no child, it should be removable in
Theory. But currently, we can not remove it when it has
either parent or child.
Example:
# btrfs quota enable /mnt
# btrfs qgroup create 1/0 /mnt
# btrfs qgroup create 2/0 /mnt
# btrfs qgroup assign 1/0 2/0 /mnt
# btrfs qgroup show -pcre /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 00--- ---
1/0 0 0 002/0 ---
2/0 0 0 00--- 1/0
At this time, there is no subvol or qgroup depending on qgroup 1/0.
Just a qgroup 2/0 is its parent, but 2/0 can work well without
1/0. So I think 1/0 should be removalbe. But:
# btrfs qgroup destroy 1/0 /mnt
ERROR: unable to destroy quota group: Device or resource busy
This patch remove the check of qgroup-parent in removing it.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/qgroup.c | 30 +-
1 file changed, 25 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index c2983a4..c3b1e4f 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1048,7 +1048,7 @@ out:
return ret;
}
-int btrfs_del_qgroup_relation(struct btrfs_trans_handle *trans,
+int __del_qgroup_relation(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 src, u64 dst)
{
struct btrfs_root *quota_root;
@@ -1058,7 +1058,6 @@ int btrfs_del_qgroup_relation(struct btrfs_trans_handle
*trans,
int ret = 0;
int err;
- mutex_lock(fs_info-qgroup_ioctl_lock);
quota_root = fs_info-quota_root;
if (!quota_root) {
ret = -EINVAL;
@@ -1089,7 +1088,18 @@ exist:
del_relation_rb(fs_info, src, dst);
spin_unlock(fs_info-qgroup_lock);
out:
+ return ret;
+}
+
+int btrfs_del_qgroup_relation(struct btrfs_trans_handle *trans,
+ struct btrfs_fs_info *fs_info, u64 src, u64 dst)
+{
+ int ret = 0;
+
+ mutex_lock(fs_info-qgroup_ioctl_lock);
+ ret = __del_qgroup_relation(trans, fs_info, src, dst);
mutex_unlock(fs_info-qgroup_ioctl_lock);
+
return ret;
}
@@ -1132,6 +1142,7 @@ int btrfs_remove_qgroup(struct btrfs_trans_handle *trans,
{
struct btrfs_root *quota_root;
struct btrfs_qgroup *qgroup;
+ struct btrfs_qgroup_list *list;
int ret = 0;
mutex_lock(fs_info-qgroup_ioctl_lock);
@@ -1146,15 +1157,24 @@ int btrfs_remove_qgroup(struct btrfs_trans_handle
*trans,
ret = -ENOENT;
goto out;
} else {
- /* check if there are no relations to this qgroup */
- if (!list_empty(qgroup-groups) ||
- !list_empty(qgroup-members)) {
+ /* check if there are no children of this qgroup */
+ if (!list_empty(qgroup-members)) {
ret = -EBUSY;
goto out;
}
}
ret = del_qgroup_item(trans, quota_root, qgroupid);
+ while (!list_empty(qgroup-groups)) {
+ list = list_first_entry(qgroup-groups,
+ struct btrfs_qgroup_list, next_group);
+ ret = __del_qgroup_relation(trans, fs_info,
+ qgroupid,
+ list-group-qgroupid);
+ if (ret)
+ goto out;
+ }
+
spin_lock(fs_info-qgroup_lock);
del_qgroup_rb(quota_root-fs_info, qgroupid);
spin_unlock(fs_info-qgroup_lock);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: get the accurate value of used_bytes in btrfs_get_block_group_info().
On 01/07/2015 05:22 PM, Qu Wenruo wrote:
Hi Satoru-san
Hi Dongsheng,
On 2015/01/05 20:19, Dongsheng Yang wrote:
Ping.
IOCTL of BTRFS_IOC_SPACE_INFO currently does not report
the data used but not synced to user. Then btrfs fi df will
give user a wrong numbers before sync. This patch solve
this problem.
On 10/27/2014 08:38 PM, Dongsheng Yang wrote:
Reproducer:
# mkfs.btrfs -f -b 20G /dev/sdb
# mount /dev/sdb /mnt/test
# fallocate -l 17G /mnt/test/largefile
# btrfs fi df /mnt/test
Data, single: total=17.49GiB, used=6.00GiB - only 6G, but
actually it should be 17G.
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
I tried to reproduce your problem with 3.19-rc1.
However, this problem doesn't happen. Could you
also try to reproduce with the upstream kernel?
I can still reproduce it in 3.18, but it seems to be fixed in 3.19-rc1
already by other patch,
so this patch is no longer needed.
Oops, my fault. I forgot to test it with upstream. :(
Satoru and Qu, thanx a lot.
Yang
Thanks,
Qu
* Detail
test script (named yang-test.sh here):
===
#!/bin/bash -x
PART1=/dev/vdb
MNT_PNT=./mnt
mkfs.btrfs -f -b 20G ${PART1}
mount ${PART1} ${MNT_PNT}
fallocate -l 17G ${MNT_PNT}/largefile
btrfs fi df ${MNT_PNT}
sync
btrfs fi df ${MNT_PNT}
umount ${MNT_PNT}
===
Result:
===
# ./yang-test.sh
+ PART1=/dev/vdb
+ MNT_PNT=./mnt
+ mkfs.btrfs -f -b 20G /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 20.00GiB
+ mount /dev/vdb ./mnt
+ fallocate -l 17G ./mnt/largefile
+ btrfs fi df ./mnt
Data, single: total=17.01GiB, used=17.00GiB # Used 17GiB properly
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
+ sync
+ btrfs fi df ./mnt
Data, single: total=17.01GiB, used=17.00GiB# (of course) used
17GiB too
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
+ umount ./mnt
===
Although I ran this test five times, the results are the same.
Thanks,
Satoru
# sync
# btrfs fi df /mnt/test
Data, single: total=17.49GiB, used=17.00GiB - After sync, it
is expected.
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
The value of 6.00GiB is actually calculated in
btrfs_get_block_group_info()
by adding the @block_group-item-used for each group together. In
this way,
it did not consider the bytes in cache.
This patch adds the value of @pinned, @reserved and @bytes_super in
struct btrfs_block_group_cache to make sure we can get the accurate
@used_bytes.
Reported-by: Qu Wenruo quwen...@cn.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/ioctl.c | 4
1 file changed, 4 insertions(+)
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 33c80f5..bc2aaeb 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -3892,6 +3892,10 @@ void btrfs_get_block_group_info(struct
list_head *groups_list,
space-total_bytes += block_group-key.offset;
space-used_bytes +=
btrfs_block_group_used(block_group-item);
+/* Add bytes-info in cache */
+space-used_bytes += block_group-pinned;
+space-used_bytes += block_group-reserved;
+space-used_bytes += block_group-bytes_super;
}
}
--
To unsubscribe from this list: send the line unsubscribe
linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe
linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 0/3] Btrfs: Enhancment for qgroup.
On 01/07/2015 08:49 AM, Satoru Takeuchi wrote: Hi Yang, On 2015/01/05 15:16, Dongsheng Yang wrote: Hi Josef and others, This patch set is about enhancing qgroup. [1/3]: fix a bug about qgroup leak when we exceed quota limit, It is reviewd by Josef. [2/3]: introduce a new accounter in qgroup to close a window where user will exceed the limit by qgroup. It looks good to Josef. [3/3]: a new patch to fix a bug reported by Satoru. I tested your the patchset v3. Although it's far better than the patchset v2, there is still one problem in this patchset. When I wrote 1.5GiB to a subvolume with 1.0 GiB limit, 1.0GiB - 139 block (in this case, 1KiB/block) was written. I consider user should be able to write just 1.0GiB in this case. Hi Satoru, Yes, Currently, user can not write 1.0GiB in this case. Because qgroup is accounting data and metadata togather. And I have posted an idea in this thread that split it into three modes, data, metadata and both. TODO issues: c). limit and account size in 3 modes, data, metadata and both. qgroup is accounting the size both of data and metadata togather, but to users, the data size is the most useful to them. But, you mentioned that the result is different in each time. Hmmm there must be something wrong in it. I need some more investigation to answer this question. Thanx a lot for your test! Yang * Test result === + mkfs.btrfs -f /dev/vdb Btrfs v3.17 See http://btrfs.wiki.kernel.org for more information. Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 fs created label (null) on /dev/vdb nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB + mount /dev/vdb /root/btrfs-auto-test/ + ret=0 + btrfs quota enable /root/btrfs-auto-test/ + btrfs subvolume create /root/btrfs-auto-test//sub Create subvolume '/root/btrfs-auto-test/sub' + btrfs qgroup limit 1G /root/btrfs-auto-test//sub + dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024 count=150 dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded 1048438+0 records in# Tried to write 1GiB - 138 KiB 1048437+0 records out # Succeeded to write 1GiB - 139 KiB 1073599488 bytes (1.1 GB) copied, 19.0247 s, 56.4 MB/s === * note I tried to run the reproducer five times and the result is a bit different for each time. = # Written - 1 1GiB - 139 KiB 2 1GiB - 139 KiB 3 1GiB - 145 KiB 4 1GiB - 135 KiB 5 1GiB - 135 KiB == So I consider it's a problem comes from timing. If I changed the block size from 1KiB to 1 MiB, the difference in bytes got larger. # Written 1 1GiB - 1 MiB 2 1GiB - 1 MiB 3 1GiB - 1 MiB 4 1GiB - 1 MiB 5 1GiB - 1 MiB Thanks, Satoru BTW, I have some other plan about qgroup in my TODO list: Kernel: a). adjust the accounters in parent qgroup when we move the child qgroup. Currently, when we move a qgroup, the parent qgroup will not updated at the same time. This will cause some wrong numbers in qgroup. b). add a ioctl to show the qgroup info. Command btrfs qgroup show is showing the qgroup info read from qgroup tree. But there is some information in memory which is not synced into device. Then it will show some outdate number. c). limit and account size in 3 modes, data, metadata and both. qgroup is accounting the size both of data and metadata togather, but to a user, the data size is the most useful to them. d). remove a subvolume related qgroup when subvolume is deleted and there is no other reference to it. user-tool: a). Add the unit of B/K/M/G to btrfs qgroup show. b). get the information via ioctl rather than reading it from btree. Will keep the old way as a fallback for compatiblity. Any comment and sugguestion is welcome. :) Yang Dongsheng Yang (3): Btrfs: qgroup: free reserved in exceeding quota. Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use. Btrfs: qgroup, Account data space in more proper timings. fs/btrfs/extent-tree.c | 41 +++--- fs/btrfs/file.c| 9 --- fs/btrfs/inode.c | 18 - fs/btrfs/qgroup.c | 68 +++--- fs/btrfs/qgroup.h | 4 +++ 5 files changed, 117 insertions(+), 23 deletions(-) . -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: qgroup: move WARN_ON() to the correct location.
In function qgroup_excl_accounting(), we need to WARN when qg-excl is less than what we want to free, same to child and parents. But currently, for parent qgroup, the WARN_ON() is located after freeing qg-excl. It will WARN out even we free it normally. This patch move this WARN_ON() before freeing qg-excl. Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/qgroup.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 48b60db..97159a8 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -1431,9 +1431,8 @@ static int qgroup_excl_accounting(struct btrfs_fs_info *fs_info, qgroup = u64_to_ptr(unode-aux); qgroup-rfer += sign * oper-num_bytes; qgroup-rfer_cmpr += sign * oper-num_bytes; + WARN_ON(sign 0 qgroup-excl oper-num_bytes); qgroup-excl += sign * oper-num_bytes; - if (sign 0) - WARN_ON(qgroup-excl oper-num_bytes); qgroup-excl_cmpr += sign * oper-num_bytes; qgroup_dirty(fs_info, qgroup); -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On 12/31/2014 08:15 AM, Zygo Blaxell wrote: On Wed, Dec 17, 2014 at 08:07:27PM -0800, Robert White wrote: [...] There are a number of pathological examples in here, but I think there are justifiable correct answers for each of them that emerge from a single interpretation of the meanings of f_bavail, f_blocks, and f_bfree. One gotcha is that some of the numbers required may be difficult to calculate precisely before all space is allocated to chunks; however, some error is tolerable as long as free space is not overestimated. In other words: when in doubt, guess low. statvfs(2) gives us six numbers, three of which are block counts. Very few users or programs ever bother to look at the inode counts (f_files, f_ffree, f_favail), but they could be overloaded for metadata block counts. The f_blocks parameter is mostly irrelevant to application behavior, except to the extent that the ratio between f_bavail and f_blocks is used by applications to calculate a percentage of occupied or free space. f_blocks must always be greater than or equal to f_bavail and f_blocks, and preferably f_blocks would be scaled to use the same effective unit size as f_bavail and f_blocks within a percent or two. Nobody cares about f_bfree since traditionally only root could use the difference between f_bfree and f_bavail. f_bfree is effectively space conditionally available (e.g. if the process euid is root or the process egid matches a configured group id), while f_bavail is space available without conditions (e.g. processes without privilege can use it). The most important number is f_bavail. It's what a bunch of software (archive unpackers, assorted garbage collectors, email MTAs, snapshot remover scripts, download managers, etc) uses to estimate how much space is available without conditions (except quotas, although arguably those should be included too). Applications that are privileged still use the unprivileged f_bavail number so their decisions based on free space don't disrupt unprivileged applications. It's generally better to underestimate than to overestimate f_bavail. Historically filesystems have reserved extra space to avoid various problems in low-disk conditions, and application software has adapted to that well over the years. Also, admin people are more pleasantly surprised when it turns out that they had more space than f_bavail, instead of when they had less. The rule should be: if we have some space, but it is not available for data extents in the current allocation mode, don't add it to f_bavail in statvfs. I think this rule handles all of these examples well. That would mean that we get cases where we add a drive to a full filesystem and it doesn't immediately give you any new f_bavail space. That may be an unexpected result for a naive admin, but much less unexpected than having all the new space show up in f_bavail when it is not available for allocation in the current data profile! Better to have the surprising behavior earlier than later. On to examples... But a more even case is downright common and likely. Say you run a nice old-fashoned MUTT mail-spool. most of your files are small enough to live in metadata. You start with one drive. and allocate 2 single-data and 10 metatata (5xDup). Then you add a second drive of equal size. (the metadata just switched to DUP-as-RAID1-alike mode) And then you do a dconvert=raid0. That uneven allocation of metadata will be a 2GiB difference between the two drives forever. So do you shave 2GiB off of your @size? Yes. f_blocks is the total size of all allocated chunks plus all free space allocated by the current data profile. Agreed. This is what my patch designed by. That 2GiB should disappear from such a calculation. Do you shave @2GiB off your @available? Yes, because it's _not_ available until something changes to make it available (e.g. balance to get rid of the dup metadata, change the metadata profile to dup or single, or change the data profile to single). The 2GiB could be added to f_bfree, but that might still be confusing for people and software. Do you overreport your available by @2GiB and end up _still_ having things available when you get your ENOSPC? No. ENOSPC when f_bavail 0 is very bad. Yes, it is very bad. Low-available-space admin alerts will not be triggered. Automated mitigation software will not be activated. Service daemons will start transactions they cannot complete. How about this :: /dev/sda == |Sf|Sf|Mf|Mf|Mf|Mf|Sf|Sf|Sp|Mp|Mp| .5GiB free| /dev/sdb == |10 GiB free | Operator fills his drive, then adds a second one, then _foolishly_ tries to convert it to RAID0 when the power fails. In order to check the FS he boots with no_balance. Then his maintenance window closes and he has to go back into production, at which point he forgets (or isn't allowed) to do the balance. The flags are set but now no more extents can be allocated. Size is 20GiB, slack is 10.5GiB.
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On 12/27/2014 09:10 AM, Robert White wrote: On 12/23/2014 04:31 AM, Dongsheng Yang wrote: On 12/18/2014 12:07 PM, Robert White wrote: ... Thanx Yang . xaE Fine. but you still haven't told me/us what you thing df (really fsstat() ) should report in those cases. Go back to that email. Grab some of those cases. Then tell me what your patch will report and why it makes more sense than what is reported now. For code to be correct it _must_ be correct for the special cases. You don't get to just ignore the special cases. So what are your specific answers to those cases? Great to hear that we can stop the endless discussion of FS level VS device level. Then we can come back to discuss the special cases you provided. I give you some answer in another mail. And make a change in my patch to cover the case of un-finished balance. Thanx . -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 1/3] Btrfs: get more accurate output in df command.
When function btrfs_statfs() calculate the tatol size of fs, it is calculating
the total size of disks and then dividing it by a factor. But in some usecase,
the result is not good to user.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 3.0G 1018M 1.3G 45% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a. df -h should report Size as 2GiB rather than as 3GiB.
Because this is 2 device raid1, the limiting factor is devid 1 @2GiB.
b. df -h should report Avail as 0.97GiB or less, rather than as 1.3GiB.
1.85 (the capacity of the allocated chunk)
-1.018 (the file stored)
+(2-1.85=0.15) (the residual capacity of the disks
considering a raid1 fs)
---
= 0.97
This patch drops the factor at all and calculate the size observable to
user without considering which raid level the data is in and what's the
size exactly in disk.
After this patch applied:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 2.0G 1.3G 713M 66% /mnt
# df /mnt
Filesystem 1K-blocksUsed Available Use% Mounted on
/dev/vdf12097152 1359424729536 66% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: e98c1321-645f-4457-b20d-4f41dc1cf2f4
Total devices 2 FS bytes used 1001.55MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a). The @Size is 2G as we expected.
b). @Available is 700M = 1.85G - 1.3G + (2G - 1.85G).
c). @Used is changed to 1.3G rather than 1018M as above. Because
this patch do not treat the free space in metadata chunk
and system chunk as available to user. It's true, user can
not use these space to store data, then it should not be
thought as available. At the same time, it will make the
@Used + @Available == @Size as possible to user.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
Changelog:
- account free space a block group by a block group,
treat the block group in unexpected raid level as used.
fs/btrfs/ctree.h | 1 -
fs/btrfs/extent-tree.c | 41
fs/btrfs/super.c | 74 --
3 files changed, 42 insertions(+), 74 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 7e60741..0b9d5c0 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3478,7 +3478,6 @@ int btrfs_set_block_group_ro(struct btrfs_root *root,
void btrfs_set_block_group_rw(struct btrfs_root *root,
struct btrfs_block_group_cache *cache);
void btrfs_put_block_group_cache(struct btrfs_fs_info *info);
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo);
int btrfs_error_unpin_extent_range(struct btrfs_root *root,
u64 start, u64 end);
int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr,
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index a80b971..f9bf8ac 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -8578,47 +8578,6 @@ int btrfs_force_chunk_alloc(struct btrfs_trans_handle
*trans,
CHUNK_ALLOC_FORCE);
}
-/*
- * helper to account the unused space of all the readonly block group in the
- * space_info. takes mirrors into account.
- */
-u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo)
-{
- struct btrfs_block_group_cache *block_group;
- u64 free_bytes = 0;
- int factor;
-
- /* It's df, we don't care if it's racey */
- if (list_empty(sinfo-ro_bgs))
- return 0;
-
- spin_lock(sinfo-lock);
- list_for_each_entry(block_group, sinfo-ro_bgs, ro_list) {
- spin_lock(block_group-lock);
-
- if (!block_group-ro) {
- spin_unlock(block_group-lock);
- continue;
- }
-
- if (block_group-flags (BTRFS_BLOCK_GROUP_RAID1 |
- BTRFS_BLOCK_GROUP_RAID10 |
- BTRFS_BLOCK_GROUP_DUP))
- factor = 2;
- else
- factor = 1;
-
- free_bytes += (block_group-key.offset -
- btrfs_block_group_used(block_group-item)) *
- factor;
-
- spin_unlock
[PATCH v2 2/3] Btrfs: raid56: simplify the parameter of nr_parity_stripes().
We just need the type of a chunk to calculate the number of parity stripes,
but we have to pass a structure of lookup_map to it. This will prevent some
callers to use it where there is no a convenient lookup_map to be passed.
This patch replace the parameter of struct map_lookup * with a profile type.
Then we can use it more easily.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
Reviewed-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com
---
fs/btrfs/raid56.h | 8
fs/btrfs/volumes.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
index 31d4a15..d2f5690 100644
--- a/fs/btrfs/raid56.h
+++ b/fs/btrfs/raid56.h
@@ -19,11 +19,11 @@
#ifndef __BTRFS_RAID56__
#define __BTRFS_RAID56__
-static inline int nr_parity_stripes(struct map_lookup *map)
+static inline int nr_parity_stripes(u64 type)
{
- if (map-type BTRFS_BLOCK_GROUP_RAID5)
+ if (type BTRFS_BLOCK_GROUP_RAID5)
return 1;
- else if (map-type BTRFS_BLOCK_GROUP_RAID6)
+ else if (type BTRFS_BLOCK_GROUP_RAID6)
return 2;
else
return 0;
@@ -31,7 +31,7 @@ static inline int nr_parity_stripes(struct map_lookup *map)
static inline int nr_data_stripes(struct map_lookup *map)
{
- return map-num_stripes - nr_parity_stripes(map);
+ return map-num_stripes - nr_parity_stripes(map-type);
}
#define RAID5_P_STRIPE ((u64)-2)
#define RAID6_Q_STRIPE ((u64)-1)
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 50c5a87..fedf76e 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -5181,7 +5181,7 @@ static int __btrfs_map_block(struct btrfs_fs_info
*fs_info, int rw,
/* RAID[56] write or recovery. Return all stripes */
num_stripes = map-num_stripes;
- max_errors = nr_parity_stripes(map);
+ max_errors = nr_parity_stripes(map-type);
raid_map = kmalloc_array(num_stripes, sizeof(u64),
GFP_NOFS);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: get the accurate value of used_bytes in btrfs_get_block_group_info().
Ping. IOCTL of BTRFS_IOC_SPACE_INFO currently does not report the data used but not synced to user. Then btrfs fi df will give user a wrong numbers before sync. This patch solve this problem. On 10/27/2014 08:38 PM, Dongsheng Yang wrote: Reproducer: # mkfs.btrfs -f -b 20G /dev/sdb # mount /dev/sdb /mnt/test # fallocate -l 17G /mnt/test/largefile # btrfs fi df /mnt/test Data, single: total=17.49GiB, used=6.00GiB - only 6G, but actually it should be 17G. System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B # sync # btrfs fi df /mnt/test Data, single: total=17.49GiB, used=17.00GiB - After sync, it is expected. System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B The value of 6.00GiB is actually calculated in btrfs_get_block_group_info() by adding the @block_group-item-used for each group together. In this way, it did not consider the bytes in cache. This patch adds the value of @pinned, @reserved and @bytes_super in struct btrfs_block_group_cache to make sure we can get the accurate @used_bytes. Reported-by: Qu Wenruo quwen...@cn.fujitsu.com Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com --- fs/btrfs/ioctl.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 33c80f5..bc2aaeb 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -3892,6 +3892,10 @@ void btrfs_get_block_group_info(struct list_head *groups_list, space-total_bytes += block_group-key.offset; space-used_bytes += btrfs_block_group_used(block_group-item); + /* Add bytes-info in cache */ + space-used_bytes += block_group-pinned; + space-used_bytes += block_group-reserved; + space-used_bytes += block_group-bytes_super; } } -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Fixing quota error when removing files from a limit exceeded subvols
On Sat, Jan 3, 2015 at 10:29 PM, Khaled Ahmed khaled@gmail.com wrote:
Hi Yang,
This is how to reproduce the bug,
[root@algodev ~]# uname -r
3.18.0+
[root@algodev ~]# btrfs version
Btrfs v3.18-2-g6938452-dirty
[root@algodev ~]# btrfs quota enable LOOP/
[root@algodev ~]# btrfs qgroup show LOOP/
qgroupid rfer excl
0/5 16384 16384
[root@algodev ~]# btrfs subvol create LOOP/subvol1
Create subvolume 'LOOP/subvol1'
[root@algodev ~]# btrfs qgroup limit 1g LOOP/subvol1/
[root@algodev ~]# btrfs qgroup show LOOP/
qgroupid rfer excl
0/5 16384 16384
0/25816384 16384
[root@algodev ~]# dd if=/dev/zero of=LOOP/subvol1/bigfile
dd: writing to ‘LOOP/subvol1/bigfile’: Disk quota exceeded
2097018+0 records in
2097017+0 records out
1073672704 bytes (1.1 GB) copied, 10.0759 s, 107 MB/s
[root@algodev ~]# rm -f LOOP/subvol1/bigfile
rm: cannot remove ‘LOOP/subvol1/bigfile’: Disk quota exceeded
Hi Ahmed,
Okey, thanx for your example.
a). I guess your problem is getting a EQUOTA when remove a file here.
It's because we need to reserve some metadata in transaction of btrfs_unlink().
b). I think you patch here will not solve your problem. The root cause is
current quota in btrfs is accounting data and metadata together.
c). I admit getting a EQUOTA is strange when you did not writing anything but
only remove a file. I had a plan in my TODO list which is making qgroup to
limit and account the size in three modes, data, metadata and both.
Then in this case
if you only limit the size of data, you will not get a EQUOTA any more.
Thanx
Yang
[root@algodev ~]#
Best Regards,
~Khaled Ahmed
On Jan 3, 2015, at 4:09 AM, Dongsheng Yang dongsheng081...@gmail.com wrote:
Hi Khaled,
Could you give use more description about the problem this patch
is trying to solve? Maybe an example will help a lot to understand it.
Thanx
On Fri, Jan 2, 2015 at 7:48 AM, Khaled Ahmed khaled@gmail.com wrote:
Signed-off-by: Khaled Ahmed khaled@gmail.com
---
fs/btrfs/qgroup.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 48b60db..b85200d 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2408,14 +2408,14 @@ int btrfs_qgroup_reserve(struct btrfs_root *root,
u64 num_bytes)
if ((qg-lim_flags BTRFS_QGROUP_LIMIT_MAX_RFER)
qg-reserved + (s64)qg-rfer + num_bytes
- qg-max_rfer) {
+ qg-max_rfer - 1 ) {
ret = -EDQUOT;
goto out;
}
if ((qg-lim_flags BTRFS_QGROUP_LIMIT_MAX_EXCL)
qg-reserved + (s64)qg-excl + num_bytes
- qg-max_excl) {
+ qg-max_excl - 1) {
ret = -EDQUOT;
goto out;
}
--
2.1.0
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/3] Btrfs: Enhancment for qgroup.
Hi Josef and others, This patch set is about enhancing qgroup. [1/3]: fix a bug about qgroup leak when we exceed quota limit, It is reviewd by Josef. [2/3]: introduce a new accounter in qgroup to close a window where user will exceed the limit by qgroup. It looks good to Josef. [3/3]: a new patch to fix a bug reported by Satoru. BTW, I have some other plan about qgroup in my TODO list: Kernel: a). adjust the accounters in parent qgroup when we move the child qgroup. Currently, when we move a qgroup, the parent qgroup will not updated at the same time. This will cause some wrong numbers in qgroup. b). add a ioctl to show the qgroup info. Command btrfs qgroup show is showing the qgroup info read from qgroup tree. But there is some information in memory which is not synced into device. Then it will show some outdate number. c). limit and account size in 3 modes, data, metadata and both. qgroup is accounting the size both of data and metadata togather, but to a user, the data size is the most useful to them. d). remove a subvolume related qgroup when subvolume is deleted and there is no other reference to it. user-tool: a). Add the unit of B/K/M/G to btrfs qgroup show. b). get the information via ioctl rather than reading it from btree. Will keep the old way as a fallback for compatiblity. Any comment and sugguestion is welcome. :) Yang Dongsheng Yang (3): Btrfs: qgroup: free reserved in exceeding quota. Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use. Btrfs: qgroup, Account data space in more proper timings. fs/btrfs/extent-tree.c | 41 +++--- fs/btrfs/file.c| 9 --- fs/btrfs/inode.c | 18 - fs/btrfs/qgroup.c | 68 +++--- fs/btrfs/qgroup.h | 4 +++ 5 files changed, 117 insertions(+), 23 deletions(-) -- 1.8.4.2 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 3/3] Btrfs: qgroup, Account data space in more proper timings.
Currenly, in data writing, -reserved is accounted in
fill_delalloc(), but -may_use is released in clear_bit_hook()
which is called by btrfs_finish_ordered_io(). That's too late,
that said, between fill_delalloc() and btrfs_finish_ordered_io(),
the data is doublely accounted by qgroup. It will cause some
unexpected -EDQUOT.
Example:
# btrfs quota enable /root/btrfs-auto-test/
# btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
# btrfs qgroup limit 1G /root/btrfs-auto-test//sub
dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024 count=150
dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded
681353+0 records in
681352+0 records out
697704448 bytes (698 MB) copied, 8.15563 s, 85.5 MB/s
It's (698 MB) when we got an -EDQUOT, but we limit it by 1G.
This patch move the btrfs_qgroup_reserve/free() for data from
btrfs_delalloc_reserve/release_metadata() to btrfs_check_data_free_space()
and btrfs_free_reserved_data_space(). Then the accounter in qgroup
will be updated at the same time with the accounter in space_info updated.
In this way, the unexpected -EDQUOT will be killed.
Reported-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
fs/btrfs/extent-tree.c | 16 +---
fs/btrfs/file.c| 9 -
2 files changed, 9 insertions(+), 16 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index d1a7ce0..67c2e28 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3774,12 +3774,16 @@ commit_trans:
data_sinfo-flags, bytes, 1);
return -ENOSPC;
}
+ ret = btrfs_qgroup_reserve(root, bytes);
+ if (ret)
+ goto out;
data_sinfo-bytes_may_use += bytes;
trace_btrfs_space_reservation(root-fs_info, space_info,
data_sinfo-flags, bytes, 1);
+out:
spin_unlock(data_sinfo-lock);
- return 0;
+ return ret;
}
/*
@@ -3796,6 +3800,7 @@ void btrfs_free_reserved_data_space(struct inode *inode,
u64 bytes)
data_sinfo = root-fs_info-data_sinfo;
spin_lock(data_sinfo-lock);
WARN_ON(data_sinfo-bytes_may_use bytes);
+ btrfs_qgroup_free(root, bytes);
data_sinfo-bytes_may_use -= bytes;
trace_btrfs_space_reservation(root-fs_info, space_info,
data_sinfo-flags, bytes, 0);
@@ -5191,8 +5196,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode,
u64 num_bytes)
spin_unlock(BTRFS_I(inode)-lock);
if (root-fs_info-quota_enabled) {
- ret = btrfs_qgroup_reserve(root, num_bytes +
- nr_extents * root-nodesize);
+ ret = btrfs_qgroup_reserve(root, nr_extents * root-nodesize);
if (ret)
goto out_fail;
}
@@ -5200,8 +5204,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode,
u64 num_bytes)
ret = reserve_metadata_bytes(root, block_rsv, to_reserve, flush);
if (unlikely(ret)) {
if (root-fs_info-quota_enabled)
- btrfs_qgroup_free(root, num_bytes +
- nr_extents * root-nodesize);
+ btrfs_qgroup_free(root, nr_extents * root-nodesize);
goto out_fail;
}
@@ -5319,8 +5322,7 @@ void btrfs_delalloc_release_metadata(struct inode *inode,
u64 num_bytes)
trace_btrfs_space_reservation(root-fs_info, delalloc,
btrfs_ino(inode), to_free, 0);
if (root-fs_info-quota_enabled) {
- btrfs_qgroup_free(root, num_bytes +
- dropped * root-nodesize);
+ btrfs_qgroup_free(root, dropped * root-nodesize);
}
btrfs_block_rsv_release(root, root-fs_info-delalloc_block_rsv,
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index e409025..0ab1333 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2527,7 +2527,6 @@ static long btrfs_fallocate(struct file *file, int mode,
{
struct inode *inode = file_inode(file);
struct extent_state *cached_state = NULL;
- struct btrfs_root *root = BTRFS_I(inode)-root;
u64 cur_offset;
u64 last_byte;
u64 alloc_start;
@@ -2555,11 +2554,6 @@ static long btrfs_fallocate(struct file *file, int mode,
ret = btrfs_check_data_free_space(inode, alloc_end - alloc_start);
if (ret)
return ret;
- if (root-fs_info-quota_enabled) {
- ret = btrfs_qgroup_reserve(root, alloc_end - alloc_start);
- if (ret)
- goto out_reserve_fail;
- }
mutex_lock(inode-i_mutex);
ret = inode_newsize_ok(inode
[PATCH v3 1/3] Btrfs: qgroup: free reserved in exceeding quota.
When we exceed quota limit in writing, we will free
some reserved extent when we need to drop but not free
account in qgroup. It means, each time we exceed quota
in writing, there will be some remain space in qg-reserved
we can not use any more. If things go on like this, the
all space will be ate up.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
Reviewed-by: Josef Bacik jba...@fb.com
---
fs/btrfs/extent-tree.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index a80b971..88b4e32 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5275,8 +5275,11 @@ out_fail:
to_free = 0;
}
spin_unlock(BTRFS_I(inode)-lock);
- if (dropped)
+ if (dropped) {
+ if (root-fs_info-quota_enabled)
+ btrfs_qgroup_free(root, dropped * root-nodesize);
to_free += btrfs_calc_trans_metadata_size(root, dropped);
+ }
if (to_free) {
btrfs_block_rsv_release(root, block_rsv, to_free);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Fixing quota error when removing files from a limit exceeded subvols
Hi Khaled,
Could you give use more description about the problem this patch
is trying to solve? Maybe an example will help a lot to understand it.
Thanx
On Fri, Jan 2, 2015 at 7:48 AM, Khaled Ahmed khaled@gmail.com wrote:
Signed-off-by: Khaled Ahmed khaled@gmail.com
---
fs/btrfs/qgroup.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 48b60db..b85200d 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2408,14 +2408,14 @@ int btrfs_qgroup_reserve(struct btrfs_root *root, u64
num_bytes)
if ((qg-lim_flags BTRFS_QGROUP_LIMIT_MAX_RFER)
qg-reserved + (s64)qg-rfer + num_bytes
- qg-max_rfer) {
+ qg-max_rfer - 1 ) {
ret = -EDQUOT;
goto out;
}
if ((qg-lim_flags BTRFS_QGROUP_LIMIT_MAX_EXCL)
qg-reserved + (s64)qg-excl + num_bytes
- qg-max_excl) {
+ qg-max_excl - 1) {
ret = -EDQUOT;
goto out;
}
--
2.1.0
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 2/2] Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use.
On Fri, Dec 26, 2014 at 1:43 PM, Satoru Takeuchi
takeuchi_sat...@jp.fujitsu.com wrote:
Hi Yang,
On 2014/12/26 10:32, Dongsheng Yang wrote:
Hi Satoru,
I saw your mail of [BUG] Quota Ignored On write problem still exist
with 3.16-rc5
http://news.gmane.org/find-root.php?message_id=53C8DEB0.1060404%40jp.fujitsu.com
in gmane.
I guess this patch will fix the problem you mentioned.
Could you help to give a try?
Although it reached disk quota before writing whole 1.5GB,
the copied size was only about 700 MB. In this case,
expected behavior is to reach disk quota just after
writing 1 GB.
Hi Satoru-san,
After some more investigation, I did meet this problem you described here.
There is a window between reserved += num_bytes and may_use -= num_bytes.
And the similar problem exists in real space accounting. I am cooking a patch
to solve this problem.
Thanx for your test a lot.
Yang
* 3.19-rc1 without your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file to
65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024 count=150
150+0 records in
150+0 records out
153600 bytes (1.5 GB) copied, 25.6601 s, 59.9 MB/s
===
* 3.19-rc1 with your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file to
65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024 count=150
dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded
681353+0 records in
681352+0 records out
697704448 bytes (698 MB) copied, 8.15563 s, 85.5 MB/s
===
Thanks,
Satoru
Thanx
Yang
On 12/12/2014 04:44 PM, Dongsheng Yang wrote:
Currently, for pre_alloc or delay_alloc, the bytes will be accounted
in space_info by the three guys.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
But on the other hand, in qgroup, there are only two counters to account
the
bytes, qgroup-reserved and qgroup-excl. And qg-reserved accounts
bytes in space_info-bytes_may_use and qg-excl accounts bytes in
space_info-used. So the bytes in space_info-reserved is not accounted
in qgroup. If so, there is a window we can exceed the quota limit when
bytes is in space_info-reserved.
Example:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 20987904 20987904 010485760 --- ---
qg-excl is 20987904 larger than max_excl 10485760.
This patch introduce a new counter named may_use to qgroup, then
there are three counters in qgroup to account bytes in space_info
as below.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
qgroup-may_use --- qgroup-reserved --- qgroup-excl
With this patch applied:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
fallocate: /mnt/data9: fallocate failed: Disk quota exceeded
fallocate: /mnt/data10: fallocate failed: Disk quota exceeded
fallocate: /mnt/data11: fallocate failed: Disk quota exceeded
fallocate: /mnt/data12: fallocate failed: Disk quota exceeded
fallocate: /mnt/data13: fallocate failed: Disk quota exceeded
fallocate: /mnt/data14: fallocate failed: Disk quota exceeded
fallocate: /mnt/data15: fallocate failed: Disk quota exceeded
fallocate: /mnt/data16: fallocate failed: Disk quota exceeded
fallocate: /mnt/data17: fallocate failed: Disk quota exceeded
fallocate: /mnt/data18: fallocate failed: Disk quota exceeded
fallocate: /mnt/data19
Re: [PATCH v2 2/2] Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use.
On 12/26/2014 01:43 PM, Satoru Takeuchi wrote:
Hi Yang,
On 2014/12/26 10:32, Dongsheng Yang wrote:
Hi Satoru,
I saw your mail of [BUG] Quota Ignored On write problem still
exist with 3.16-rc5
http://news.gmane.org/find-root.php?message_id=53C8DEB0.1060404%40jp.fujitsu.com
in gmane.
I guess this patch will fix the problem you mentioned.
Could you help to give a try?
Although it reached disk quota before writing whole 1.5GB,
the copied size was only about 700 MB. In this case,
expected behavior is to reach disk quota just after
writing 1 GB.
Hi Satoru,
Thanx for your testing a lot. But stranger, I tried it in 3.19-rc1 and
3.18, both works well
writing 1GB. Could you give me some more information about your testing
machine?
And could you execute sync btrfs qgroup show -pecr $MNT after writing?
*3.19-rc1 with my 2/2 patch applied.
===
[root@atest-guest linux_btrfs]# uname -a
Linux atest-guest 3.19.0-rc1+ #66 SMP Fri Dec 26 10:38:12 EST 2014
x86_64 x86_64 x86_64 GNU/Linux
[root@atest-guest linux_btrfs]# sh quota.sh
umount: /mnt: not mounted
Btrfs v3.17-1-gbef9fd4
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per file
to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 50.00GiB
Create subvolume '/mnt/quota_test'
dd: error writing \u2018/mnt/quota_test/test\u2019: Disk quota exceeded
999842+0 records in
999841+0 records out
1023837184 bytes (1.0 GB) copied, 8.06996 s, 127 MB/s
[PASS] quota works correctly
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 50G 996M 49G 2% /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -pecr /mnt/
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0--- ---
0/2561023868928 1023868928 102400 0--- ---
=
* 3.19-rc1 without your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024
count=150
150+0 records in
150+0 records out
153600 bytes (1.5 GB) copied, 25.6601 s, 59.9 MB/s
===
* 3.19-rc1 with your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024
count=150
dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded
681353+0 records in
681352+0 records out
697704448 bytes (698 MB) copied, 8.15563 s, 85.5 MB/s
===
Thanks,
Satoru
Thanx
Yang
On 12/12/2014 04:44 PM, Dongsheng Yang wrote:
Currently, for pre_alloc or delay_alloc, the bytes will be accounted
in space_info by the three guys.
space_info-bytes_may_use --- space_info-reserved ---
space_info-used.
But on the other hand, in qgroup, there are only two counters to
account the
bytes, qgroup-reserved and qgroup-excl. And qg-reserved accounts
bytes in space_info-bytes_may_use and qg-excl accounts bytes in
space_info-used. So the bytes in space_info-reserved is not accounted
in qgroup. If so, there is a window we can exceed the quota limit when
bytes is in space_info-reserved.
Example:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5
Re: [PATCH v2 2/2] Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use.
On 12/26/2014 02:49 PM, Dongsheng Yang wrote:
On 12/26/2014 01:43 PM, Satoru Takeuchi wrote:
Hi Yang,
On 2014/12/26 10:32, Dongsheng Yang wrote:
Hi Satoru,
I saw your mail of [BUG] Quota Ignored On write problem still
exist with 3.16-rc5
http://news.gmane.org/find-root.php?message_id=53C8DEB0.1060404%40jp.fujitsu.com
in gmane.
I guess this patch will fix the problem you mentioned.
Could you help to give a try?
Although it reached disk quota before writing whole 1.5GB,
the copied size was only about 700 MB. In this case,
expected behavior is to reach disk quota just after
writing 1 GB.
Hi Satoru,
Thanx for your testing a lot. But stranger
Typo, strange. :)
I tried it in 3.19-rc1 and 3.18, both works well
writing 1GB. Could you give me some more information about your
testing machine?
And could you execute sync btrfs qgroup show -pecr $MNT after
writing?
*3.19-rc1 with my 2/2 patch applied.
I also tried with the two patches applied, and got the same result.
Thanx
===
[root@atest-guest linux_btrfs]# uname -a
Linux atest-guest 3.19.0-rc1+ #66 SMP Fri Dec 26 10:38:12 EST 2014
x86_64 x86_64 x86_64 GNU/Linux
[root@atest-guest linux_btrfs]# sh quota.sh
umount: /mnt: not mounted
Btrfs v3.17-1-gbef9fd4
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 50.00GiB
Create subvolume '/mnt/quota_test'
dd: error writing \u2018/mnt/quota_test/test\u2019: Disk quota exceeded
999842+0 records in
999841+0 records out
1023837184 bytes (1.0 GB) copied, 8.06996 s, 127 MB/s
[PASS] quota works correctly
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 50G 996M 49G 2% /mnt
[root@atest-guest linux_btrfs]# btrfs qgroup show -pecr /mnt/
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 16384 16384 0 0--- ---
0/2561023868928 1023868928 102400 0--- ---
=
* 3.19-rc1 without your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024
count=150
150+0 records in
150+0 records out
153600 bytes (1.5 GB) copied, 25.6601 s, 59.9 MB/s
===
* 3.19-rc1 with your two v2 patches
===
+ mkfs.btrfs -f /dev/vdb
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.
Turning ON incompat feature 'extref': increased hardlink limit per
file to 65536
fs created label (null) on /dev/vdb
nodesize 16384 leafsize 16384 sectorsize 4096 size 30.00GiB
+ mount /dev/vdb /root/btrfs-auto-test/
+ ret=0
+ btrfs quota enable /root/btrfs-auto-test/
+ btrfs subvolume create /root/btrfs-auto-test//sub
Create subvolume '/root/btrfs-auto-test/sub'
+ btrfs qgroup limit 1G /root/btrfs-auto-test//sub
+ dd if=/dev/zero of=/root/btrfs-auto-test//sub/file bs=1024
count=150
dd: error writing '/root/btrfs-auto-test//sub/file': Disk quota exceeded
681353+0 records in
681352+0 records out
697704448 bytes (698 MB) copied, 8.15563 s, 85.5 MB/s
===
Thanks,
Satoru
Thanx
Yang
On 12/12/2014 04:44 PM, Dongsheng Yang wrote:
Currently, for pre_alloc or delay_alloc, the bytes will be accounted
in space_info by the three guys.
space_info-bytes_may_use --- space_info-reserved ---
space_info-used.
But on the other hand, in qgroup, there are only two counters to
account the
bytes, qgroup-reserved and qgroup-excl. And qg-reserved accounts
bytes in space_info-bytes_may_use and qg-excl accounts bytes in
space_info-used. So the bytes in space_info-reserved is not
accounted
in qgroup. If so, there is a window we can exceed the quota limit when
bytes is in space_info-reserved.
Example:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
# sync
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On 12/18/2014 12:07 PM, Robert White wrote: I don't disagree with the _ideal_ of your patch. I just think that it's impossible to implement it without lying to the user or making things just as bad in a different way. I would _like_ you to be right. But my thing is finding and quantifying failure cases and the entire question is full of fail. ... See you keep giving me these examples where the history of the filesystem is uniform. It was made a certain way and stayed that way. But in real life this sort of thing is going to happen and your patch simply report's a _different_ _wrong_ number. A _friendlier_ wrong number, I'll grant you that, but still wrong. Hi Robert, sorry for the late. (It's busy to deal with some other work.) Yes, you are providing some examples here to show that in some cases the numbers from df is not helpful to understand the real space state in disk. But I'm afraid we can not blame df reporting a wrong number. You could say Hey df, you are stupid, we can store the small file in metadata to exceed your @size.. But he just reports the information from what he is seeing from FS level honestly. He is reporting what he can see, and do not care about the other things out of his business. The all special cases you provided in this thread, actually lead to one result that Btrfs is special, df command will report an improper in some case.. It means we need some other methods to tell user about the space info. And yes, we had. It's btrfs fi df. You are trying to make df showing information as more as possible, even change the behaviour of df in btrfs to show the numbers of different meaning with what it is in other filesystems. And my answer is keeping df command doing what he want, and solve the problems in our private df , btrfs fi df. Thanx Yang . xaE -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On 12/17/2014 03:52 AM, Robert White wrote:
On 12/16/2014 03:30 AM, Dongsheng Yang wrote:
Hi Robert, thanx for your proposal about this.
IMHO, output of df command shoud be more friendly to user.
Well, I think we have a disagreement on this point, let's take a look at
what the zfs is doing.
/dev/sda7- 10G
/dev/sda8- 10G
# zpool create myzpool mirror /dev/sda7 /dev/sda8 -f
# df -h /myzpool/
Filesystem Size Used Avail Use% Mounted on
myzpool 9.8G 21K 9.8G 1% /myzpool
That said that df command should tell user the space info they can see.
It means the output is the information from the FS level rather than
device level or _storage_manager level.
That's great for ZFS, but ZFS isn't BTRFS. ZFS can't get caught
halfway between existing modailties and sit there forever. ZFS doesn't
restructure itself. So the simple answer you want isn't _possible_
outside very simple cases.
So again, you've displayed a _simple_ case as if it covers or
addresses all the complex cases.
(I don't have the storage to actually do the exercise) But what do you
propose the correct answer is for the following case:
/dev/sda - 7.5G
/dev/sdb - 7.5G
mkfs.btrfs /dev/sd{a,b} -d raid0
mount /dev/sda /mnt
dd if=/dev/urandom of=/mnt/consumed bs=1G count=7
btrfsck balance start -dconvert=raid1 -dlimit=1 /mnt
(wait)
/bin/df
The filesystem is now in a superposition where all future blocks are
going to be written as raid1, one 2G stripe has been converted into
two two-gig stripes that have been mirrored, and six gig is still RAID0.
Similar with your mail about inline file case, I think this is another
thing.
I really appreciate your persistence and so much analyse in semasiology.
I think I will never convince you at this point even with thousands mails
to express myself. What about implementing your proposal here and sending
a patchset at least with RFC. Then it can be more clear to us and we can
make
the choice more easily. :)
Again, below is all points from my side:
The df command we discussed here is on the top level which is directly
facing the user.
1), For a linux user, he does not care about the detail how the data is
stored in devices. They do not care even not know what's Single, what
does DUP mean, and how a fs implement the RAID10. What they want to
know is *what is the size of the filesystem I am using and how much
space is still available to me*. That's what I said by FS space level
2). For a btrfs user, they know about the single, dup and RAIDX. When they
want to know what's the raid level in each space info, they can use btrfs
fi df to print the information they want.
3). Device level. for debugging.
Sometimes, you need to know how the each chunk is stored in device. Please
use btrfs-debug-tree to show details you want as more as possible.
4). df in ZFS is showing the FS space information.
5). For the elder btrfs_statfs(), We have discussed about df command
and chosen to hide the detail information to user.
And only show the FS space information to user. Current btrfs_statfs()
is working like
this.
6). IIUC, you are saying that:
a). BTRFS is not ZFS, df in zfs is not referenced to us.
b). Current btrfs_statfs() is woring in a wrong way, we
need to reimplement it from another new view point.
I am not sure your proposal here is not better. As I said above,
I am pleased to see a demo of it. If it works better, I am really
very happy to agree with you in this argument.
My patch could be objected, but the problem in current btrfs_statfs()
should be fixed by some ways. If you submit a better solution, I am pleased
to see btrfs becoming better. :)
Thanx
Yang
In your proposal we now have
@size=7G
@used=??? (clearly 7G, at the least, is consumed)
@filesize[consumed]=7G
@available is really messed up since there is now _probably_ 1G of one
of the original raid0 extents with free space and so available, almost
all of the single RAID1 metadata block, Room for three more metadata
stripes, and room for one more RAID1 extent.
so @available=2-ish gigs.
But since statfs() pivots on @size and @available /bin/df is going to
report @used as 3-ish gigs even though we've got an uncompressed and
uncompressable @7G file.
NOW waht if we went the _other_ way?
/dev/sda - 7.5G
/dev/sdb - 7.5G
mkfs.btrfs /dev/sd{a,b} -d raid1
mount /dev/sda /mnt
dd if=/dev/urandom of=/mnt/consumed bs=1G count=7
btrfsck balance start -dconvert=raid0 -dlimit=1 /mnt
(wait)
/bin/df
filesystem is _full_ when the convert starts.
@size=14Gig
@used=7G
@actual_available=0
@reported_available=??? (at least 2x1G extents are up for grabs so
minimum 2G)
@reported_used=???
@calculated_used=???
We are either back to reporting available space when non-trivial
allocation will report ENOSPC (if I did the math right etc).
Now do partial conversions to other formats and repeat the exercise.
Now add or remove storage here-or-there.
The working set and the current model are not _required_
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On 12/16/2014 11:30 AM, Robert White wrote:
On 12/15/2014 01:36 AM, Robert White wrote:
So we don't just hand-wave over statfs(). We include the
dev_item.bytes_excluded in the superblock and we decide once-and-for-all
(with any geometry creation, or completed conversion) how many bytes
just _can't_ be reached but only once we _know_ they cant be reached.
And we memorialize that unreachable data in the superblocks.
Thereafter we report the raw numbers after subtracting anything we know
cannot be reached.
All other helpful solutions are NP-complete and insoluble.
On multiple re-readings of my own words and running off to the POSIX
definitions _and_ kernel sources (which don't agree).
The practical bits first ::
I would add a -c | --compatable option to btrfs fi df
that let it produce /bin/df format-compatable output that gave the
real numbers as defined near the end.
/dev/sda 1TiB
/dev/sdb 2TiB
mkfs.btrfs /dev/sd{a,b} -d raid1
@size=3TiB @used=0TiB @available=2TiB
The above would be ideal. But POSIX says no. f_blocks is defined
(only in the comments) as total data blocks in the filesystem and
/bin/df pivots on that assumption, so the only usable option left is ::
@size=2TiB @used=0TiB @available=2TiB
After which @used would be the real, raw space consumed. If it takes
2GiB or 4GiB to store 1GiB (q.v. RAID 1 and 10) then @used would go up
by that 2 or 4 GiB.
Hi Robert, thanx for your proposal about this.
IMHO, output of df command shoud be more friendly to user.
Well, I think we have a disagreement on this point, let's take a look at
what the zfs is doing.
/dev/sda7- 10G
/dev/sda8- 10G
# zpool create myzpool mirror /dev/sda7 /dev/sda8 -f
# df -h /myzpool/
Filesystem Size Used Avail Use% Mounted on
myzpool 9.8G 21K 9.8G 1% /myzpool
That said that df command should tell user the space info they can see.
It means the output is the information from the FS level rather than
device level or _storage_manager level.
Thanx
Yang
Given the not-displayed, not reported, excluded_by_geometry values
(e.g. @waste) the equation should always be ::
@size - @waste = @used + @available
The fact that /bin/df doesn't display all four values is just tough,
The fact that it calculates one for us is really annoying,
show-super would be the place to go find the truth.
The @waste value is soft because while 1TiB of /dev/sdb that is not
going to be used isn't a _particular_ 1TiB. It could be low blocks or
high blocks or randomly distributed blocks that end up not having data.
So keeping with my thought that (ideally) @size should be the safe dd
size for doing a raw-block transcribe of the devices and filesystem,
it is most correct for @size to be real storage size. But sadly, posix
didn't define that value for that role, so we are forced to munge
around. (particularly since /bin/df calculates stuff for us).
Calculation of the @waste would have to happen in two phases. At
initiation phase of any convert @waste would be set to zero. At
completion of any _full_ convert, when we know that there are no
leftover bits that could lead to rampant mis-report, @waste would be
calculated for each device as a dev_item. Then the total would be
stored as a global item.
btrfs tools would report all four items.
statfs() would have to report (@size-@waste) and @available, but
that's a problem with limits to the assumptions made by statfs()
designers two decades ago.
I don't know which numbers we keep on hand and which we derive so...
@available, if calculated dynamically would be
sum(@size, -@waste, -@used).
@used, if calculated dynamically, would be
sum(@size, -@waste, -@available).
This would also keep all the determinations of @waste well defined and
relegated to specific, infrequently executed blocks of code.
GIVEN ALSO ::
The BTRFS dynamic internal layout allows for completely valid states
that are inconsistent with the current filesystem flags... Such as it
is legal to set the RAID1 mode for data but still having RAID0, RAID5,
and any manner of other extents present... there is no finite solution
to every particular layout that exists.
This condition is even _mandatory_ in an evolving system. May persist
if conversion is interrupted and then the balance is aborted. And
might be purely legal if you supply a convert option and limit the
number of blocks to process in the same run.
Each individual extent block is it's own master in terms of what mode
the filesystem is actally in when that extent is being accessed. This
fact is _unchangeable_.
STANDARDS REFERENCES and Issues...
The actual standard from POSIX at The Open Group refers to f_blocks as
Total number of blocks on file system in units of f_frsize.
See ::
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/statvfs.h.html
The linux kernel source and man pages say total data blocks in
filesystem.
I don't know where/when/why the total blocks got re-qualified as
total
Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]
On Tue, Dec 16, 2014 at 7:30 PM, Dongsheng Yang
yangds.f...@cn.fujitsu.com wrote:
On 12/16/2014 11:30 AM, Robert White wrote:
On 12/15/2014 01:36 AM, Robert White wrote:
So we don't just hand-wave over statfs(). We include the
dev_item.bytes_excluded in the superblock and we decide once-and-for-all
(with any geometry creation, or completed conversion) how many bytes
just _can't_ be reached but only once we _know_ they cant be reached.
And we memorialize that unreachable data in the superblocks.
Thereafter we report the raw numbers after subtracting anything we know
cannot be reached.
All other helpful solutions are NP-complete and insoluble.
On multiple re-readings of my own words and running off to the POSIX
definitions _and_ kernel sources (which don't agree).
The practical bits first ::
I would add a -c | --compatable option to btrfs fi df
that let it produce /bin/df format-compatable output that gave the real
numbers as defined near the end.
/dev/sda 1TiB
/dev/sdb 2TiB
mkfs.btrfs /dev/sd{a,b} -d raid1
@size=3TiB @used=0TiB @available=2TiB
The above would be ideal. But POSIX says no. f_blocks is defined (only
in the comments) as total data blocks in the filesystem and /bin/df pivots
on that assumption, so the only usable option left is ::
@size=2TiB @used=0TiB @available=2TiB
After which @used would be the real, raw space consumed. If it takes 2GiB
or 4GiB to store 1GiB (q.v. RAID 1 and 10) then @used would go up by that 2
or 4 GiB.
Hi Robert, thanx for your proposal about this.
IMHO, output of df command shoud be more friendly to user.
Well, I think we have a disagreement on this point, let's take a look at
what the zfs is doing.
/dev/sda7- 10G
/dev/sda8- 10G
# zpool create myzpool mirror /dev/sda7 /dev/sda8 -f
# df -h /myzpool/
Filesystem Size Used Avail Use% Mounted on
myzpool 9.8G 21K 9.8G 1% /myzpool
That said that df command should tell user the space info they can see.
It means the output is the information from the FS level rather than device
level or _storage_manager level.
Addition:
There are some other ways to get the space information in btrfs, btrfs
fi df, btrfs fi show, btrfs-debug-tree.
The df command we discussed here is on the top level which is directly
facing the user. Let me try to show the difference in different level.
1), TOP level, for a linux user: df command.
For a linux user, he does not care about the detail how the data is
stored in devices.
They do not care even not know what's Single, what does DUP mean, and how
a fs implement the RAID10. What they want to know is *what is the size
of the filesystem
I am using and how much space is still available to me*. That's what I
said by FS space level)
2). Middle level, for a btrfs user: btrfs fi df/show.
For a btrfs user, they know about the single, dup and RAIDX. When they
want to know
what's the raid level in each space info, they can use btrfs fi df to
print the information
they want.
3). Device level. for debugging.
Sometimes, you need to know how the each chunk is stored in device. Please
use btrfs-debug-tree to show details you want as more as possible.
After all, I would say, the information you want to show is *not*
incorrect to me, but it's not the
business of df command.
Thanx
Thanx
Yang
Given the not-displayed, not reported, excluded_by_geometry values (e.g.
@waste) the equation should always be ::
@size - @waste = @used + @available
The fact that /bin/df doesn't display all four values is just tough, The
fact that it calculates one for us is really annoying, show-super would be
the place to go find the truth.
The @waste value is soft because while 1TiB of /dev/sdb that is not going
to be used isn't a _particular_ 1TiB. It could be low blocks or high blocks
or randomly distributed blocks that end up not having data.
So keeping with my thought that (ideally) @size should be the safe dd
size for doing a raw-block transcribe of the devices and filesystem, it is
most correct for @size to be real storage size. But sadly, posix didn't
define that value for that role, so we are forced to munge around.
(particularly since /bin/df calculates stuff for us).
Calculation of the @waste would have to happen in two phases. At
initiation phase of any convert @waste would be set to zero. At completion
of any _full_ convert, when we know that there are no leftover bits that
could lead to rampant mis-report, @waste would be calculated for each device
as a dev_item. Then the total would be stored as a global item.
btrfs tools would report all four items.
statfs() would have to report (@size-@waste) and @available, but that's a
problem with limits to the assumptions made by statfs() designers two
decades ago.
I don't know which numbers we keep on hand and which we derive so...
@available, if calculated dynamically would be
sum(@size, -@waste, -@used).
@used, if calculated dynamically, would be
sum(@size
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
Hi Goffredo, On Tue, Dec 16, 2014 at 1:47 AM, Goffredo Baroncelli kreij...@inwind.it wrote: I Yang, On 12/14/2014 12:29 PM, Dongsheng Yang wrote: On Sat, Dec 13, 2014 at 3:25 AM, Goffredo Baroncelli kreij...@inwind.it wrote: On 12/11/2014 09:31 AM, Dongsheng Yang wrote: When function btrfs_statfs() calculate the tatol size of fs, it is calculating the total size of disks and then dividing it by a factor. But in some usecase, the result is not good to user. I Yang; during my test I discovered an error: $ sudo lvcreate -L +10G -n disk vgtest $ sudo /home/ghigo/mkfs.btrfs -f /dev/vgtest/disk Btrfs v3.17 See http://btrfs.wiki.kernel.org for more information. Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 fs created label (null) on /dev/vgtest/disk nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB $ sudo mount /dev/vgtest/disk /mnt/btrfs1/ $ df /mnt/btrfs1/ Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/vgtest-disk 9428992 1069312 8359680 12% /mnt/btrfs1 $ sudo ~/btrfs fi df /mnt/btrfs1/ Data, single: total=8.00MiB, used=256.00KiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B What seems me strange is the 9428992KiB of total disk space as reported by df. About 600MiB are missing ! Without your patch, I got: $ df /mnt/btrfs1/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vgtest-disk 10485760 16896 8359680 1% /mnt/btrfs1 Hi Goffredo, thanx for pointing this. Let me try to show the reason of it. In this case you provided here, the metadata is 1G in DUP. It means we spent 2G device space for it. But from the view point of user, they can only see 9G size for this fs. It's show as below: 1G -| Filesystem space (9G) | \ | \ | \ |2G \ -||- Device space (10G) DUP | The main idea about the new btrfs_statfs() is hiding the detail in Device space and only show the information in the FS space level. I am not entirely convinced. On the basis of your statement, I would find a difference of 1G or 1G/2... but missing are 600MB... Ha, forgot to clarify this one. 9428992K is almost 9G actually. 9G = (9 * 2^20)K = 9437184K. So, as I showed in the graphic, @size is 9G. :) Grzegorz found a difference of 2.3GB (but on a bigger disk). Looking at your patch set, it seems to me that the computation of the total disk space is done differently than before. Before, the total disk space was - buf-f_blocks = div_u64(btrfs_super_total_bytes(disk_super), factor); - buf-f_blocks = bits; I read this as: total disk space is the sum of the disk size (corrected by a factor depending by the profile). Instead after your patches the total disk space is + buf-f_blocks = total_alloc + total_free_data; This means that if a disk is not fully mapped by chunks, there is a difference between the total disk space and the value reported by df. May be that you are highlighting a bug elsewhere (not caused by your patch) ? BTW I will continue my tests... Thank you very much. :) Yang BR G.Baroncelli Actually, the similar idea is adopted in btrfs fi df. Example: # lvcreate -L +10G -n disk Group0 # mkfs.btrfs -f /dev/Group0/disk # dd if=/dev/zero of=/mnt/data bs=1M count=10240 dd: error writing ‘/mnt/data’: No space left on device 8163+0 records in 8162+0 records out 8558477312 bytes (8.6 GB) copied, 207.565 s, 41.2 MB/s # df /mnt Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/Group0-disk 9428992 8382896 2048 100% /mnt # btrfs fi df /mnt Data, single: total=7.97GiB, used=7.97GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=8.41MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B Now, the all space is used and got a ENOSPC error. But from the output of btrfs fi df. We can only find almost 9G (data 8G + metadata 1G) is used. The difference is that it show the DUP here to make it more clear. Wish this description is clear to you :). If there is anything confusing or sounds incorrect to you, please point it out. Thanx Yang Example: # mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1 # mount /dev/vdf1 /mnt # dd if=/dev/zero of=/mnt/zero bs=1M count=1000 # df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/vdf1 3.0G 1018M 1.3G 45% /mnt # btrfs fi show /dev/vdf1 Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294 Total devices 2 FS bytes used 1001.53MiB devid1 size 2.00GiB used 1.85GiB path /dev/vdf1 devid2 size
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
On 12/15/2014 03:49 PM, Robert White wrote:
On 12/14/2014 10:06 PM, Robert White wrote:
On 12/14/2014 05:21 PM, Dongsheng Yang wrote:
Anyone have some suggestion about it?
(... strong advocacy for raw numbers...)
Hi Robert, thanx for your so detailed reply.
You are proposing to report the raw numbers in df command, right?
Let's compare the space information in FS level and Device level.
Example:
/dev/sda == 1TiB
/dev/sdb == 2TiB
mkfs.btrfs /dev/sda /dev/sdb -d raid1
(1). If we report the raw numbers in df command, we will get the result of
@size=3T, @used=0 @available=3T. It's not a bad idea until now, as you said
user can consider the raid when they are using the fs. Then if we fill
1T data
into it. we will get @size=3, @used=2T, @avalable=1T. And at this moment, we
will get ENOSPC when writing any more data. It's unacceptable. Why you
tell me there is 1T space available, but I can't write one byte into it?
(2). Actually, there was an elder btrfs_statfs(), it reported the raw
numbers to user.
To solve the problem mentioned in (1), we need report the space
information in the FS level.
Current btrfs_statfs() is designed like this, but not working in any cases.
My new btrfs_statfs() here is following this design and implementing it
to show a *better* output to user.
Thanx
Yang
Concise Example to attempt to be clearer:
/dev/sda == 1TiB
/dev/sdb == 2TiB
/dev/sdc == 3TiB
/dev/sdd == 3TiB
mkfs.btrfs /dev/sd{a..d} -d raid0
mount /dev/sda /mnt
Now compare ::
#!/bin/bash
dd if=/dev/urandom of=/mnt/example bs=1G
vs
#!/bin/bash
typeset -i counter
for ((counter=0;;counter++)); do
dd if=/dev/urandom of=/mnt/example$conter bs=44 count=1
done
vs
#!/bin/bash
typeset -i counter
for ((counter=0;;counter++)); do
dd if=/dev/urandom of=/mnt/example$conter bs=44 count=1
done
dd if=/dev/urandom of=/mnt/example bs=1G
Now repeat the above 3 models for
mkfs.btrfs /dev/sd{a..d} -d raid5
..
As you watch these six examples evolve you can ponder the ultimate
futility of doing adaptive prediction within statfs().
Then go back and change the metadata from the default of RAID1 to
RAID5 or RAID6 or RAID10.
Then go back and try
mkfs.btrfs /dev/sd{a..d} -d raid10
then balance when the big file runs out of space, then resume the big
file with oflag=append
..
Unlike _all_ our predecessors, we are active at both the semantic file
storage level _and_ the physical media management level.
None of the prior filesystems match this new ground exactly.
The only real option is to expose the raw numbers and then tell people
the corner cases.
Absolutely unavailable blocks, such as the massive waste of 5TiB in
the above sized media if raid10 were selected for both data and
metadata would be subtracted from size if and only if it's
_impossible_ for it to be accessed by this sort of restriction. But
even in this case, the correct answer for size is 4TiB because that
exactly answers how big is this filesystem.
It might be worth having a dev_item.bytes_excluded or unusable or
whatever to account for the difference between total_bytes and
bytes_used and the implicit bytes available. This would account for
the 0,1,2,2 TiB that a raid10 of the example sizes could never reach
in the current geometry. I'm betting that this sort of number also
shows up as some number of sectors in any filesystem that has an odd
tidbit of size up at the top where no structure is ever gong to fit.
That's just a feature of the way disks use GB instead of GiB and msdos
style partitions love the number 63.
So resize sets the size. Geometry limitations may reduce the effective
size by some, or a _lot_, but then the used-vs-available should _not_
try to correct for whatever geometry is in use. Even when it might be
simple because if it does it well in the simple cases like
raid10/raid10, it would have to botch it up on the hard cases.
.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
On Sat, Dec 13, 2014 at 3:25 AM, Goffredo Baroncelli kreij...@inwind.it wrote: On 12/11/2014 09:31 AM, Dongsheng Yang wrote: When function btrfs_statfs() calculate the tatol size of fs, it is calculating the total size of disks and then dividing it by a factor. But in some usecase, the result is not good to user. I Yang; during my test I discovered an error: $ sudo lvcreate -L +10G -n disk vgtest $ sudo /home/ghigo/mkfs.btrfs -f /dev/vgtest/disk Btrfs v3.17 See http://btrfs.wiki.kernel.org for more information. Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 fs created label (null) on /dev/vgtest/disk nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB $ sudo mount /dev/vgtest/disk /mnt/btrfs1/ $ df /mnt/btrfs1/ Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/vgtest-disk 9428992 1069312 8359680 12% /mnt/btrfs1 $ sudo ~/btrfs fi df /mnt/btrfs1/ Data, single: total=8.00MiB, used=256.00KiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B What seems me strange is the 9428992KiB of total disk space as reported by df. About 600MiB are missing ! Without your patch, I got: $ df /mnt/btrfs1/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vgtest-disk 10485760 16896 8359680 1% /mnt/btrfs1 Hi Goffredo, thanx for pointing this. Let me try to show the reason of it. In this case you provided here, the metadata is 1G in DUP. It means we spent 2G device space for it. But from the view point of user, they can only see 9G size for this fs. It's show as below: 1G -| Filesystem space (9G) | \ | \ | \ |2G \ -||- Device space (10G) DUP | The main idea about the new btrfs_statfs() is hiding the detail in Device space and only show the information in the FS space level. Actually, the similar idea is adopted in btrfs fi df. Example: # lvcreate -L +10G -n disk Group0 # mkfs.btrfs -f /dev/Group0/disk # dd if=/dev/zero of=/mnt/data bs=1M count=10240 dd: error writing ‘/mnt/data’: No space left on device 8163+0 records in 8162+0 records out 8558477312 bytes (8.6 GB) copied, 207.565 s, 41.2 MB/s # df /mnt Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/Group0-disk 9428992 8382896 2048 100% /mnt # btrfs fi df /mnt Data, single: total=7.97GiB, used=7.97GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=8.41MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B Now, the all space is used and got a ENOSPC error. But from the output of btrfs fi df. We can only find almost 9G (data 8G + metadata 1G) is used. The difference is that it show the DUP here to make it more clear. Wish this description is clear to you :). If there is anything confusing or sounds incorrect to you, please point it out. Thanx Yang Example: # mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1 # mount /dev/vdf1 /mnt # dd if=/dev/zero of=/mnt/zero bs=1M count=1000 # df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/vdf1 3.0G 1018M 1.3G 45% /mnt # btrfs fi show /dev/vdf1 Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294 Total devices 2 FS bytes used 1001.53MiB devid1 size 2.00GiB used 1.85GiB path /dev/vdf1 devid2 size 4.00GiB used 1.83GiB path /dev/vdf2 a. df -h should report Size as 2GiB rather than as 3GiB. Because this is 2 device raid1, the limiting factor is devid 1 @2GiB. b. df -h should report Avail as 0.97GiB or less, rather than as 1.3GiB. 1.85 (the capacity of the allocated chunk) -1.018 (the file stored) +(2-1.85=0.15) (the residual capacity of the disks considering a raid1 fs) --- = 0.97 This patch drops the factor at all and calculate the size observable to user without considering which raid level the data is in and what's the size exactly in disk. After this patch applied: # mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1 # mount /dev/vdf1 /mnt # dd if=/dev/zero of=/mnt/zero bs=1M count=1000 # df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/vdf1 2.0G 1.3G 713M 66% /mnt # df /mnt Filesystem 1K-blocksUsed Available Use% Mounted on /dev/vdf12097152 1359424729536 66% /mnt # btrfs fi show /dev/vdf1 Label: none uuid: e98c1321-645f-4457-b20d-4f41dc1cf2f4 Total devices 2 FS bytes used 1001.55MiB devid1 size 2.00GiB used 1.85GiB path /dev/vdf1 devid2 size 4.00GiB used 1.83GiB path /dev/vdf2 a). The @Size is 2G as we
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
On 12/14/2014 10:32 PM, Grzegorz Kowal wrote: Hi, I see another problem on 1 device fs after applying this patch. I set up 30GB system partition: in gdisk key 'i' Partition size: 62914560 sectors (30.0 GiB) - 62914560 * 512 = 32212254720 = 30.0GiB before applying the patch df -B1 shows Filesystem1B-blocks UsedAvailable Use% Mounted on /dev/sda3 32212254720 18149384192 12821815296 59% / The total size matches exactly the one reported by gdisk. After applying the patch df -B1 shows Filesystem1B-blocks UsedAvailable Use% Mounted on /dev/sda3 29761732608 17037860864 12723871744 58% / The total size is 29761732608 bytes now, which is 2337 MiB (2.28GB) smaller then the partition size. Hi Grzegorz, this is very similar with the question discussed in my last mail. I believe the 2.28GB is used for metadata in DUP level. That said, there is a space of almost 4.56GB (2.28*2GB) used for metadata and the metadata is in DUP level. Then user can only see a 2.28GB from the FS space level. 2.28G -| Filesystem space (27.72G) | \ | \ | \ | 4.56GB \ -||- Device space (30.0G) DUP | The main idea in new btrfs_statfs() is to consider the space information in a FS level rather than the device space level. Then the @size in df is the size in FS space level. As we all agree to report @size as 5G in the case of mkfs.btrfs /dev/vdb (5G) /dev/vdc (5G) -d raid1, this means we are agreed to think the df in a FS space without showing the detail in device space. In this time, the total size in FS space is 27.72G, so the @size in df is 27.72G. Does it make sense to you? As there are 2 complaints for the same change of @size in df, I have to say it maybe not so easy to understand. Anyone have some suggestion about it? Thanx Yang IMHO the total size should correspond exactly to the partition size, at least in the case of fs on one device, and anything used by the file system and user should go to the used space. Am I missing something here? Thanks, Grzegorz On Sun, Dec 14, 2014 at 12:27 PM, Grzegorz Kowal custos.men...@gmail.com wrote: Hi, I see another problem on 1 device fs after applying this patch. I set up 30GB system partition: in gdisk key 'i' Partition size: 62914560 sectors (30.0 GiB) - 62914560 * 512 = 32212254720 = 30.0GiB before applying the patch df -B1 shows Filesystem1B-blocks UsedAvailable Use% Mounted on /dev/sda3 32212254720 18149384192 12821815296 59% / The total size matches exactly the one reported by gdisk. After applying the patch df -B1 shows Filesystem1B-blocks UsedAvailable Use% Mounted on /dev/sda3 29761732608 17037860864 12723871744 58% / The total size is 29761732608 bytes now, which is 2337 MiB (2.28GB) smaller then the partition size. IMHO the total size should correspond exactly to the partition size, at least in the case of fs on one device, and anything used by the file system and user should go to the used space. Am I missing something here? Thanks, Grzegorz On Sun, Dec 14, 2014 at 9:29 AM, Dongsheng Yang dongsheng081...@gmail.com wrote: On Sat, Dec 13, 2014 at 3:25 AM, Goffredo Baroncelli kreij...@inwind.it wrote: On 12/11/2014 09:31 AM, Dongsheng Yang wrote: When function btrfs_statfs() calculate the tatol size of fs, it is calculating the total size of disks and then dividing it by a factor. But in some usecase, the result is not good to user. I Yang; during my test I discovered an error: $ sudo lvcreate -L +10G -n disk vgtest $ sudo /home/ghigo/mkfs.btrfs -f /dev/vgtest/disk Btrfs v3.17 See http://btrfs.wiki.kernel.org for more information. Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 fs created label (null) on /dev/vgtest/disk nodesize 16384 leafsize 16384 sectorsize 4096 size 10.00GiB $ sudo mount /dev/vgtest/disk /mnt/btrfs1/ $ df /mnt/btrfs1/ Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/vgtest-disk 9428992 1069312 8359680 12% /mnt/btrfs1 $ sudo ~/btrfs fi df /mnt/btrfs1/ Data, single: total=8.00MiB, used=256.00KiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B What seems me strange is the 9428992KiB of total disk space as reported by df. About 600MiB are missing ! Without your patch, I got: $ df /mnt/btrfs1/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vgtest-disk 10485760 16896 8359680 1% /mnt/btrfs1 Hi Goffredo, thanx for pointing this. Let me try to show the reason of it. In this case you provided here, the metadata is 1G in DUP. It means we spent 2G device space for it. But from the view point of user, they can only see 9G size for this fs. It's show as below: 1G
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
On 12/13/2014 02:00 AM, Goffredo Baroncelli wrote:
On 12/11/2014 09:31 AM, Dongsheng Yang wrote:
When function btrfs_statfs() calculate the tatol size of fs, it is calculating
the total size of disks and then dividing it by a factor. But in some usecase,
the result is not good to user.
I am checking it; to me it seems a good improvement. However
I noticed that df now doesn't seem to report anymore the space consumed
by the meta-data chunk; eg:
# I have two disks of 5GB each
$ sudo ~/mkfs.btrfs -f -m raid1 -d raid1 /dev/vgtest/disk /dev/vgtest/disk1
$ df -h /mnt/btrfs1/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgtest-disk 5.0G 1.1G 4.0G 21% /mnt/btrfs1
$ sudo btrfs fi show
Label: none uuid: 884414c6-9374-40af-a5be-3949cdf6ad0b
Total devices 2 FS bytes used 640.00KB
devid2 size 5.00GB used 2.01GB path /dev/dm-1
devid1 size 5.00GB used 2.03GB path /dev/dm-0
$ sudo ./btrfs fi df /mnt/btrfs1/
Data, RAID1: total=1.00GiB, used=512.00KiB
Data, single: total=8.00MiB, used=0.00B
System, RAID1: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, RAID1: total=1.00GiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B
In this case the filesystem is empty (it was a new filesystem !). However a
1G metadata chunk was already allocated. This is the reasons why the free
space is only 4Gb.
Actually, in the original btrfs_statfs(), the space in metadata chunk
was also not be considered
as available. But you will get 5G in this case with original
btrfs_statfs() because there is a bug
in it. As I use another implementation to calculate the @available in df
command, I did not
mention this problem. I can describe it as below.
In original btrfs_statfs(), we only consider the free space in data
chunk as available.
list_for_each_entry_rcu(found, head, list) {
if (found-flags BTRFS_BLOCK_GROUP_DATA) {
int i;
total_free_data += found-disk_total -
found-disk_used;
In the later, we will add the total_free_data to @available in output of df.
buf-f_bavail = total_free_data;
BUT:
This is incorrect! It should be:
buf-f_bavail = div_u64(total_free_data, factor);
That said this bug will add one more (data_chunk_disk_total -
data_chunk_disk_used = 1G)
to @available by mistake. Unfortunately, the free space in
metadata_chunk is also 1G.
Then we will get 5G in this case you provided here.
Conclusion:
Even in the original btrfs_statfs(), the free space in metadata is
also considered as not available.
But one bug in it make it to be misunderstood. My new btrfs_statfs()
still consider the free space
in metadata as not available, furthermore, I add it into @used.
On my system the ratio metadata/data is 234MB/8.82GB = ~3%, so ignoring the
metadata chunk from the free space may not be a big problem.
Example:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 3.0G 1018M 1.3G 45% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: f85d93dc-81f4-445d-91e5-6a5cd9563294
Total devices 2 FS bytes used 1001.53MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a. df -h should report Size as 2GiB rather than as 3GiB.
Because this is 2 device raid1, the limiting factor is devid 1 @2GiB.
b. df -h should report Avail as 0.97GiB or less, rather than as 1.3GiB.
1.85 (the capacity of the allocated chunk)
-1.018 (the file stored)
+(2-1.85=0.15) (the residual capacity of the disks
considering a raid1 fs)
---
= 0.97
This patch drops the factor at all and calculate the size observable to
user without considering which raid level the data is in and what's the
size exactly in disk.
After this patch applied:
# mkfs.btrfs -f /dev/vdf1 /dev/vdf2 -d raid1
# mount /dev/vdf1 /mnt
# dd if=/dev/zero of=/mnt/zero bs=1M count=1000
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdf1 2.0G 1.3G 713M 66% /mnt
# df /mnt
Filesystem 1K-blocksUsed Available Use% Mounted on
/dev/vdf12097152 1359424729536 66% /mnt
# btrfs fi show /dev/vdf1
Label: none uuid: e98c1321-645f-4457-b20d-4f41dc1cf2f4
Total devices 2 FS bytes used 1001.55MiB
devid1 size 2.00GiB used 1.85GiB path /dev/vdf1
devid2 size 4.00GiB used 1.83GiB path /dev/vdf2
a). The @Size is 2G as we expected.
b). @Available is 700M = 1.85G - 1.3G + (2G - 1.85G).
c). @Used is changed to 1.3G rather than 1018M as above. Because
this patch do not treat the free space in metadata chunk
Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.
On 12/13/2014 08:50 AM, Duncan wrote: Goffredo Baroncelli posted on Fri, 12 Dec 2014 19:00:20 +0100 as excerpted: $ sudo ./btrfs fi df /mnt/btrfs1/ Data, RAID1: total=1.00GiB, used=512.00KiB Data, single: total=8.00MiB, used=0.00B System, RAID1: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, RAID1: total=1.00GiB, used=112.00KiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B In this case the filesystem is empty (it was a new filesystem !). However a 1G metadata chunk was already allocated. This is the reasons why the free space is only 4Gb. Trivial(?) correction. Metadata chunks are quarter-gig, not 1 gig. So that's 4 quarter-gig metadata chunks allocated, not a (one/single) 1-gig metadata chunk. Sorry but from my reading of the code, I have to say that in this case it is 1G. Maybe I should be wrong, I would say, yes, the chunk_size for btrfs which is smaller than 50G is 256M by default. But in mkfs, we alloc 1G for the first metadata chunk. On my system the ratio metadata/data is 234MB/8.82GB = ~3%, so ignoring the metadata chunk from the free space may not be a big problem. Presumably your use-case is primarily reasonably large files; too large for their data to be tucked directly into metadata instead of allocating an extent from a data chunk. That's not always going to be the case. And given the multi-device default allocation of raid1 metadata, single data, files small enough to fit into metadata have a default size effect double their actual size! (Tho it can be noted that given btrfs' 4 KiB standard block size, without metadata packing there'd still be an outsized effect for files smaller than half that, 2 KiB or under, but there it'd be in data chunks, not metadata.) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 2/2] Btrfs: qgroup: Introduce a may_use to account space_info-bytes_may_use.
Currently, for pre_alloc or delay_alloc, the bytes will be accounted
in space_info by the three guys.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
But on the other hand, in qgroup, there are only two counters to account the
bytes, qgroup-reserved and qgroup-excl. And qg-reserved accounts
bytes in space_info-bytes_may_use and qg-excl accounts bytes in
space_info-used. So the bytes in space_info-reserved is not accounted
in qgroup. If so, there is a window we can exceed the quota limit when
bytes is in space_info-reserved.
Example:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rfer excl max_rfer max_excl parent child
-- -
0/5 20987904 20987904 010485760 --- ---
qg-excl is 20987904 larger than max_excl 10485760.
This patch introduce a new counter named may_use to qgroup, then
there are three counters in qgroup to account bytes in space_info
as below.
space_info-bytes_may_use --- space_info-reserved --- space_info-used.
qgroup-may_use --- qgroup-reserved --- qgroup-excl
With this patch applied:
# btrfs quota enable /mnt
# btrfs qgroup limit -e 10M /mnt
# for((i=0;i20;i++));do fallocate -l 1M /mnt/data$i; done
fallocate: /mnt/data9: fallocate failed: Disk quota exceeded
fallocate: /mnt/data10: fallocate failed: Disk quota exceeded
fallocate: /mnt/data11: fallocate failed: Disk quota exceeded
fallocate: /mnt/data12: fallocate failed: Disk quota exceeded
fallocate: /mnt/data13: fallocate failed: Disk quota exceeded
fallocate: /mnt/data14: fallocate failed: Disk quota exceeded
fallocate: /mnt/data15: fallocate failed: Disk quota exceeded
fallocate: /mnt/data16: fallocate failed: Disk quota exceeded
fallocate: /mnt/data17: fallocate failed: Disk quota exceeded
fallocate: /mnt/data18: fallocate failed: Disk quota exceeded
fallocate: /mnt/data19: fallocate failed: Disk quota exceeded
# sync
# btrfs qgroup show -pcre /mnt
qgroupid rferexclmax_rfer max_excl parent child
-- -
0/5 9453568 9453568 010485760 --- ---
Reported-by: Cyril SCETBON cyril.scet...@free.fr
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
---
Changelog:
v1 - v2:
Remove the redundant check for fs_info-quota_enabled.
fs/btrfs/extent-tree.c | 20 ++-
fs/btrfs/inode.c | 18 -
fs/btrfs/qgroup.c | 68 +++---
fs/btrfs/qgroup.h | 4 +++
4 files changed, 104 insertions(+), 6 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 014b7f2..f4ad737 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5500,8 +5500,12 @@ static int pin_down_extent(struct btrfs_root *root,
set_extent_dirty(root-fs_info-pinned_extents, bytenr,
bytenr + num_bytes - 1, GFP_NOFS | __GFP_NOFAIL);
- if (reserved)
+ if (reserved) {
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ num_bytes, -1);
trace_btrfs_reserved_extent_free(root, bytenr, num_bytes);
+ }
return 0;
}
@@ -6230,6 +6234,9 @@ void btrfs_free_tree_block(struct btrfs_trans_handle
*trans,
btrfs_update_reserved_bytes(cache, buf-len, RESERVE_FREE, 0);
trace_btrfs_reserved_extent_free(root, buf-start, buf-len);
pin = 0;
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ buf-len, -1);
}
out:
if (pin)
@@ -6964,7 +6971,11 @@ static int __btrfs_free_reserved_extent(struct
btrfs_root *root,
else {
btrfs_add_free_space(cache, start, len);
btrfs_update_reserved_bytes(cache, len, RESERVE_FREE, delalloc);
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid,
+ len, -1);
}
+
btrfs_put_block_group(cache);
trace_btrfs_reserved_extent_free(root, start, len);
@@ -7200,6 +7211,9 @@ int btrfs_alloc_logged_file_extent(struct
btrfs_trans_handle *trans,
BUG_ON(ret); /* logic error */
ret = alloc_reserved_file_extent(trans, root, 0, root_objectid,
0, owner, offset, ins, 1);
+ btrfs_qgroup_update_reserved_bytes(root-fs_info,
+ root-root_key.objectid
[PATCH v2 1/2] Btrfs: qgroup: free reserved in exceeding quota.
When we exceed quota limit in writing, we will free
some reserved extent when we need to drop but not free
account in qgroup. It means, each time we exceed quota
in writing, there will be some remain space in qg-reserved
we can not use any more. If things go on like this, the
all space will be ate up.
Signed-off-by: Dongsheng Yang yangds.f...@cn.fujitsu.com
Reviewed-by: Josef Bacik jba...@fb.com
---
fs/btrfs/extent-tree.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index a84e00d..014b7f2 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5262,8 +5262,11 @@ out_fail:
to_free = 0;
}
spin_unlock(BTRFS_I(inode)-lock);
- if (dropped)
+ if (dropped) {
+ if (root-fs_info-quota_enabled)
+ btrfs_qgroup_free(root, dropped * root-nodesize);
to_free += btrfs_calc_trans_metadata_size(root, dropped);
+ }
if (to_free) {
btrfs_block_rsv_release(root, block_rsv, to_free);
--
1.8.4.2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: get more accurate output in fd command.
On 12/11/2014 03:05 AM, Goffredo Baroncelli wrote: On 12/10/2014 04:02 PM, Dongsheng Yang wrote: On Wed, Dec 10, 2014 at 9:21 PM, Duncan 1i5t5.dun...@cox.net wrote: Robert White posted on Wed, 10 Dec 2014 02:53:40 -0800 as excerpted: [...] And in the example, the mkfs was supplied with two devices, so there's no dup metadata remaining from a formerly single-device filesystem, either. (Tho there will be the small single-mode stubs, empty, remaining from the mkfs process, as no balance has been run to delete them yet, but those are much smaller and empty.) Yes. One question not related here: how about delete them in the end of mkfs? Thanx A btrfs balance should remove them. If you don't want to balance a full filesystem, you can filter the chunk by usage (set a low usage). Recently it was discussed in a tread... Thanx Goffredo, it works well to me. BR Goffredo -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
