Re: ERROR: parent determination failed (btrfs send-receive)

2017-09-17 Thread Andrei Borzenkov 
18.09.2017 05:31, Dave пишет:
> Sometimes when using btrfs send-receive, I get errors like this:
> 
> ERROR: parent determination failed for 
> 
> When this happens, btrfs send-receive backups fail. And all subsequent
> backups fail too.
> 
> The issue seems to stem from the fact that an automated cleanup
> process removes certain earlier subvolumes. (I'm using Snapper.)
> 
> I'd like to understand exactly what is happening so that my backups do
> not unexpectedly fail.
> 

You try to send incremental changes but you deleted subvolume to compute
changes against. It is hard to tell more without seeing subvolume list
with uuid/parent uuid.

> In my scenario, no parent subvolumes have been deleted from the
> target. Some subvolumes have been deleted from the source, but why
> does that matter? I am able to take a valid snapshot at this time and
> every snapshot ever taken continues to reside at the target backup
> destination (seemingly meaning that a parent subvolume can be found at
> the target).
> 
> This issue seems to make btrfs send-receive a very fragile backup
> solution. 

btrfs send/receive is not backup solution - it is low level tool that
does exactly what it is told to do. You may create backup solution that
is using btrfs send/receive to transfer data stream, but then do not
blame tool for incorrect usage.

To give better advice how to fix your situation you need to describe
your backup solution - how exactly you select/create snapshots.

I hope, instead, there is some knowledge I'm missing, that
> when learned, will make this a robust backup solution.
> 
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs Issues

2017-09-17 Thread Satoru Takeuchi 
At Wed, 13 Sep 2017 11:53:35 -0400,
Ruoxin Jiang wrote:
> 
> [1  ]
> Hello,
> 
> We are researchers from Columbia University, New York. As part of our
> current research we have found some semantic discrepancies between
> btrfs and other popular filesystems.
> 
> We have attached two cases. The first one involves an invalid O_DIRECT
> write() that fails back to buffered write instead of failing with
> error EINVAL. In directory 2, we discovered that btrfs calculates
> write_bytes in __btrfs_buffered_write differently from that in
> generic_perform_writes in fs/mmap.c. This can cause inconsistent
> behavior between btrfs and other filesystems when program invokes the
> same writev/write() syscall.
> 
> In each directory, you will find a Readme.md describing the issue and
> pointing to the code that may cause the problem. For your convenience,
> we also included test programs (min.cpp) and instructions in Readme to
> help reproduce the issues.
> 
> We would appreciate very much if you could confirm the two cases at
> your conveniences.

I took a look at your test programs, btrfs_issues/{1,2}/min.cpp. It looks
very hard to read since you call syscalls in odd ways and all flags are
hardcoded as literal hexadecimal numbers. Could rewrite these program
to improve readability?

In addition, I have two questions about btrfs_issues/1/min.cpp.

1. Why you set 'filename' as the 1st argument of mmap()?
2. What's the purpose of mmap() call? I guess mmap() is not related to issue 1.

Thanks,
Satoru

> 
> Thanks,
> Amy
> [2 btrfs_issues.tar.gz ]
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

ERROR: parent determination failed (btrfs send-receive)

2017-09-17 Thread Dave 
Sometimes when using btrfs send-receive, I get errors like this:

ERROR: parent determination failed for 

When this happens, btrfs send-receive backups fail. And all subsequent
backups fail too.

The issue seems to stem from the fact that an automated cleanup
process removes certain earlier subvolumes. (I'm using Snapper.)

I'd like to understand exactly what is happening so that my backups do
not unexpectedly fail.

In my scenario, no parent subvolumes have been deleted from the
target. Some subvolumes have been deleted from the source, but why
does that matter? I am able to take a valid snapshot at this time and
every snapshot ever taken continues to reside at the target backup
destination (seemingly meaning that a parent subvolume can be found at
the target).

This issue seems to make btrfs send-receive a very fragile backup
solution. I hope, instead, there is some knowledge I'm missing, that
when learned, will make this a robust backup solution.

Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to run balance successfully (No space left on device)?

2017-09-17 Thread Duncan 
Tomasz Chmielewski posted on Mon, 18 Sep 2017 00:02:46 +0900 as excerpted:

> I'm trying to run balance on a 4.13.2 kernel without much luck:
> 
> # time btrfs balance start -v /var/lib/lxd -dusage=5 -musage=5
> [works, but only 1 chunk balanced]

> # time btrfs balance start -v /var/lib/lxd -dusage=0 -musage=0
> [no chunks with 0 usage to balance]
> 
> 
> # time btrfs balance start -v /var/lib/lxd
> [...]
> ERROR: error during balancing '/var/lib/lxd': No space left on device

OK, that fails.  Let's see what your unallocated space looks like, 
below...

> # df -h /var/lib/lxd

FWIW, standard (aka util-linux) df is effectively useless in a situation 
such as this, as it really doesn't give you the information you need (it 
can say you have lots of space available, but if btrfs has all of it 
allocated into chunks, even if the chunks have space in them still, there 
can be problems).

And actually, (util-linux) df really doesn't give you a whole lot of 
useful information on a btrfs in enough cases that most list regulars 
tend to discount its output almost entirely.  The only thing it's really 
useful for is getting a reasonable idea as to whether your next major 
file operation can be expected to succeed or not -- if it says you have 
50 MB left and you're trying to put a new 1 GiB file on the btrfs, it's 
unlikely to work, but if it says you have 300 GiB left in a multi-TB 
multi-device filesystem, you might have 300, or 3000 (its estimates are 
deliberately on the pessimistic side).

For better numbers, always use the btrfs tools, btrfs fi usage is the one 
I tend to use most, but btrfs dev usage can be very useful if you're more 
interested in a per-device listing, and btrfs fi show combined with btrfs 
fi df provide much the same information, tho it needs a bit more 
interpreting.

But you do provide them too. =:^)

> # btrfs fi df /var/lib/lxd
> Data, RAID1: total=318.00GiB, used=313.82GiB
> System, RAID1: total=32.00MiB, used=80.00KiB
> Metadata, RAID1: total=5.00GiB, used=3.17GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

Looks reasonably healthy.  No global reserve used, good as that's a major 
indicator of problems, and data and metadata usage is reasonably close to 
totals -- no huge number of mostly empty allocated chunks.

> # btrfs fi show /var/lib/lxd Label: 'btrfs'  uuid:
> f5f30428-ec5b-4497-82de-6e20065e6f61
>  Total devices 2 FS bytes used 316.98GiB
>  devid1 size 423.13GiB used 323.03GiB path /dev/sda3
>  devid2 size 423.13GiB used 323.03GiB path /dev/sdb3

OK, given the ENOSPC error on balance above, those device lines are the 
real interesting numbers, and...

Healthy here too.  Very much so, in fact, as only 323 gigs out of 423 is 
allocated on each device -- 100 gigs not chunk-allocated and therefore 
free for chunk allocation on each device. =:^)

The ENOSPC is therefore a bug -- it shouldn't be happening.

And as it happens, AFAIK from reading the list, there's a currently known 
bug with over-reservation under certain circumstances that among other 
things, can (wrongly) trigger ENOSPC on balances, when there's plenty of 
space.

Also AFAIK, there's a patch on-list and (I think) in 4.14-rc1, that is I 
believe marked for stable as well, that will very likely fix your 
problem.  If it doesn't, there's another bug triggering similar symptoms.

But I'm not a dev and haven't been tracking the specific patch, so you'll 
need to either track it down (or wait to see if a dev or someone else 
points you at it) and apply it on your 4.13.x, or wait until it hits 
stable backports and you can get it there, or try 4.14-rc1 or wait until 
later/safer rcs or full release.

Meanwhile...

> # btrfs fi usage /var/lib/lxd Overall:
>  Device size: 846.25GiB
>  Device allocated:646.06GiB
>  Device unallocated:  200.19GiB
>  Device missing:  0.00B
>  Used:633.97GiB
>  Free (estimated):104.28GiB  (min: 104.28GiB)
>  Data ratio:   2.00
>  Metadata ratio:   2.00
>  Global reserve:  512.00MiB  (used: 0.00B)
> 
> Data,RAID1: Size:318.00GiB, Used:313.82GiB
> /dev/sda3 318.00GiB
> /dev/sdb3 318.00GiB
> 
> Metadata,RAID1: Size:5.00GiB, Used:3.17GiB
> /dev/sda3   5.00GiB
> /dev/sdb3   5.00GiB
> 
> System,RAID1: Size:32.00MiB, Used:80.00KiB
> /dev/sda3  32.00MiB
> /dev/sdb3  32.00MiB
> 
> Unallocated:
> /dev/sda3 100.10GiB
> /dev/sdb3 100.10GiB

As I said above, btrfs fi usage output provides much of the same info, 
but in a much nicer format and with a bit more detail, than the 
combination of btrfs fi show and btrfs fi df.

This confirms the above 100 gigs per device unallocated, plenty for a 
balance if it's not bugging out, and data and metadata chunk usage in the 
same ball park as the totals, so as I said above, the

Re: [PATCH v2] btrfs-progs: Output time elapsed for each major tree it checked

2017-09-17 Thread Satoru Takeuchi 
At Mon, 18 Sep 2017 09:47:29 +0900,
Qu Wenruo wrote:
> 
> Marc reported that "btrfs check --repair" runs much faster than "btrfs
> check", which is quite weird.
> 
> This patch will add time elapsed for each major tree it checked, for
> both original mode and lowmem mode, so we can have a clue what's going
> wrong.
> 
> Reported-by: Marc MERLIN 
> Signed-off-by: Qu Wenruo 
> ---
> v2:
>   Add prefix for each time report, as error message may make the output
>   hard to understand. Suggested by Satoru.

Reviewed-by: Satoru Takeuchi 

> ---
>  cmds-check.c | 21 +++--
>  utils.h  | 24 
>  2 files changed, 43 insertions(+), 2 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index 006edbde..f1074c73 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -5318,13 +5318,16 @@ static int do_check_fs_roots(struct btrfs_fs_info 
> *fs_info,
> struct cache_tree *root_cache)
>  {
>   int ret;
> + struct timer timer;
>  
>   if (!ctx.progress_enabled)
>   fprintf(stderr, "checking fs roots\n");
> + start_timer();
>   if (check_mode == CHECK_MODE_LOWMEM)
>   ret = check_fs_roots_v2(fs_info);
>   else
>   ret = check_fs_roots(fs_info, root_cache);
> + printf("fs roots checked in %d seconds\n", stop_timer());
>  
>   return ret;
>  }
> @@ -11584,14 +11587,16 @@ out:
>  static int do_check_chunks_and_extents(struct btrfs_fs_info *fs_info)
>  {
>   int ret;
> + struct timer timer;
>  
>   if (!ctx.progress_enabled)
>   fprintf(stderr, "checking extents\n");
> + start_timer();
>   if (check_mode == CHECK_MODE_LOWMEM)
>   ret = check_chunks_and_extents_v2(fs_info);
>   else
>   ret = check_chunks_and_extents(fs_info);
> -
> + printf("extents checked in %d seconds\n", stop_timer());
>   return ret;
>  }
>  
> @@ -12772,6 +12777,7 @@ int cmd_check(int argc, char **argv)
>   int qgroups_repaired = 0;
>   unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
>   int force = 0;
> + struct timer timer;
>  
>   while(1) {
>   int c;
> @@ -12953,8 +12959,11 @@ int cmd_check(int argc, char **argv)
>   if (repair)
>   ctree_flags |= OPEN_CTREE_PARTIAL;
>  
> + printf("opening btrfs filesystem\n");
> + start_timer();
>   info = open_ctree_fs_info(argv[optind], bytenr, tree_root_bytenr,
> chunk_root_bytenr, ctree_flags);
> + printf("btrfs filesystem opened in %d seconds\n", stop_timer());
>   if (!info) {
>   error("cannot open file system");
>   ret = -EIO;
> @@ -13115,8 +13124,10 @@ int cmd_check(int argc, char **argv)
>   else
>   fprintf(stderr, "checking free space cache\n");
>   }
> + start_timer();
>   ret = check_space_cache(root);
>   err |= !!ret;
> + printf("free space cache (tree) checked in %d seconds\n", 
> stop_timer());
>   if (ret) {
>   if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
>   error("errors found in free space tree");
> @@ -13140,18 +13151,22 @@ int cmd_check(int argc, char **argv)
>   }
>  
>   fprintf(stderr, "checking csums\n");
> + start_timer();
>   ret = check_csums(root);
>   err |= !!ret;
> + printf("csums tree checked in %d seconds\n", stop_timer());
>   if (ret) {
>   error("errors found in csum tree");
>   goto out;
>   }
>  
> - fprintf(stderr, "checking root refs\n");
>   /* For low memory mode, check_fs_roots_v2 handles root refs */
>   if (check_mode != CHECK_MODE_LOWMEM) {
> + fprintf(stderr, "checking root refs\n");
> + start_timer();
>   ret = check_root_refs(root, _cache);
>   err |= !!ret;
> + printf("root refs checked in %d seconds\n", stop_timer());
>   if (ret) {
>   error("errors found in root refs");
>   goto out;
> @@ -13186,8 +13201,10 @@ int cmd_check(int argc, char **argv)
>  
>   if (info->quota_enabled) {
>   fprintf(stderr, "checking quota groups\n");
> + start_timer();
>   ret = qgroup_verify_all(info);
>   err |= !!ret;
> + printf("quota groups checked in %d seconds\n", 
> stop_timer());
>   if (ret) {
>   error("failed to check quota groups");
>   goto out;
> diff --git a/utils.h b/utils.h
> index d28a05a6..159487db 100644
> --- a/utils.h
> +++ b/utils.h
> @@ -172,4 +172,28 @@ u64 rand_u64(void);
>  unsigned int rand_range(unsigned int upper);
>  void init_rand_seed(u64 seed);
>  
> +/* Utils to report time duration */
> +struct timer {
> + time_t start;
> +};
> +
> +static inline void

[PATCH v2] btrfs-progs: Output time elapsed for each major tree it checked

2017-09-17 Thread Qu Wenruo 
Marc reported that "btrfs check --repair" runs much faster than "btrfs
check", which is quite weird.

This patch will add time elapsed for each major tree it checked, for
both original mode and lowmem mode, so we can have a clue what's going
wrong.

Reported-by: Marc MERLIN 
Signed-off-by: Qu Wenruo 
---
v2:
  Add prefix for each time report, as error message may make the output
  hard to understand. Suggested by Satoru.
---
 cmds-check.c | 21 +++--
 utils.h  | 24 
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 006edbde..f1074c73 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5318,13 +5318,16 @@ static int do_check_fs_roots(struct btrfs_fs_info 
*fs_info,
  struct cache_tree *root_cache)
 {
int ret;
+   struct timer timer;
 
if (!ctx.progress_enabled)
fprintf(stderr, "checking fs roots\n");
+   start_timer();
if (check_mode == CHECK_MODE_LOWMEM)
ret = check_fs_roots_v2(fs_info);
else
ret = check_fs_roots(fs_info, root_cache);
+   printf("fs roots checked in %d seconds\n", stop_timer());
 
return ret;
 }
@@ -11584,14 +11587,16 @@ out:
 static int do_check_chunks_and_extents(struct btrfs_fs_info *fs_info)
 {
int ret;
+   struct timer timer;
 
if (!ctx.progress_enabled)
fprintf(stderr, "checking extents\n");
+   start_timer();
if (check_mode == CHECK_MODE_LOWMEM)
ret = check_chunks_and_extents_v2(fs_info);
else
ret = check_chunks_and_extents(fs_info);
-
+   printf("extents checked in %d seconds\n", stop_timer());
return ret;
 }
 
@@ -12772,6 +12777,7 @@ int cmd_check(int argc, char **argv)
int qgroups_repaired = 0;
unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
int force = 0;
+   struct timer timer;
 
while(1) {
int c;
@@ -12953,8 +12959,11 @@ int cmd_check(int argc, char **argv)
if (repair)
ctree_flags |= OPEN_CTREE_PARTIAL;
 
+   printf("opening btrfs filesystem\n");
+   start_timer();
info = open_ctree_fs_info(argv[optind], bytenr, tree_root_bytenr,
  chunk_root_bytenr, ctree_flags);
+   printf("btrfs filesystem opened in %d seconds\n", stop_timer());
if (!info) {
error("cannot open file system");
ret = -EIO;
@@ -13115,8 +13124,10 @@ int cmd_check(int argc, char **argv)
else
fprintf(stderr, "checking free space cache\n");
}
+   start_timer();
ret = check_space_cache(root);
err |= !!ret;
+   printf("free space cache (tree) checked in %d seconds\n", 
stop_timer());
if (ret) {
if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
error("errors found in free space tree");
@@ -13140,18 +13151,22 @@ int cmd_check(int argc, char **argv)
}
 
fprintf(stderr, "checking csums\n");
+   start_timer();
ret = check_csums(root);
err |= !!ret;
+   printf("csums tree checked in %d seconds\n", stop_timer());
if (ret) {
error("errors found in csum tree");
goto out;
}
 
-   fprintf(stderr, "checking root refs\n");
/* For low memory mode, check_fs_roots_v2 handles root refs */
if (check_mode != CHECK_MODE_LOWMEM) {
+   fprintf(stderr, "checking root refs\n");
+   start_timer();
ret = check_root_refs(root, _cache);
err |= !!ret;
+   printf("root refs checked in %d seconds\n", stop_timer());
if (ret) {
error("errors found in root refs");
goto out;
@@ -13186,8 +13201,10 @@ int cmd_check(int argc, char **argv)
 
if (info->quota_enabled) {
fprintf(stderr, "checking quota groups\n");
+   start_timer();
ret = qgroup_verify_all(info);
err |= !!ret;
+   printf("quota groups checked in %d seconds\n", 
stop_timer());
if (ret) {
error("failed to check quota groups");
goto out;
diff --git a/utils.h b/utils.h
index d28a05a6..159487db 100644
--- a/utils.h
+++ b/utils.h
@@ -172,4 +172,28 @@ u64 rand_u64(void);
 unsigned int rand_range(unsigned int upper);
 void init_rand_seed(u64 seed);
 
+/* Utils to report time duration */
+struct timer {
+   time_t start;
+};
+
+static inline void start_timer(struct timer *t)
+{
+   time(>start);
+}
+
+/*
+ * Stop timer and return the time elapsed in int
+ *
+ * int should be large enough to "btrfs check" and avoid
+ * type converting.
+ */
+static inline int stop_timer(struct timer *t)
+{
+

[PATCH v2] btrfs-progs: allow "none" to disable compression for convenience

2017-09-17 Thread Satoru Takeuchi 
At Sun, 17 Sep 2017 14:08:40 +0100,
Mike Fleetwood wrote:
> 
> On 17 September 2017 at 01:36, Satoru Takeuchi
>  wrote:
> > It's messy to use "" to disable compression. Introduce the new value "no"
> > which can also be used for this purpose.
> 
> From an English language point of view, "none" would be better.  None
> says the absence of, where as no is more general negative.

Thank you for your comment. How about is it?

---
It's messy to use "" to disable compression. Introduce the new value "none"
which can also be used for this purpose.

Signed-off-by: Satoru Takeuchi 
---
 Documentation/btrfs-property.asciidoc | 2 +-
 props.c   | 6 --
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/Documentation/btrfs-property.asciidoc 
b/Documentation/btrfs-property.asciidoc
index 7ed6a7d..786af9b 100644
--- a/Documentation/btrfs-property.asciidoc
+++ b/Documentation/btrfs-property.asciidoc
@@ -43,7 +43,7 @@ read-only flag of subvolume: true or false
 label
 label of device
 compression
-compression setting for an inode: lzo, zlib, zstd, or "" (empty string)
+compression setting for an inode: lzo, zlib, zstd, none, or "" (empty string). 
Both none and "" are for disabling compression.
 
 *list* [-t ] ::
 Lists available properties with their descriptions for the given object.
diff --git a/props.c b/props.c
index a7e3e96..8d85181 100644
--- a/props.c
+++ b/props.c
@@ -142,9 +142,11 @@ static int prop_compression(enum prop_object_type type,
memcpy(xattr_name + XATTR_BTRFS_PREFIX_LEN, name, strlen(name));
xattr_name[XATTR_BTRFS_PREFIX_LEN + strlen(name)] = '\0';
 
-   if (value)
+   if (value) {
+   if (!strcmp(value, "none"))
+   value = "";
sret = fsetxattr(fd, xattr_name, value, strlen(value), 0);
-   else
+   } else
sret = fgetxattr(fd, xattr_name, NULL, 0);
if (sret < 0) {
ret = -errno;
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/2] Add required IETF Trust copyright

2017-09-17 Thread Nicholas D Steeves 
Signed-off-by: Nicholas D Steeves 
---
 tests/sha-private.h | 4 
 1 file changed, 4 insertions(+)

diff --git a/tests/sha-private.h b/tests/sha-private.h
index 6e9c4520..d94d4374 100644
--- a/tests/sha-private.h
+++ b/tests/sha-private.h
@@ -1,5 +1,9 @@
 / sha-private.h /
 /* See RFC 6234 for details. ***/
+/* Copyright (c) 2011 IETF Trust and the persons identified as */
+/* authors of the code.  All rights reserved.  */
+/* See sha.h for terms of use and redistribution.  */
+
 #ifndef _SHA_PRIVATE__H
 #define _SHA_PRIVATE__H
 /*
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 0/2] btrfs-progs: tests/sha* copyright clarity and compliance fixes

2017-09-17 Thread Nicholas D Steeves 
Hi,

Thank you very much for fixing up the tests!  A couple of releases ago
I noticed that they were reliably passing, but I had to exclude the
tests from my uploads because the BCP 78 license is explicitely banned
from Debian.  I plan to study Debian's autopkgtest framework and then
configure the package to automatically run the tests in a VM on one of
the buildservers for every version update.

The first patch adds what I believe is a necessary header for license
compliance, and the second removes the misleading boilerplate.  Please
consult the annotated commit messages for more information.

Thank you,
Nicholas

Nicholas D Steeves (2):
  Add required IETF Trust copyright
  Remove misleading BCP 78 boilerplate

 tests/sha-private.h |  4 
 tests/sha224-256.c  | 20 
 2 files changed, 4 insertions(+), 20 deletions(-)

-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2] Remove misleading BCP 78 boilerplate

2017-09-17 Thread Nicholas D Steeves 
BCP 78 applies to RFC 6234, but sha224-256.c is Simplified BSD.

This causes the following lintian error when building on Debian and
Debian derivatives:

E: btrfs-progs source: license-problem-non-free-RFC-BCP78
   tests/sha224-256.c

Please consult the following email from debian-le...@lists.debian.org
for more information:

https://lists.debian.org/debian-legal/2017/08/msg4.html
Signed-off-by: Nicholas D Steeves 
---
 tests/sha224-256.c | 20 
 1 file changed, 20 deletions(-)

diff --git a/tests/sha224-256.c b/tests/sha224-256.c
index 2d963e65..82124a03 100644
--- a/tests/sha224-256.c
+++ b/tests/sha224-256.c
@@ -1,23 +1,3 @@
-/*
-RFC 6234SHAs, HMAC-SHAs, and HKDF   May 2011
-
-
-Copyright Notice
-
-   Copyright (c) 2011 IETF Trust and the persons identified as the
-   document authors.  All rights reserved.
-
-   This document is subject to BCP 78 and the IETF Trust's Legal
-   Provisions Relating to IETF Documents
-   (http://trustee.ietf.org/license-info) in effect on the date of
-   publication of this document.  Please review these documents
-   carefully, as they describe your rights and restrictions with respect
-   to this document.  Code Components extracted from this document must
-   include Simplified BSD License text as described in Section 4.e of
-   the Trust Legal Provisions and are provided without warranty as
-   described in the Simplified BSD License.
-*/
-
 /* sha224-256.c /
 /* See RFC 6234 for details. ***/
 /* Copyright (c) 2011 IETF Trust and the persons identified as */
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cp --reflink and qgroup limit

2017-09-17 Thread Qu Wenruo 



On 2017年09月17日 22:45, Antoine Belvire wrote:

Hello,

The behavior of 'cp --reflink' puzzles me from the quota point of view.

It looks like it's possible to easily exceeds a qgroup's max_rfer limit 
like this:


~# # Create '/test' and '/test/a' subvolumes
~# btrfs subvolume create /test
~# btrfs subvolume create /test/a
~#
~# # Set limits
~# btrfs qgroup limit 10m (qgroupid) /
~# btrfs qgroup limit -e 5m (qgroupid) /
~#
~# # Limits are set
~# btrfs qgroup show -erF /test/a/
qgroupid rfer excl max_rfer max_excl
    
(the id) 16.00KiB 16.00KiB 10.00MiB  5.00MiB
~#
~# # Write a big file in '/test'
~# dd if=/dev/urandom of=/test/bigfile bs=1M count=50
~#
~# # Copy it to '/test/a' using cp --reflink
~# cp --reflink /test/bigfile /test/a
~#
~# # No EDQUOT error, quota limit is exceeded
~# btrfs qgroup show --sync -erF /test/a/
qgroupid rfer excl max_rfer max_excl
    
(the id) 50.02MiB 16.00KiB 10.00MiB  5.00MiB
~#

However, now the limit is exceeded, EDQUOT is raised upon a new 'cp 
--reflink':


~# touch /test/nothing
~# cp --reflink /test/nothing /test/a
cp: cannot create regular file 'a/nothing': Disk quota exceeded
~#

As far as I understand the code, btrfs_clone doesn't try to reserve data 
it clones. It only reserves metadata through btrfs_start_transaction.


That's a bug, and should be fixed.

But it's a little complicated to fix.
The biggest problem is, until reflink is done and we commit a 
transaction, we don't know if the the reflink operation will increase 
"rfer" or not.
(Considering a case where both reflink source and destination is inside 
the same subvolume)


Yes, we can always try to reserve the size of the data, and cause false 
alert, but the handling of reserved data range will be another problem.


I'll check if we can solve it.

Thanks,
Qu


Is it an expected behavior?

Regards,

--
Antoine
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 1/1] btrfs: Add mechanism to configure automatic level-0 qgroup removal

2017-09-17 Thread Sargun Dhillon 
This patch introduces a persisted sysfs knob - qgroup_autoremove.
The purpose of this knob is to cause btrfs (kernel) to automatically
remove level-0 qgroups on subvolume removal. It does not try to
traverse the qgroup tree, and delete other dangling qgroups.

The knob is  disabled by default to avoid breaking userspace.
Once the knob is enabled, it is persisted across remounts in
qgroup_flags.

Signed-off-by: Sargun Dhillon 
---
 fs/btrfs/ctree.h|  2 +-
 fs/btrfs/ioctl.c| 14 +++
 fs/btrfs/sysfs.c| 91 -
 include/uapi/linux/btrfs_tree.h |  7 
 4 files changed, 103 insertions(+), 11 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 9ded3e9154a5..65f8fa246a4d 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1018,7 +1018,7 @@ struct btrfs_fs_info {
 #ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
 #endif
-   /* is qgroup tracking in a consistent state? */
+   /* qgroup configuration; is qgroup tracking in a consistent state? */
u64 qgroup_flags;
 
/* holds configuration and tracking. Protected by qgroup_lock */
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 94934901d58c..d7ef13720374 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -2315,6 +2315,7 @@ static noinline int btrfs_ioctl_snap_destroy(struct file 
*file,
struct btrfs_ioctl_vol_args *vol_args;
struct btrfs_trans_handle *trans;
struct btrfs_block_rsv block_rsv;
+   bool remove_qgroup = false;
u64 root_flags;
u64 qgroup_reserved;
int namelen;
@@ -2497,6 +2498,19 @@ static noinline int btrfs_ioctl_snap_destroy(struct file 
*file,
}
}
 
+
+   spin_lock(_info->qgroup_lock);
+   if (fs_info->qgroup_flags & BTRFS_QGROUP_AUTOREMOVE_FLAG)
+   remove_qgroup = true;
+   spin_unlock(_info->qgroup_lock);
+   if (remove_qgroup) {
+   ret = btrfs_remove_qgroup(trans, fs_info,
+ dest->root_key.objectid);
+   if (ret && ret != -ENOENT)
+   btrfs_warn(fs_info,
+  "Failed to cleanup qgroup. err: %d", ret);
+   }
+
 out_end_trans:
trans->block_rsv = NULL;
trans->bytes_reserved = 0;
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index c2d5f3580b4c..9604778038fd 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -90,6 +90,15 @@ static int can_modify_feature(struct btrfs_feature_attr *fa)
return val;
 }
 
+static void set_pending_commit(struct btrfs_fs_info *fs_info)
+{
+   /*
+* We don't want to do full transaction commit from inside sysfs
+*/
+   btrfs_set_pending(fs_info, COMMIT);
+   wake_up_process(fs_info->transaction_kthread);
+}
+
 static ssize_t btrfs_feature_attr_show(struct kobject *kobj,
   struct kobj_attribute *a, char *buf)
 {
@@ -165,11 +174,7 @@ static ssize_t btrfs_feature_attr_store(struct kobject 
*kobj,
set_features(fs_info, fa->feature_set, features);
spin_unlock(_info->super_lock);
 
-   /*
-* We don't want to do full transaction commit from inside sysfs
-*/
-   btrfs_set_pending(fs_info, COMMIT);
-   wake_up_process(fs_info->transaction_kthread);
+   set_pending_commit(fs_info);
 
return count;
 }
@@ -405,11 +410,7 @@ static ssize_t btrfs_label_store(struct kobject *kobj,
memcpy(fs_info->super_copy->label, buf, p_len);
spin_unlock(_info->super_lock);
 
-   /*
-* We don't want to do full transaction commit from inside sysfs
-*/
-   btrfs_set_pending(fs_info, COMMIT);
-   wake_up_process(fs_info->transaction_kthread);
+   set_pending_commit(fs_info);
 
return len;
 }
@@ -487,12 +488,82 @@ static ssize_t quota_override_store(struct kobject *kobj,
 
 BTRFS_ATTR_RW(quota_override, quota_override_show, quota_override_store);
 
+static ssize_t qgroup_autoremove_show(struct kobject *kobj,
+ struct kobj_attribute *a, char *buf)
+{
+   struct btrfs_fs_info *fs_info = to_fs_info(kobj);
+   int qgroup_autoremove = 0;
+
+   mutex_lock(_info->qgroup_ioctl_lock);
+   /* Check if qgroups are enabled */
+   if (!test_bit(BTRFS_FS_QUOTA_ENABLED, _info->flags))
+   goto out;
+   if (!fs_info->quota_root)
+   goto out;
+
+   if (fs_info->qgroup_flags & BTRFS_QGROUP_AUTOREMOVE_FLAG)
+   qgroup_autoremove = 1;
+
+out:
+   mutex_unlock(_info->qgroup_ioctl_lock);
+
+   return snprintf(buf, PAGE_SIZE, "%d\n", qgroup_autoremove);
+}
+
+static ssize_t qgroup_autoremove_store(struct kobject *kobj,
+  struct kobj_attribute *a,
+  const char *buf, size_t len)
+{
+

[PATCH v2 0/1] Add qgroup_autoremove flag

2017-09-17 Thread Sargun Dhillon 
This patch makes it so level-0 qgroups are automatically deleted. The
flag that enables this behaviour is persisted in btrfs_qgroup_status_item.
Although, it may make sense to introduce a btrfs_qgroup_configuration_item,
it seems somewhat overkill to do that just for one simple knob that can
fit into the existing flags.

The reason why this is a sysfs knob, and not a mount option is because
qgroups have to be enabled after the first mount. Having to do a mount,
enable qgroups, flip the knob, and then remounting seems somewhat more
awkward than this single sysfs knob.

Changes since v1:
 * Move from mount option to sysfs knob
 * Persist the knob across remounts

Sargun Dhillon (1):
  btrfs: Add mechanism to configure automatic level-0 qgroup removal

 fs/btrfs/ctree.h|  2 +-
 fs/btrfs/ioctl.c| 14 +++
 fs/btrfs/sysfs.c| 91 -
 include/uapi/linux/btrfs_tree.h |  7 
 4 files changed, 103 insertions(+), 11 deletions(-)

-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs-progs: load_chunk_info() wrong set search key 'type'

2017-09-17 Thread Goffredo Baroncelli 
The function load_chunk_info() doesn't initialize correctly the
sk->min/max_type when it calls the TREE_SEARCH ioctl: these keys are swapped.
Moreover this function assumes that all the items contained in the tree
BTRFS_CHUNK_TREE_OBJECTID are of type BTRFS_CHUNK_ITEM_KEY, however some
items are of type DEV_ITEM. Add an if( ) to pick only the right ones.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] load_chunk_info() : wrong search key 'type'

2017-09-17 Thread Goffredo Baroncelli 
From: Goffredo Baroncelli 

The function load_chunk_info() doesn't initialize correctly the
sk->min/max_type when it calls the TREE_SEARCH ioctl: these keys are swapped.
Moreover this function assumes that all the items contained in the tree
BTRFS_CHUNK_TREE_OBJECTID are of type BTRFS_CHUNK_ITEM_KEY, however some
items are of type DEV_ITEM. Add an if( ) to pick only the right ones.

Signed-off-by: G.Baroncelli 
---
 cmds-fi-usage.c | 21 +++--
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/cmds-fi-usage.c b/cmds-fi-usage.c
index 6c846c15..52d63524 100644
--- a/cmds-fi-usage.c
+++ b/cmds-fi-usage.c
@@ -147,8 +147,8 @@ static int load_chunk_info(int fd, struct chunk_info 
**info_ptr, int *info_count
 
sk->min_objectid = 0;
sk->max_objectid = (u64)-1;
-   sk->max_type = 0;
-   sk->min_type = (u8)-1;
+   sk->min_type = 0;
+   sk->max_type = (u8)-1;
sk->min_offset = 0;
sk->max_offset = (u64)-1;
sk->min_transid = 0;
@@ -174,20 +174,21 @@ static int load_chunk_info(int fd, struct chunk_info 
**info_ptr, int *info_count
off = 0;
for (i = 0; i < sk->nr_items; i++) {
struct btrfs_chunk *item;
+   u8 type;
sh = (struct btrfs_ioctl_search_header *)(args.buf +
  off);
-
+   type = btrfs_search_header_type(sh);
off += sizeof(*sh);
-   item = (struct btrfs_chunk *)(args.buf + off);
-
-   ret = add_info_to_list(info_ptr, info_count, item);
-   if (ret) {
-   *info_ptr = NULL;
-   return 1;
+   if (type == BTRFS_CHUNK_ITEM_KEY) {
+   item = (struct btrfs_chunk *)(args.buf + off);
+   ret = add_info_to_list(info_ptr, info_count, 
item);
+   if (ret) {
+   *info_ptr = NULL;
+   return 1;
+   }
}
 
off += btrfs_search_header_len(sh);
-
sk->min_objectid = btrfs_search_header_objectid(sh);
sk->min_type = btrfs_search_header_type(sh);
sk->min_offset = btrfs_search_header_offset(sh)+1;
-- 
2.14.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

cp --reflink and qgroup limit

2017-09-17 Thread Antoine Belvire 

Hello,

The behavior of 'cp --reflink' puzzles me from the quota point of view.

It looks like it's possible to easily exceeds a qgroup's max_rfer limit 
like this:


~# # Create '/test' and '/test/a' subvolumes
~# btrfs subvolume create /test
~# btrfs subvolume create /test/a
~#
~# # Set limits
~# btrfs qgroup limit 10m (qgroupid) /
~# btrfs qgroup limit -e 5m (qgroupid) /
~#
~# # Limits are set
~# btrfs qgroup show -erF /test/a/
qgroupid rfer excl max_rfer max_excl
    
(the id) 16.00KiB 16.00KiB 10.00MiB  5.00MiB
~#
~# # Write a big file in '/test'
~# dd if=/dev/urandom of=/test/bigfile bs=1M count=50
~#
~# # Copy it to '/test/a' using cp --reflink
~# cp --reflink /test/bigfile /test/a
~#
~# # No EDQUOT error, quota limit is exceeded
~# btrfs qgroup show --sync -erF /test/a/
qgroupid rfer excl max_rfer max_excl
    
(the id) 50.02MiB 16.00KiB 10.00MiB  5.00MiB
~#

However, now the limit is exceeded, EDQUOT is raised upon a new 'cp 
--reflink':


~# touch /test/nothing
~# cp --reflink /test/nothing /test/a
cp: cannot create regular file 'a/nothing': Disk quota exceeded
~#

As far as I understand the code, btrfs_clone doesn't try to reserve data 
it clones. It only reserves metadata through btrfs_start_transaction.


Is it an expected behavior?

Regards,

--
Antoine
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

how to run balance successfully (No space left on device)?

2017-09-17 Thread Tomasz Chmielewski 

I'm trying to run balance on a 4.13.2 kernel without much luck:

# time btrfs balance start -v /var/lib/lxd -dusage=5 -musage=5
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
  METADATA (flags 0x2): balancing, usage=5
  SYSTEM (flags 0x2): balancing, usage=5
Done, had to relocate 1 out of 353 chunks

real0m2.356s
user0m0.005s
sys 0m0.175s


# time btrfs balance start -v /var/lib/lxd -dusage=0 -musage=0
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
  METADATA (flags 0x2): balancing, usage=0
  SYSTEM (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 353 chunks

real0m0.076s
user0m0.004s
sys 0m0.008s


# time btrfs balance start -v /var/lib/lxd
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x0): balancing
  METADATA (flags 0x0): balancing
  SYSTEM (flags 0x0): balancing
WARNING:

Full balance without filters requested. This operation is very
intense and takes potentially very long. It is recommended to
use the balance filters to narrow down the balanced data.
Use 'btrfs balance start --full-balance' option to skip this
warning. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/var/lib/lxd': No space left on device
There may be more info in syslog - try dmesg | tail

real284m58.541s
user0m0.000s
sys 47m39.037s




# df -h /var/lib/lxd
Filesystem  Size  Used Avail Use% Mounted on
/dev/sda3   424G  318G  105G  76% /var/lib/lxd


# btrfs fi df /var/lib/lxd
Data, RAID1: total=318.00GiB, used=313.82GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=5.00GiB, used=3.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi show /var/lib/lxd
Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
Total devices 2 FS bytes used 316.98GiB
devid1 size 423.13GiB used 323.03GiB path /dev/sda3
devid2 size 423.13GiB used 323.03GiB path /dev/sdb3


# btrfs fi usage /var/lib/lxd
Overall:
Device size: 846.25GiB
Device allocated:646.06GiB
Device unallocated:  200.19GiB
Device missing:  0.00B
Used:633.97GiB
Free (estimated):104.28GiB  (min: 104.28GiB)
Data ratio:   2.00
Metadata ratio:   2.00
Global reserve:  512.00MiB  (used: 0.00B)

Data,RAID1: Size:318.00GiB, Used:313.82GiB
   /dev/sda3 318.00GiB
   /dev/sdb3 318.00GiB

Metadata,RAID1: Size:5.00GiB, Used:3.17GiB
   /dev/sda3   5.00GiB
   /dev/sdb3   5.00GiB

System,RAID1: Size:32.00MiB, Used:80.00KiB
   /dev/sda3  32.00MiB
   /dev/sdb3  32.00MiB

Unallocated:
   /dev/sda3 100.10GiB
   /dev/sdb3 100.10GiB


Mount flags in /etc/fstab are:

LABEL=btrfs /var/lib/lxd btrfs 
defaults,noatime,space_cache=v2,device=/dev/sda3,device=/dev/sdb3,discard 
0 0




Last pieces logged in dmesg:

[46867.225334] BTRFS info (device sda3): relocating block group 
2996254998528 flags data|raid1

[46874.563631] BTRFS info (device sda3): found 9250 extents
[46894.827895] BTRFS info (device sda3): found 9250 extents
[46898.463053] BTRFS info (device sda3): found 201 extents
[46898.562564] BTRFS info (device sda3): relocating block group 
2995181256704 flags data|raid1

[46903.555976] BTRFS info (device sda3): found 7299 extents
[46914.188044] BTRFS info (device sda3): found 7299 extents
[46914.303476] BTRFS info (device sda3): relocating block group 
2947936616448 flags metadata|raid1

[46939.570810] BTRFS info (device sda3): found 42022 extents
[46945.053488] BTRFS info (device sda3): 2 enospc errors during balance



Tomasz Chmielewski
https://lxadm.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: allow "no" to disable compression for convenience

2017-09-17 Thread Mike Fleetwood 
On 17 September 2017 at 01:36, Satoru Takeuchi
 wrote:
> It's messy to use "" to disable compression. Introduce the new value "no"
> which can also be used for this purpose.

>From an English language point of view, "none" would be better.  None
says the absence of, where as no is more general negative.

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: Output time elapsed for each major tree it checked

2017-09-17 Thread Qu Wenruo 



On 2017年09月17日 15:08, Satoru Takeuchi wrote:

At Mon, 11 Sep 2017 14:26:23 +0900,
Qu Wenruo wrote:


Marc reported that "btrfs check --repair" runs much faster than "btrfs
check", which is quite weird.

This patch will add time elapsed for each major tree it checked, for
both original mode and lowmem mode, so we can have a clue what's going
wrong.

Reported-by: Marc MERLIN 
Signed-off-by: Qu Wenruo 
---
  cmds-check.c | 21 +++--
  utils.h  | 24 
  2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 006edbde..fee806cd 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5318,13 +5318,16 @@ static int do_check_fs_roots(struct btrfs_fs_info 
*fs_info,
  struct cache_tree *root_cache)
  {
int ret;
+   struct timer timer;
  
  	if (!ctx.progress_enabled)

fprintf(stderr, "checking fs roots\n");
+   start_timer();
if (check_mode == CHECK_MODE_LOWMEM)
ret = check_fs_roots_v2(fs_info);
else
ret = check_fs_roots(fs_info, root_cache);
+   printf("done in %d seconds\n", stop_timer());


How about adding the prefixes to show which part of check/repair is done
before showing the elaplsed time, like "checking fs roots done in XX seconds"
and "checking extents done in XX seconds"?


Nice advice.

I'll update the patch to add a prefix for this.

Thanks,
Qu



Current message, "done in XX seconds" for all parts, is hard to understand
which part takes such the time, especially when something wrong happens
in some parts. For example, please see the following example of repairing
the image based on tests/fsck-tests/002-bad-transid/default_case.img

===
...
opening btrfs filesystem
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
Ignoring transid failure
done in 0 seconds
...
checking free space cache
cache and super generation don't match, space cache will be invalidated
done in 0 seconds
...
===

Thanks,
Satoru

  
  	return ret;

  }
@@ -11584,14 +11587,16 @@ out:
  static int do_check_chunks_and_extents(struct btrfs_fs_info *fs_info)
  {
int ret;
+   struct timer timer;
  
  	if (!ctx.progress_enabled)

fprintf(stderr, "checking extents\n");
+   start_timer();
if (check_mode == CHECK_MODE_LOWMEM)
ret = check_chunks_and_extents_v2(fs_info);
else
ret = check_chunks_and_extents(fs_info);
-
+   printf("done in %d seconds\n", stop_timer());
return ret;
  }
  
@@ -12772,6 +12777,7 @@ int cmd_check(int argc, char **argv)

int qgroups_repaired = 0;
unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
int force = 0;
+   struct timer timer;
  
  	while(1) {

int c;
@@ -12953,8 +12959,11 @@ int cmd_check(int argc, char **argv)
if (repair)
ctree_flags |= OPEN_CTREE_PARTIAL;
  
+	printf("opening btrfs filesystem\n");

+   start_timer();
info = open_ctree_fs_info(argv[optind], bytenr, tree_root_bytenr,
  chunk_root_bytenr, ctree_flags);
+   printf("done in %d seconds\n", stop_timer());
if (!info) {
error("cannot open file system");
ret = -EIO;
@@ -13115,8 +13124,10 @@ int cmd_check(int argc, char **argv)
else
fprintf(stderr, "checking free space cache\n");
}
+   start_timer();
ret = check_space_cache(root);
err |= !!ret;
+   printf("done in %d seconds\n", stop_timer());
if (ret) {
if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
error("errors found in free space tree");
@@ -13140,18 +13151,22 @@ int cmd_check(int argc, char **argv)
}
  
  	fprintf(stderr, "checking csums\n");

+   start_timer();
ret = check_csums(root);
err |= !!ret;
+   printf("done in %d seconds\n", stop_timer());
if (ret) {
error("errors found in csum tree");
goto out;
}
  
-	fprintf(stderr, "checking root refs\n");

/* For low memory mode, check_fs_roots_v2 handles root refs */
if (check_mode != CHECK_MODE_LOWMEM) {
+   fprintf(stderr, "checking root refs\n");
+   start_timer();
ret = check_root_refs(root, _cache);
err |= !!ret;
+   printf("done in %d seconds\n", stop_timer());
if (ret) {
error("errors found in root refs");
goto out;
@@ -13186,8 +13201,10 @@ int cmd_check(int argc, char **argv)
  
  	if (info->quota_enabled) {

Re: [PATCH] btrfs: Report error on removing qgroup if del_qgroup_item fails

2017-09-17 Thread Qu Wenruo 



On 2017年09月17日 17:02, Sargun Dhillon wrote:

Previously, we were calling del_qgroup_item, and ignoring the return code
resulting in a potential to have divergent in-memory state without an
error. Perhaps, it makes sense to handle this error code, and put the
filesystem into a read only, or similar state.

This patch only adds reporting of the error if the error is fatal,
(any error other than qgroup not found).

Signed-off-by: Sargun Dhillon 


Looks good to me.

Reviewed-by: Qu Wenruo 

Thanks,
Qu


---
  fs/btrfs/qgroup.c | 2 ++
  1 file changed, 2 insertions(+)

diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 770f667269f5..e172d4843eae 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1305,6 +1305,8 @@ int btrfs_remove_qgroup(struct btrfs_trans_handle *trans,
}
}
ret = del_qgroup_item(trans, quota_root, qgroupid);
+   if (ret && ret != -ENOENT)
+   goto out >
while (!list_empty(>groups)) {
list = list_first_entry(>groups,


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: Report error on removing qgroup if del_qgroup_item fails

2017-09-17 Thread Sargun Dhillon 
Previously, we were calling del_qgroup_item, and ignoring the return code
resulting in a potential to have divergent in-memory state without an
error. Perhaps, it makes sense to handle this error code, and put the
filesystem into a read only, or similar state.

This patch only adds reporting of the error if the error is fatal,
(any error other than qgroup not found).

Signed-off-by: Sargun Dhillon 
---
 fs/btrfs/qgroup.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 770f667269f5..e172d4843eae 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1305,6 +1305,8 @@ int btrfs_remove_qgroup(struct btrfs_trans_handle *trans,
}
}
ret = del_qgroup_item(trans, quota_root, qgroupid);
+   if (ret && ret != -ENOENT)
+   goto out;
 
while (!list_empty(>groups)) {
list = list_first_entry(>groups,
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: subvolume list as user?

2017-09-17 Thread Ulli Horlacher 
On Sat 2017-09-16 (22:56), Goffredo Baroncelli wrote:

>> how can a user get a list of his subvolumes? He may created them some
>> time ago and forget it. He now wants to have a list of them.
> 
> Far to be perfect, but this could help
> 
> find / -inum 256  2>/dev/null
> 
> each subvolume has inode number 256

But non-btrfs filesystems can also have an inode 256!
And it is terrible slow and inefficent.
Ok, I will add a sudo wrapper for my btrfs_subvolume_list :

#!/usr/bin/perl -w

$mounts = '/proc/mounts';
open $mounts,$mounts or die "$0: cannot open $mounts - $!\n";
while (<$mounts>) {
  if (m:^(/\S+) (/\S*) btrfs:) {
push @bfs,$2;
  }
}
close $mounts;
if (@bfs) {
  @bfs = sort @bfs;
} else {
  die "$0: no mounted btrfs filesystems\n";
}

@sv = ();
foreach my $fs (@bfs) {
  push @sv,"$fs\n";
  open $fs,"btrfs subvolume list '$fs'|" or next;;
  while (<$fs>) {
chomp;
s:.* \d+ path :$fs/:;
push @sv,"$_\n" if -d;
  }
  close $fs;
}

print sort @sv;


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
REF:<54b8bb4b-3807-15da-1db5-fab7b0ced...@libero.it>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: cleanup 'start' subtraction from try uncompressed inline extent

2017-09-17 Thread Satoru Takeuchi 
At Fri, 15 Sep 2017 01:57:26 +0300,
Timofey Titovets wrote:
> 
> Was added in:
>   c8b978188c9a0fd3d535c13debd19d522b726f1f
>   "Btrfs: Add zlib compression support"
> Survive to near time (from 08.10.2008).
> 
> Because 'start' checked for zero before branch, so it's
> safe to remove that subtraction.
> 
> Signed-off-by: Timofey Titovets 

Reviewed-by: Satoru Takeuchi 

> ---
>  fs/btrfs/inode.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 02ef32149c15..81123408e82e 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -570,7 +570,7 @@ static noinline void compress_file_range(struct inode 
> *inode,
>  cont:
>   if (start == 0) {
>   /* lets try to make an inline extent */
> - if (ret || total_in < (actual_end - start)) {
> + if (ret || total_in < actual_end) {
>   /* we didn't compress the entire range, try
>* to make an uncompressed inline extent.
>*/
> -- 
> 2.14.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: Output time elapsed for each major tree it checked

2017-09-17 Thread Satoru Takeuchi 
At Mon, 11 Sep 2017 14:26:23 +0900,
Qu Wenruo wrote:
> 
> Marc reported that "btrfs check --repair" runs much faster than "btrfs
> check", which is quite weird.
> 
> This patch will add time elapsed for each major tree it checked, for
> both original mode and lowmem mode, so we can have a clue what's going
> wrong.
> 
> Reported-by: Marc MERLIN 
> Signed-off-by: Qu Wenruo 
> ---
>  cmds-check.c | 21 +++--
>  utils.h  | 24 
>  2 files changed, 43 insertions(+), 2 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index 006edbde..fee806cd 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -5318,13 +5318,16 @@ static int do_check_fs_roots(struct btrfs_fs_info 
> *fs_info,
> struct cache_tree *root_cache)
>  {
>   int ret;
> + struct timer timer;
>  
>   if (!ctx.progress_enabled)
>   fprintf(stderr, "checking fs roots\n");
> + start_timer();
>   if (check_mode == CHECK_MODE_LOWMEM)
>   ret = check_fs_roots_v2(fs_info);
>   else
>   ret = check_fs_roots(fs_info, root_cache);
> + printf("done in %d seconds\n", stop_timer());

How about adding the prefixes to show which part of check/repair is done
before showing the elaplsed time, like "checking fs roots done in XX seconds"
and "checking extents done in XX seconds"?

Current message, "done in XX seconds" for all parts, is hard to understand
which part takes such the time, especially when something wrong happens
in some parts. For example, please see the following example of repairing
the image based on tests/fsck-tests/002-bad-transid/default_case.img

===
...
opening btrfs filesystem
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
Ignoring transid failure
done in 0 seconds
...
checking free space cache
cache and super generation don't match, space cache will be invalidated
done in 0 seconds
...
===

Thanks,
Satoru

>  
>   return ret;
>  }
> @@ -11584,14 +11587,16 @@ out:
>  static int do_check_chunks_and_extents(struct btrfs_fs_info *fs_info)
>  {
>   int ret;
> + struct timer timer;
>  
>   if (!ctx.progress_enabled)
>   fprintf(stderr, "checking extents\n");
> + start_timer();
>   if (check_mode == CHECK_MODE_LOWMEM)
>   ret = check_chunks_and_extents_v2(fs_info);
>   else
>   ret = check_chunks_and_extents(fs_info);
> -
> + printf("done in %d seconds\n", stop_timer());
>   return ret;
>  }
>  
> @@ -12772,6 +12777,7 @@ int cmd_check(int argc, char **argv)
>   int qgroups_repaired = 0;
>   unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
>   int force = 0;
> + struct timer timer;
>  
>   while(1) {
>   int c;
> @@ -12953,8 +12959,11 @@ int cmd_check(int argc, char **argv)
>   if (repair)
>   ctree_flags |= OPEN_CTREE_PARTIAL;
>  
> + printf("opening btrfs filesystem\n");
> + start_timer();
>   info = open_ctree_fs_info(argv[optind], bytenr, tree_root_bytenr,
> chunk_root_bytenr, ctree_flags);
> + printf("done in %d seconds\n", stop_timer());
>   if (!info) {
>   error("cannot open file system");
>   ret = -EIO;
> @@ -13115,8 +13124,10 @@ int cmd_check(int argc, char **argv)
>   else
>   fprintf(stderr, "checking free space cache\n");
>   }
> + start_timer();
>   ret = check_space_cache(root);
>   err |= !!ret;
> + printf("done in %d seconds\n", stop_timer());
>   if (ret) {
>   if (btrfs_fs_compat_ro(info, FREE_SPACE_TREE))
>   error("errors found in free space tree");
> @@ -13140,18 +13151,22 @@ int cmd_check(int argc, char **argv)
>   }
>  
>   fprintf(stderr, "checking csums\n");
> + start_timer();
>   ret = check_csums(root);
>   err |= !!ret;
> + printf("done in %d seconds\n", stop_timer());
>   if (ret) {
>   error("errors found in csum tree");
>   goto out;
>   }
>  
> - fprintf(stderr, "checking root refs\n");
>   /* For low memory mode, check_fs_roots_v2 handles root refs */
>   if (check_mode != CHECK_MODE_LOWMEM) {
> + fprintf(stderr, "checking root refs\n");
> + start_timer();
>   ret = check_root_refs(root, _cache);
>   err |= !!ret;
> + printf("done in %d seconds\n", stop_timer());
>   if (ret) {
>   error("errors found in root refs");
>   goto out;
> @@ -13186,8 +13201,10 @@ int cmd_check(int argc, char **argv)
>  
>   if (info->quota_enabled) {
>   fprintf(stderr, "checking