Re: Identifying reflink / CoW files

[Zygo Blaxell]

On Thu, Oct 27, 2016 at 01:30:11PM +0200, Saint Germain wrote:
> Hello,
> 
> Following the previous discussion:
> https://www.spinics.net/lists/linux-btrfs/msg19075.html
> 
> I would be interested in finding a way to reliably identify reflink /
> CoW files in order to use deduplication programs (like fdupes, jdupes,
> rmlint) efficiently.
> 
> Using FIEMAP doesn't seem to be reliable according to this discussion
> on rmlint:
> https://github.com/sahib/rmlint/issues/132#issuecomment-157665154

Inline extents have no physical address (FIEMAP returns 0 in that field).
You can't dedup them and each file can have only one, so if you see
the FIEMAP_EXTENT_INLINE bit set, you can just skip processing the entire
file immediately.

You can create a separate non-inline extent in a temporary file then
use dedup to replace _both_ copies of the original inline extent.
Or don't bother, as the savings are negligible.

> Is there another way that deduplication programs can easily use ?

The problem is that it's not files that are reflinked--individual extents
are.  "reflink file copy" really just means "a file whose extents are
100% shared with another file." It's possible for files on btrfs to have
any percentage of shared extents from 0 to 100% in increments of the
host page size.  It's also possible for the blocks to be shared with
different extent boundaries.

The quality of the result therefore depends on the amount of effort
put into measuring it.  If you look for the first non-hole extent in
each file and use its physical address as a physical file identifier,
then you get a fast reflink detector function that has a high risk of
false positives.  If you map out two files and compare physical addresses
block by block, you get a slow function with a low risk of false positives
(but maybe a small risk of false negatives too).

If your dedup program only does full-file reflink copies then the first
extent physical address method is sufficient.  If your program does
block- or extent-level dedup then it shouldn't be using files in its
data model at all, except where necessary to provide a mechanism to
access the physical blocks through the POSIX filesystem API.

FIEMAP will tell you about all the extents (physical address for extents
that have them, zero for other extent types).  It's also slow and has
assorted accuracy problems especially with compressed files.  Any user
can run FIEMAP, and it uses only standard structure arrays.

SEARCH_V2 is root-only and requires parsing variable-length binary
btrfs data encoding, but it's faster than FIEMAP and gives more accurate
results on compressed files.

> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


signature.asc
Description: Digital signature

[PATCH v4 3/4] btrfs-progs: receive: introduce option to dump send stream

[Qu Wenruo]

Introduce new option, '--dump' for receive subcommand.

With this command, user can dump the metadata of a send stream.
Which is quite useful for education purpose or bug reporting.

Signed-off-by: Qu Wenruo 
Signed-off-by: David Sterba 
---
 Documentation/btrfs-receive.asciidoc | 15 ++-
 cmds-receive.c   | 35 +++
 2 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/Documentation/btrfs-receive.asciidoc 
b/Documentation/btrfs-receive.asciidoc
index e246603..f12e949 100644
--- a/Documentation/btrfs-receive.asciidoc
+++ b/Documentation/btrfs-receive.asciidoc
@@ -9,12 +9,19 @@ SYNOPSIS
 
 *btrfs receive* [options] 
 
+or
+
+*btrfs receive* --dump [options]
+
 DESCRIPTION
 ---
 
 Receive a stream of changes and replicate one or more subvolumes that were
 previously used with *btrfs send* The received subvolumes are stored to
-'path'.
+'path', if '--dump' option is not given.
+
+If '--dump' option is given, *btrfs receive* will only do the validation of
+the stream, and print the stream metadata.
 
 *btrfs receive* will fail int the following cases:
 
@@ -56,6 +63,12 @@ By default the mountpoint is searched in '/proc/self/mounts'.
 If you do not have '/proc', eg. in a chroot environment, use this option to 
tell
 us where this filesystem is mounted.
 
+--dump::
+print the stream metadata
++
+Does not accept the 'path' parameter. So with this option, *btrfs receive* 
won't
+modify your filesystem, and can be run by non-privileged users.
+
 EXIT STATUS
 ---
 *btrfs receive* returns a zero exit status if it succeeds. Non zero is
diff --git a/cmds-receive.c b/cmds-receive.c
index d0525bf..1dcdb1a 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -49,6 +49,7 @@
 #include "send.h"
 #include "send-stream.h"
 #include "send-utils.h"
+#include "send-dump.h"
 
 static int g_verbose = 0;
 
@@ -1214,6 +1215,7 @@ int cmd_receive(int argc, char **argv)
struct btrfs_receive r;
int receive_fd = fileno(stdin);
u64 max_errors = 1;
+   int dump = 0;
int ret = 0;
 
memset(&r, 0, sizeof(r));
@@ -1226,9 +1228,11 @@ int cmd_receive(int argc, char **argv)
 
while (1) {
int c;
+   enum { GETOPT_VAL_DUMP = 257 };
static const struct option long_opts[] = {
{ "max-errors", required_argument, NULL, 'E' },
{ "chroot", no_argument, NULL, 'C' },
+   { "dump", no_argument, NULL, GETOPT_VAL_DUMP },
{ NULL, 0, NULL, 0 }
};
 
@@ -1265,6 +1269,9 @@ int cmd_receive(int argc, char **argv)
goto out;
}
break;
+   case GETOPT_VAL_DUMP:
+   dump = 1;
+   break;
case '?':
default:
error("receive args invalid");
@@ -1272,7 +1279,9 @@ int cmd_receive(int argc, char **argv)
}
}
 
-   if (check_argc_exact(argc - optind, 1))
+   if (dump && check_argc_exact(argc - optind, 0))
+   usage(cmd_receive_usage);
+   if (!dump && check_argc_exact(argc - optind, 1))
usage(cmd_receive_usage);
 
tomnt = argv[optind];
@@ -1285,17 +1294,33 @@ int cmd_receive(int argc, char **argv)
}
}
 
-   ret = do_receive(&r, tomnt, realmnt, receive_fd, max_errors);
+   if (dump) {
+   struct btrfs_dump_send_args dump_args;
+
+   dump_args.root_path[0] = '.';
+   dump_args.root_path[1] = '\0';
+   dump_args.full_subvol_path[0] = '.';
+   dump_args.full_subvol_path[1] = '\0';
+   ret = btrfs_read_and_process_send_stream(receive_fd,
+   &btrfs_print_send_ops, &dump_args, 0, 0);
+   if (ret < 0)
+   error("failed to dump the send stream: %s",
+ strerror(-ret));
+   } else {
+   ret = do_receive(&r, tomnt, realmnt, receive_fd, max_errors);
+   }
+
if (receive_fd != fileno(stdin))
close(receive_fd);
-
 out:
 
return !!ret;
 }
 
 const char * const cmd_receive_usage[] = {
-   "btrfs receive [-ve] [-f ] [--max-errors ] ",
+   "btrfs receive [options] ",
+   "or",
+   "btrfs receive --dump [options]",
"Receive subvolumes from stdin.",
"Receives one or more subvolumes that were previously",
"sent with btrfs send. The received subvolumes are stored",
@@ -1322,5 +1347,7 @@ const char * const cmd_receive_usage[] = {
"-m   The root mount point of the destination fs.",
" If you do not have /proc use this to tell us where ",
" this file system is mounted.",
+   "--dump   Exam and output metadata info o

[PATCH v4 0/4] Introduce dump option for btrfs-receive

[Qu Wenruo]

The branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/send_dump

The branch doesn't contain the fake test case.

Introduce new "--dump" option for btrfs-receive, which will exam and dump
metadata info of a send stream.
This is quite handy to debug send stream.

Since such function is provided by old send-test tool, which doesn't even
compile now, remove the old send-test tool.

changelog:
v2:
  Move from inspect subcommand to receive subcommand.
v3:
  Add output for ctime/mtime/atime
  (Human readable local time like "-11-01 11:11:11")
  Rearrange the output from "key1: value1, key2: value2" to
  "key1=value1 key2=value2". Suggested by David
  Rename macro path_cat_or_error() to PATH_CAT_OR_RET(). Suggested by David
  Change pointer to array to avoid memory allocation error. Suggested by David
  Add the 5th patch to add a fake test case to show how the new output looks
  like.
v4:
  Fix two small output defeats exposed by David.
  Add printf format checker attritube for __print_dump, suggested by David.
  Manually disable -Wformat-zero-length in send-dump.
  Ouput fattr's data, suggested by David.
  Change c/m/a time to RFC3339 format, suggested by David.
  Avoid calling memmove() in a loop in escape_string_inplace().
  Proviode non-inplace version escape_string(), which will alloc memory by 
itself.

Qu Wenruo (4):
  btrfs-progs: utils: Introduce function to escape characters
  btrfs-progs: introduce new send-dump object
  btrfs-progs: receive: introduce option to dump send stream
  btrfs-progs: misc-test: Add send stream dump test

 Documentation/btrfs-receive.asciidoc   |  15 +-
 Makefile.in|   2 +-
 cmds-receive.c |  35 ++-
 send-dump.c| 298 +
 send-dump.h|  29 ++
 .../016-send-dump-output/creation.stream.xz| Bin 0 -> 984 bytes
 .../016-send-dump-output/deletion.stream.xz| Bin 0 -> 408 bytes
 tests/misc-tests/016-send-dump-output/test.sh  |  23 ++
 utils.c|  24 ++
 utils.h|  14 +
 10 files changed, 434 insertions(+), 6 deletions(-)
 create mode 100644 send-dump.c
 create mode 100644 send-dump.h
 create mode 100644 tests/misc-tests/016-send-dump-output/creation.stream.xz
 create mode 100644 tests/misc-tests/016-send-dump-output/deletion.stream.xz
 create mode 100755 tests/misc-tests/016-send-dump-output/test.sh

-- 
2.10.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 1/4] btrfs-progs: utils: Introduce function to escape characters

[Qu Wenruo]

Introduce new function, escape_string_inplace(), to escape specified
characters in place.

Signed-off-by: Qu Wenruo 
---
 utils.c | 24 
 utils.h | 14 ++
 2 files changed, 38 insertions(+)

diff --git a/utils.c b/utils.c
index 3f54245..3c50d84 100644
--- a/utils.c
+++ b/utils.c
@@ -4251,3 +4251,27 @@ unsigned int rand_range(unsigned int upper)
 */
return (unsigned int)(jrand48(rand_seed) % upper);
 }
+
+static void escape_one_char(char *restrict string, char escape)
+{
+   int i = 0;
+   int j = 0;
+
+   while (string[j] != '\0') {
+   if (string[j] != escape) {
+   string[i] = string[j];
+   i++;
+   }
+   j++;
+   }
+   /* pend the finishing '\0' */
+   string[i] = '\0';
+}
+
+void string_escape_inplace(char *restrict string, char *restrict escape_chars)
+{
+   int i;
+
+   for (i = 0; i < strlen(escape_chars); i++)
+   escape_one_char(string, escape_chars[i]);
+}
diff --git a/utils.h b/utils.h
index 1a2dbcd..dae8db4 100644
--- a/utils.h
+++ b/utils.h
@@ -457,4 +457,18 @@ unsigned int rand_range(unsigned int upper);
 /* Also allow setting the seed manually */
 void init_rand_seed(u64 seed);
 
+void string_escape_inplace(char *restrict string, char *restrict escape_chars);
+
+static inline char* string_escape(char *restrict src,
+ char *restrict escape_chars)
+{
+   char *ret;
+
+   ret = malloc(strlen(src) + 1);
+   if (!ret)
+   return NULL;
+   string_escape_inplace(ret, escape_chars);
+   return ret;
+}
+
 #endif
-- 
2.10.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 2/4] btrfs-progs: introduce new send-dump object

[Qu Wenruo]

Introduce send-dump.[ch] which implements a new btrfs_send_ops to
exam and output all operations inside a send stream.

It has a better output format than the old and no longer compilable
send-test tool, but still tries to be script friendly.

Provides the basis for later "inspect-internal dump-send" command.

Signed-off-by: Qu Wenruo 
Signed-off-by: David Sterba 
---
 Makefile.in |   2 +-
 send-dump.c | 298 
 send-dump.h |  29 ++
 3 files changed, 328 insertions(+), 1 deletion(-)
 create mode 100644 send-dump.c
 create mode 100644 send-dump.h

diff --git a/Makefile.in b/Makefile.in
index b53cf2c..c535c19 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -93,7 +93,7 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o 
extent-tree.o print-tree.o \
  extent-cache.o extent_io.o volumes.o utils.o repair.o \
  qgroup.o raid56.o free-space-cache.o kernel-lib/list_sort.o props.o \
  ulist.o qgroup-verify.o backref.o string-table.o task-utils.o \
- inode.o file.o find-root.o free-space-tree.o help.o
+ inode.o file.o find-root.o free-space-tree.o help.o send-dump.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
   cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o cmds-check.o \
diff --git a/send-dump.c b/send-dump.c
new file mode 100644
index 000..5de8cdd
--- /dev/null
+++ b/send-dump.c
@@ -0,0 +1,298 @@
+/*
+ * Copyright (C) 2016 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include "utils.h"
+#include "commands.h"
+#include "send-utils.h"
+#include "send-stream.h"
+#include "send-dump.h"
+
+/*
+ * Disable format zero length warning since we use zero length format
+ * for unified function parameters.
+ */
+#pragma GCC diagnostic ignored "-Wformat-zero-length"
+
+#define PATH_CAT_OR_RET(function_name, outpath, path1, path2, ret) \
+({ \
+   ret = path_cat_out(outpath, path1, path2);  \
+   if (ret < 0) {  \
+   error("%s: path invalid: %s\n", function_name, path2);  \
+   return ret; \
+   }   \
+})
+
+/*
+ * Underlying PRINT_DUMP, the only difference is how we handle
+ * the full path.
+ */
+__attribute__ ((format (printf, 5, 6)))
+static int __print_dump(int subvol, void *user, const char *path,
+   const char *title, const char *fmt, ...)
+{
+   struct btrfs_dump_send_args *r = user;
+   char full_path[PATH_MAX] = {0};
+   char *out_path;
+   va_list args;
+   int ret;
+
+   if (subvol) {
+   PATH_CAT_OR_RET(title, r->full_subvol_path, r->root_path, path, 
ret);
+   out_path = r->full_subvol_path;
+   } else {
+   PATH_CAT_OR_RET(title, full_path, r->full_subvol_path, path, 
ret);
+   out_path = full_path;
+   }
+   string_escape_inplace(out_path, " \n\t\\");
+
+   /* Unified header, */
+   printf("%-16s%-32s", title, out_path);
+   va_start(args, fmt);
+   /* Operation specified ones */
+   vprintf(fmt, args);
+   va_end(args);
+   printf("\n");
+   return 0;
+}
+
+/* For subvolume/snapshot operation only */
+#define PRINT_DUMP_SUBVOL(user, path, title, fmt, ...) \
+   __print_dump(1, user, path, title, fmt, ##__VA_ARGS__)
+
+/* For other operations */
+#define PRINT_DUMP(user, path, title, fmt, ...) \
+   __print_dump(0, user, path, title, fmt, ##__VA_ARGS__)
+
+static int print_subvol(const char *path, const u8 *uuid, u64 ctransid,
+   void *user)
+{
+   char uuid_str[BTRFS_UUID_UNPARSED_SIZE];
+
+   uuid_unparse(uuid, uuid_str);
+
+   return PRINT_DUMP_SUBVOL(user, path, "subvol", "uuid=%s transid=%llu",
+uuid_str, ctransid);
+}
+
+static int print_snapshot(const char *path, const u8 *uuid, u64 ctransid,
+ const u8 *parent_uuid, u64 parent_ctransid,
+ void *user)
+{
+ 

[PATCH v4 4/4] btrfs-progs: misc-test: Add send stream dump test

[Qu Wenruo]

* PLEASE DON'T MERGE THIS PATCH*

With 2 send stream which contains all operations for user to check the
output.

This is just used for checking output format and find any possible
deflects or ugly layout.

It doesn't really checks any thing and doesn't follow normal test output
redirection.

Signed-off-by: Qu Wenruo 
---
 .../016-send-dump-output/creation.stream.xz| Bin 0 -> 984 bytes
 .../016-send-dump-output/deletion.stream.xz| Bin 0 -> 408 bytes
 tests/misc-tests/016-send-dump-output/test.sh  |  23 +
 3 files changed, 23 insertions(+)
 create mode 100644 tests/misc-tests/016-send-dump-output/creation.stream.xz
 create mode 100644 tests/misc-tests/016-send-dump-output/deletion.stream.xz
 create mode 100755 tests/misc-tests/016-send-dump-output/test.sh

diff --git a/tests/misc-tests/016-send-dump-output/creation.stream.xz 
b/tests/misc-tests/016-send-dump-output/creation.stream.xz
new file mode 100644
index 
..521c5d97170e7ad41ac8b3305461df09e2cb4c86
GIT binary patch
literal 984
zcmV;}11J3bH+ooF000E$*0e?f03iVu0001VFXf})8bkw?T>vp13XGw}P}F=IbdLR(
z>U`@9XCIGr1@6OEPWlaQgT5hCS#G{B*Zx$SeQxx(I;AlUUp>USnJ^DpJDxOMlV57u
zelh2}f67+r{m^CB^YzYr3tjQ|k)GN6FCPlJzZl~%dVK~u^utMa*9rwMg9+N$g{SND
zcZ-NRsrbN=(OpirB^JtZ_@@|0ko&m=lrZK&kwT4$K~N%4wfsccFuJN7=oxM?(C{$t
zrJ3+}UkQ}_E*UQZWqzP7z377g)WK7w_xN6@JI=Ju(YvI$Mt_bx^7=ASE}(b8_B}4i
zasV`w=9w+*M6KB1PI0epqZRs16o(yJ_TJRHtLJs)H=r2l{lUm#HgtEO+R509mJ9Dp
zH53E2E>5wavINe=AaNq|VGT6MOCuv!R7O%jU@r%Yrdpfo?l6$2s-G=tmJ&#ZJK-XI
zc3_*HY85LsZo<$!ckHazSUr=utE9f?X_~CA>t03#bx~FT8lXi$?P|(
zyUn-J`1NVoq>_vLA+8)in6aeaTk1B53fFNk)2uRv5OTLy;afVi?wl>cMS@Y{dRkH;AHw
zS^qftW|-zRq~4aB8Kg{kG*5vYWv*suPOOIx5<3%t>J~EcplJGDQ@?EpA?a!WZ^tuQB7Lid{7l
zWLuhRJZ3uiQ(-7c!+d4j2#bJ4GD4P%lv6G(cG-A7%R}d${N+_2>{&W|9ejC}b5lc$A}M$5e(E4qn&EfR`1?U4vp13XGw}P}F=IbdLR(
z>U`@R#K~!;K6&PD9_87`}Ysb&w5Tjy)cDTA~&eSS_FzE%Qx
z1=H-qw6D#$620}uFzMY^0002pjh8@1RLQ^q0rLWy1pojcFQxvm#Ao{g01X)_e
CV8dwu

literal 0
HcmV?d1

diff --git a/tests/misc-tests/016-send-dump-output/test.sh 
b/tests/misc-tests/016-send-dump-output/test.sh
new file mode 100755
index 000..cc8586c
--- /dev/null
+++ b/tests/misc-tests/016-send-dump-output/test.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# output receive --dump output for further tunning
+
+source $TOP/tests/common
+
+check_prereq btrfs
+
+# send streams are in *.stream.xz format, can't reuse check_all_images()
+
+rm *.stream -f
+
+for image in $(find . -iname '*.stream.xz'); do
+   xz --decompress --keep "$image" || \
+   _fail "failed to decompress $image" >&2
+   image=${image%%.xz}
+
+   echo "## $image: ##"
+   # We call btrfs directly to output result without redirection
+   $TOP/btrfs receive --dump -f $image || \
+   _fail "failed to exam send stream"
+   rm -f $image
+done
-- 
2.10.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 02/14] btrfs-progs: check: introduce function to find dir_item

[Qu Wenruo]

At 11/02/2016 11:21 PM, David Sterba wrote:

On Wed, Sep 21, 2016 at 11:15:52AM +0800, Qu Wenruo wrote:

From: Lu Fengqi 

Introduce a new function find_dir_item() to find DIR_ITEM for the given
key, and check it with the specified INODE_REF/INODE_EXTREF match.

Signed-off-by: Lu Fengqi 
Signed-off-by: Qu Wenruo 
---
 cmds-check.c | 140 +++
 1 file changed, 140 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index 998ba63..4e25804 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3848,6 +3848,146 @@ out:
return err;
 }

+#define ROOT_DIR_ERROR (1<<1)/* bad root_dir */
+#define DIR_ITEM_MISSING   (1<<2)/* DIR_ITEM not found */
+#define DIR_ITEM_MISMATCH  (1<<3)/* DIR_ITEM found but not match */


What's the reason for another definition of the error codes? They're
mostly copied from te I_ERR_* counterparts. I'd rather have one set of
error codes.


The main reason is, in lowmem fsck mode, we are not checking inodes or 
ref/backref in batch.


If using I_ERR and REF_ERR, we can mixing them up as they share the same 
bits.


So we introduced the new error bitmap, to make sure all error bits won't 
cover each other.


It may be better if we rearrange I_ERR/REF_ERR to avoid conflicts.
E.g, let REF_ERR_ starts from lowest bit and let I_ERR_ starts from high 
bits.


Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[Qu Wenruo]

At 11/02/2016 06:55 PM, David Sterba wrote:

On Wed, Nov 02, 2016 at 09:19:10AM +0800, Qu Wenruo wrote:



At 11/01/2016 06:08 PM, David Sterba wrote:

On Tue, Nov 01, 2016 at 04:01:43PM +0800, Qu Wenruo wrote:

Introduce new function, escape_string_inplace(), to escape specified
characters in place.


Sorry, the pointer to seq_path was misleading. The actual escape
function is mangle_path and it copies one string to another. As we just
print the path, we can simply switch and call putchar.



Putchar() method is indeed much easier to implement.

But it makes us hard to do further formatting, like aligning the path to
given width. (At least we are still using 32 chars alignment for path)

So I still prefer the current full function string escaping and still
use %-32s for formatting.


And the idea of implementing escape_string_inplace() as a pure string
manipulation function can make it more agile for later use.
For example, we can reuse it for print-tree.


Reusing is fine, but I really don't like that the function modifies the
argument. What if the function is called twice on the same string?
Forgot to ask, what's the problem calling the function twice on the same 
string?


escape_string_inplace(dest, " ");
escape_string_inplace(dest, "\n");

is just the same as

escape_string_inplace(dest, " \n");

So I see no problem here.

Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[Qu Wenruo]

At 11/02/2016 06:55 PM, David Sterba wrote:

On Wed, Nov 02, 2016 at 09:19:10AM +0800, Qu Wenruo wrote:



At 11/01/2016 06:08 PM, David Sterba wrote:

On Tue, Nov 01, 2016 at 04:01:43PM +0800, Qu Wenruo wrote:

Introduce new function, escape_string_inplace(), to escape specified
characters in place.


Sorry, the pointer to seq_path was misleading. The actual escape
function is mangle_path and it copies one string to another. As we just
print the path, we can simply switch and call putchar.



Putchar() method is indeed much easier to implement.

But it makes us hard to do further formatting, like aligning the path to
given width. (At least we are still using 32 chars alignment for path)

So I still prefer the current full function string escaping and still
use %-32s for formatting.


And the idea of implementing escape_string_inplace() as a pure string
manipulation function can make it more agile for later use.
For example, we can reuse it for print-tree.


Reusing is fine, but I really don't like that the function modifies the
argument. What if the function is called twice on the same string? Also,
in the print-tree, this would mean the extent buffer would be modified,
potentially overwriting other items.



Then just encapsulate the in-place version into memory allocation version.

In-place version can be encapsulated very easily, while it's not 
possible vice verse.


So there will be two functions, escape_string_inplace() and 
escape_string() for caller to choose.


Would this be OK?

Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 2/4] btrfs-progs: introduce new send-dump object

[Qu Wenruo]

At 11/02/2016 06:52 PM, David Sterba wrote:

On Wed, Nov 02, 2016 at 08:37:20AM +0800, Qu Wenruo wrote:

PATH_WIDTH is used only here, please hardcode it into the format string.

The rest of the patch looks good. I think I've seen some artifacts in
the output, but we can tune this later.


Would you like to share the artifacts so I can handle them in next update?


From the output of misc test 016:

snapshot: [...] parent_transid= 14
   ^ extra space

set_xattr:  ./snap_creation/xattr_file  name=user.test, len=1
  ^ comma

also set xattr does not print the attribute data.


Thanks, I'll fix them.

Thanks,
Qu

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: State of the fuzzer

[David Sterba]

Hi,

On Tue, Oct 11, 2016 at 02:07:48PM +0200, Lukas Lueg wrote:
> I've now shut down all fuzzer nodes since they only cost money and
> there is no progress on most of the aforementioned bugs.

sorry, I didn't know there are cost implications. I've triaged the bugs
some time ago and tried to fix one of the underlying problems that would
probably close serveal bugs, but it turned out to be not that easy.

There's not that much interest in k.org bugs from other developers and I
have to split my time among several tasks or keep up with release
schedules.

I appreciate that you took the time to do the tests and reports, I'll
get to them eventually. I also hope you're not discouraged from further
testing. This would make best sense only after all current fuzzer bugs
are fixed, but I can't give you an ETA.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs scrub with unexpected results

[Tom Arild Naess]

Hello,

I have been running btrfs on a file server and backup server for a 
couple of years now, both set up as RAID 10. The file server has been 
running along without any problems since day one. My problems has been 
with the backup server.


A little background about the backup server before I dive into the 
problems. The server was a new build that was set to replace an aging 
machine, and my intention was to start using btrfs send/receive instead 
of hard links for the backups. Since I had 8x the space on the new 
server, I just rsynced the whole lot of old backups to the new server. I 
then made some scripts that created snapshots from the old file 
hierarchy. As I started rewriting my backup scripts (on file server and 
backup server) to use send/receive, I also tested scrubbing to see that 
everything was OK. After doing this a few times, scrub found 
unrecoverable files. This, I thought, should not be possible on new 
disks. I tried to get some help on this list, but no answers were found, 
and since I was unable to find what triggered this, I just stopped using 
send/receive, and let my old backup regime live on on this new backup 
server as well. I don't remember how I fixed the errors, but I guess I 
just replaced the offending files with fresh ones, and scrub ran without 
any more problems. I decided to let things just run like this, and set 
up scrubbing on a monthly schedule.


Last night I got the unpleasant mail from cron telling me that scrub had 
failed (for the first time in over a year). Since I was running on an 
older kernel (4.2.x), I decided to upgrade, and went for the latest of 
the longterm branches, namely 4.4.30. After rebooting I did (for 
whatever reason) check one of the offending files, and I could read the 
file just fine! I checked the rest of the bunch, and all files read 
fine, and had the same md5 sum as the originals! All these files were 
located in those old snapshots. I thought that maybe this was because of 
a bug resolved since my last kernel. Then I ran a new scrub, and this 
one also reported unrecoverable errors. This time on two other files but 
also in some of the old snapshots. I tried reading the files, and got 
the expected I/O errors. One reboot later, these files reads just fine 
again!


Some system info:

$ uname -a
Linux backup 4.4.30-1-lts #1 SMP Tue Nov 1 22:09:20 CET 2016 x86_64 
GNU/Linux


$ btrfs --version
btrfs-progs v4.8.2

$ btrfs fi show /backup
Label: none  uuid: 8825ce78-d620-48f5-9f03-8c4568d3719d
Total devices 4 FS bytes used 2.81TiB
devid1 size 2.73TiB used 1.41TiB path /dev/sdb
devid2 size 2.73TiB used 1.41TiB path /dev/sda
devid3 size 2.73TiB used 1.41TiB path /dev/sdd
devid4 size 2.73TiB used 1.41TiB path /dev/sdc


Thanks!

Tom Arild Naess


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send/receive snapshot from/between backup (also restore example)

[Hans van Kranenburg]

On 11/02/2016 04:14 PM, René Bühlmann wrote:
> On 11/02/2016 03:49 PM, Hans van Kranenburg wrote:
>> On 11/02/2016 03:23 PM, René Bühlmann wrote:
>>> I have a system running on btrfs which is backed up to two (one local
>>> USB and one through SSH) external drives (also using btrfs) using
>>> send/receive (with the help of btrbk). For each backup a snapshot is
>>> created and transferred to the external drives. After some time, the
>>> snapshot on the origin is removed but kept on the two backups.
>>>
>>> Now my problem is, that the SSH-drive has not received any snapshots for
>>> some time such that there is no snapshot any more which is on the origin
>>> AND the SSH-drive. I would like to prevent transferring a full snapshot
>>> over the network and am now trying to work around it.
>>>
>>> Here the current situation (S? for a Snapshot):
>>>
>>> Origin: S2 S3
>>>
>>> USB: S1 S2
>>>
>>> SSH: S1
>>>
>>> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
>>> how can I transfer S3 to SSH?
>> Since there is no snapshot left that is present on both drives, you
>> cannot use incremental send.
>>
>> btrbk does not remember what snapshots were last transferred or were
>> present on the missing disk, since the tool does not do any extra meta
>> administation on top of just having the snapshots on btrfs level. So
>> it's figuring out the relationships between each of them (combining it
>> with info from the remotes) again every time it runs.
> Yes this explains why 1. did not work. But why can't I transfer S1 from
> USB back to Origin as incremental to S2?

You can, but what you end up with is not the same as the original S1 any
more. It will have a new uuid, which is not the received_uuid of the S1
that is still on SSH, so they won't be 'common'.

Allow me to demonstrate... A slightly different use case, but based on
this you can debug your own scenario again in the same way:

The following is how to snapshot to a remote, how to roll back to a
snapshot that is only on the remote (send it back), and then continue
sending incrementals again to the remote again.

Let's create some snapshots, and send them:

/mnt/zooi 12-# mkdir send
/mnt/zooi 12-# mkdir receive
/mnt/zooi 12-# mount /dev/zooi/send send/
/mnt/zooi 12-# mount /dev/zooi/receive receive/
/mnt/zooi 12-# cd send/

/mnt/zooi/send 12-# btrfs sub create A
Create subvolume './A'

/mnt/zooi/send 12-# dd if=/dev/zero of=A/1 bs=1 count=0 seek=1G; shred
-n 1 A/1
/mnt/zooi/send 12-# btrfs sub snap -r A A1
Create a readonly snapshot of 'A' in './A1'
/mnt/zooi/send 12-# btrfs send A1 | btrfs receive ../receive/
At subvol A1
At subvol A1

/mnt/zooi/send 12-# dd if=/dev/zero of=A/2 bs=1 count=0 seek=1G; shred
-n 1 A/2
/mnt/zooi/send 12-# btrfs sub snap -r A A2
Create a readonly snapshot of 'A' in './A2'
/mnt/zooi/send 12-# btrfs send -p A1 A2 | btrfs receive ../receive/
At subvol A2
At snapshot A2

/mnt/zooi/send 12-# dd if=/dev/zero of=A/3 bs=1 count=0 seek=1G; shred
-n 1 A/3
/mnt/zooi/send 12-# btrfs sub snap -r A A3
Create a readonly snapshot of 'A' in './A3'
/mnt/zooi/send 12-# btrfs send -p A2 A3 | btrfs receive ../receive/
At subvol A3
At snapshot A3

Now, the list of subvolumes at the send side looks like (linebreaks added):

/mnt/zooi/send 12-# btrfs sub list -u -q -R .
ID 271
 gen 40
 top level 5
 parent_uuid -
 received_uuid -
 uuid f08112b1-223d-cb46-b54b-2d90c8339617
 path A
ID 274
 gen 39
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid 862f3808-ccdf-cb40-8cf1-c31b8cbea635
 path A1
ID 276
 gen 40
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid ab4bd82b-39f9-a24c-9ead-02023e392e9f
 path A2
ID 278
 gen 41
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 path A3

You can see that all of the snapshots have f08112b1 as parent.

At the receiving side:

/mnt/zooi/receive 12-# btrfs sub list -u -q -R .

ID 263
 gen 31
 top level 5
 parent_uuid -
 received_uuid 862f3808-ccdf-cb40-8cf1-c31b8cbea635
 uuid 41a4495c-89bd-f949-8bfe-601e197eb96d
 path A1
ID 265
 gen 34
 top level 5
 parent_uuid 41a4495c-89bd-f949-8bfe-601e197eb96d
 received_uuid ab4bd82b-39f9-a24c-9ead-02023e392e9f
 uuid 332fd67a-cfd4-7e45-9114-a910fc8d5b95
 path A2
ID 267
 gen 36
 top level 5
 parent_uuid 332fd67a-cfd4-7e45-9114-a910fc8d5b95
 received_uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 uuid bb3a14fd-668c-6c4d-be1b-fc11a4cda6a0
 path A3

Here we see that if -p is used for send, the receiving side snapshots
the subvolume which has the sender-side uuid of the one referenced with
-p in its received_uuid field, and based on that clone (which on the
receiving side now has the previous received one as parent_uuid), it
adds the diff and it ends up as new read-only snapshot.

The original parent_uuid two steps back, is nowhere to be found any more.

Now let's remove some snapshots from the origin...

/mnt/zooi/send 12-# btrfs sub del A1 A

Re: Send/receive snapshot from/between backup

[Piotr Pawłow]

On 02.11.2016 15:23, René Bühlmann wrote:
> Origin: S2 S3
>
> USB: S1 S2
>
> SSH: S1
>
> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
> how can I transfer S3 to SSH?

If I understand correctly how send / receive works, for the incremental
receive to work there must be a subvolume on the destination which has
"received uuid" equal to the uuid of parent choosen for the incremental
send.

> I tried to transfer...
>
> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
>
> 2. S2 from USB to SSH -> did not work.

The "received uuid" of S1 on SSH is the uuid S1 had on Origin. The uuid
of S1 on USB is different, so when choosen as parent for the incremental
send it doesn't match.

> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
> -> did not work.

There are no previously received subvolumes on Origin at all, so it
isn't going to work.

> Is it correct that 1. would work if a common snapshot is present on
> Origin and SSH?

If there was a snapshot received from Origin that still exists on
Origin, then yes, you could use it as a clone source for incremental send.

> Is it expected that 2. and 3. do not work?
>
> Is there some other way to achieve it?

I doubt you can do it without some "hacking" to fool btrfs receive.

You would need a tool that can issue BTRFS_IOC_SET_RECEIVED_SUBVOL ioctl
to change the received uuid. Then you could:

1. Change received uuid of S1 on SSH to match S1 uuid on USB.
2. Send incremental S1-S2 from USB to SSH.
3. Change received uuid of S2 on SSH to match S2 on Origin.
4. Send incremental S2-S3 from Origin to SSH.

Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

df missing filesystem when run on subvolume

[Kyle Gates]

As the title suggests, running the df command on a subvolume doesn't return a 
filesystem. I'm not sure where the problem lies or if anyone else has noticed 
this. Some programs fail to detect free space as a result.

Example for clarification:
kyle@home:~$ sudo mount -o subvol=@data /mnt/btrfs/
kyle@home:~$ mkdir /mnt/btrfs/directory
kyle@home:~$ btrfs subvolume create /mnt/btrfs/subvolume
Create subvolume '/mnt/btrfs/subvolume'
kyle@home:~$ sudo btrfs subvolume list /mnt/btrfs/
ID 258 gen 2757271 top level 5 path @data
ID 5684 gen 2718215 top level 258 path subvolume
kyle@home:~$ df /mnt/btrfs/
Filesystem  1K-blocks   Used Available Use% Mounted on
/dev/sdc2  1412456448 1170400072 240688008  83% /mnt/btrfs
kyle@home:~$ df /mnt/btrfs/directory
Filesystem  1K-blocks   Used Available Use% Mounted on
/dev/sdc2  1412456448 1170400072 240688008  83% /mnt/btrfs
kyle@home:~$ df /mnt/btrfs/subvolume
Filesystem  1K-blocks   Used Available Use% Mounted on
-  1412456448 1170400072 240688008  83% /mnt/btrfs/subvolume

Thanks,
Kyle--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 02/14] btrfs-progs: check: introduce function to find dir_item

[David Sterba]

On Wed, Sep 21, 2016 at 11:15:52AM +0800, Qu Wenruo wrote:
> From: Lu Fengqi 
> 
> Introduce a new function find_dir_item() to find DIR_ITEM for the given
> key, and check it with the specified INODE_REF/INODE_EXTREF match.
> 
> Signed-off-by: Lu Fengqi 
> Signed-off-by: Qu Wenruo 
> ---
>  cmds-check.c | 140 
> +++
>  1 file changed, 140 insertions(+)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index 998ba63..4e25804 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -3848,6 +3848,146 @@ out:
>   return err;
>  }
>  
> +#define ROOT_DIR_ERROR   (1<<1)  /* bad root_dir */
> +#define DIR_ITEM_MISSING (1<<2)  /* DIR_ITEM not found */
> +#define DIR_ITEM_MISMATCH(1<<3)  /* DIR_ITEM found but not match */

What's the reason for another definition of the error codes? They're
mostly copied from te I_ERR_* counterparts. I'd rather have one set of
error codes.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send/receive snapshot from/between backup

[René Bühlmann]

On 11/02/2016 03:49 PM, Hans van Kranenburg wrote:
> On 11/02/2016 03:23 PM, René Bühlmann wrote:
>> I have a system running on btrfs which is backed up to two (one local
>> USB and one through SSH) external drives (also using btrfs) using
>> send/receive (with the help of btrbk). For each backup a snapshot is
>> created and transferred to the external drives. After some time, the
>> snapshot on the origin is removed but kept on the two backups.
>>
>> Now my problem is, that the SSH-drive has not received any snapshots for
>> some time such that there is no snapshot any more which is on the origin
>> AND the SSH-drive. I would like to prevent transferring a full snapshot
>> over the network and am now trying to work around it.
>>
>> Here the current situation (S? for a Snapshot):
>>
>> Origin: S2 S3
>>
>> USB: S1 S2
>>
>> SSH: S1
>>
>> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
>> how can I transfer S3 to SSH?
> Since there is no snapshot left that is present on both drives, you
> cannot use incremental send.
>
> btrbk does not remember what snapshots were last transferred or were
> present on the missing disk, since the tool does not do any extra meta
> administation on top of just having the snapshots on btrfs level. So
> it's figuring out the relationships between each of them (combining it
> with info from the remotes) again every time it runs.
Yes this explains why 1. did not work. But why can't I transfer S1 from
USB back to Origin as incremental to S2?
And why can't I transfer S2 from USB to SSH? Is S1 not recognized as the
same snapshot there?
>
>> I tried to transfer...
>>
>> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
>>
>> 2. S2 from USB to SSH -> did not work.
>>
>> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
>> -> did not work.
>>
>> Is it correct that 1. would work if a common snapshot is present on
>> Origin and SSH?
>>
>> Is it expected that 2. and 3. do not work?
>>
>> Is there some other way to achieve it?
> At home, I do a similar thing, I periodically send/receive changes of a
> filesystem to two external disks, also using btrbk. Since I don't want
> to have both backup disks and the originating filesystem all online and
> in the same geographical location at the same moment, I have the same
> problem.
>
> What I did is just setting expiry the snapshots on the origin manually,
> and keep the meta-administration in my head. I don't do it that often
> anyway.
>


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 4/4] btrfs-progs: remove send-test tool

[David Sterba]

On Tue, Nov 01, 2016 at 04:01:46PM +0800, Qu Wenruo wrote:
> Since new "receive --dump" has better output and structure, it's time
> to remove old and function-weak send-test tool.
> 
> Signed-off-by: Qu Wenruo 
> Signed-off-by: David Sterba 

As this patch is independet, applied now, no need to resend with the
rest.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send/receive snapshot from/between backup

[Hans van Kranenburg]

On 11/02/2016 03:23 PM, René Bühlmann wrote:
> 
> I have a system running on btrfs which is backed up to two (one local
> USB and one through SSH) external drives (also using btrfs) using
> send/receive (with the help of btrbk). For each backup a snapshot is
> created and transferred to the external drives. After some time, the
> snapshot on the origin is removed but kept on the two backups.
> 
> Now my problem is, that the SSH-drive has not received any snapshots for
> some time such that there is no snapshot any more which is on the origin
> AND the SSH-drive. I would like to prevent transferring a full snapshot
> over the network and am now trying to work around it.
> 
> Here the current situation (S? for a Snapshot):
> 
> Origin: S2 S3
> 
> USB: S1 S2
> 
> SSH: S1
> 
> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
> how can I transfer S3 to SSH?

Since there is no snapshot left that is present on both drives, you
cannot use incremental send.

btrbk does not remember what snapshots were last transferred or were
present on the missing disk, since the tool does not do any extra meta
administation on top of just having the snapshots on btrfs level. So
it's figuring out the relationships between each of them (combining it
with info from the remotes) again every time it runs.

> I tried to transfer...
> 
> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
> 
> 2. S2 from USB to SSH -> did not work.
> 
> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
> -> did not work.
> 
> Is it correct that 1. would work if a common snapshot is present on
> Origin and SSH?
> 
> Is it expected that 2. and 3. do not work?
> 
> Is there some other way to achieve it?

At home, I do a similar thing, I periodically send/receive changes of a
filesystem to two external disks, also using btrbk. Since I don't want
to have both backup disks and the originating filesystem all online and
in the same geographical location at the same moment, I have the same
problem.

What I did is just setting expiry the snapshots on the origin manually,
and keep the meta-administration in my head. I don't do it that often
anyway.

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: Remove some dead code

[David Sterba]

On Tue, Nov 01, 2016 at 11:26:06AM +0100, Christophe JAILLET wrote:
> 'btrfs_iget()' can not return an error pointer, so this test can be
> removed.
> 
> Signed-off-by: Christophe JAILLET 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Send/receive snapshot from/between backup

[René Bühlmann]

Hi all,

I have a system running on btrfs which is backed up to two (one local
USB and one through SSH) external drives (also using btrfs) using
send/receive (with the help of btrbk). For each backup a snapshot is
created and transferred to the external drives. After some time, the
snapshot on the origin is removed but kept on the two backups.

Now my problem is, that the SSH-drive has not received any snapshots for
some time such that there is no snapshot any more which is on the origin
AND the SSH-drive. I would like to prevent transferring a full snapshot
over the network and am now trying to work around it.

Here the current situation (S? for a Snapshot):

Origin: S2 S3

USB: S1 S2

SSH: S1

Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
how can I transfer S3 to SSH?

I tried to transfer...

1. S3 from Origin to SSH -> does not work as there is no common snapshot.

2. S2 from USB to SSH -> did not work.

3. S1 from USB to Origin (such that there is a common snapshot with SSH)
-> did not work.

Is it correct that 1. would work if a common snapshot is present on
Origin and SSH?

Is it expected that 2. and 3. do not work?

Is there some other way to achieve it?

Thanks for the help

René


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: Call kunmap if zlib_inflateInit2 fails

[David Sterba]

On Tue, Nov 01, 2016 at 08:25:27PM -0700, Nick Terrell wrote:
> If zlib_inflateInit2 fails, the input page is never unmapped.
> Add a call to kunmap when it fails.
> 
> Signed-off-by: Nick Terrell 

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/3] btrfs-progs: send: fix handling of -c option

[David Sterba]

On Wed, Oct 19, 2016 at 11:35:40AM +0900, Tsutomu Itoh wrote:
> When two or more -c options are specified, cannot find a suitable
> parent. So, output stream is bigger than correct one.
> 
> [before]
> # btrfs send -f /tmp/data1 -c Snap0 -c ../SnapX Snap[12] ../SnapY
> At subvol Snap1
> At subvol Snap2
> At subvol ../SnapY
> # ls -l /tmp/data1
> -rw--- 1 root root 3153 Oct 19 10:37 /tmp/data1
> #
> 
> [after]
> # btrfs send -f /tmp/data1 -c Snap0 -c ../SnapX Snap[12] ../SnapY
> At subvol Snap1
> At subvol Snap2
> At subvol ../SnapY
> # ls -l /tmp/data1
> -rw--- 1 root root 1492 Oct 19 10:39 /tmp/data1
> #

Can you please create a test for that?

> Signed-off-by: Tsutomu Itoh 
> ---
>  cmds-send.c | 12 
>  1 file changed, 12 insertions(+)
> 
> diff --git a/cmds-send.c b/cmds-send.c
> index 2a8a697..b93a667 100644
> --- a/cmds-send.c
> +++ b/cmds-send.c
> @@ -651,6 +651,18 @@ int cmd_send(int argc, char **argv)
>   }
>  
>   if (!full_send && root_id) {
> + ret = init_root_path(&send, subvol);
> + if (ret < 0)
> + goto out;
> +
> + ret = get_root_id(&send,
> + subvol_strip_mountpoint(send.root_path, subvol),
> + &root_id);
> + if (ret < 0) {
> + error("cannot resolve rootid for %s", subvol);
> + goto out;
> + }

This duplicates the code that handles the '-c' option, please factori it
toa helper function.

> +
>   ret = find_good_parent(&send, root_id, &parent_root_id);
>   if (ret < 0) {
>   error("parent determination failed for %lld",
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] btrfs-progs: send: fix handling of multiple snapshots (-p option)

[David Sterba]

On Wed, Nov 02, 2016 at 10:11:55AM +0900, Tsutomu Itoh wrote:
> > I'm not sure it's fixed, I wrote a simple test, attached, that triggers
> > the bug even with the patch applied.
> 
> In the attached script,
> 
>run_check $SUDO_HELPER btrfs send -f "$here"/send.stream -p subv-snap1 
> subv-snap2 subv-snap3
> 
> I think that 'btrfs' is a mistake of '$TOP/btrfs'.

Oh right, with the fix the test passes.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Austin S. Hemmelgarn]

On 2016-11-02 07:18, Ahmed Badr wrote:

On Wed, Nov 2, 2016 at 8:31 AM, Alex Powell
 wrote:

Taking a step back as well- there is also the possibility that you
might not need snapshots


I do need it for my root partition at least in the initial phase of
setting things up and trying out different software. Once the system
is stable, I might might not need it that much.
So, it's worth mentioning that there are about 4 practical uses for 
snapshots:
1. File versioning/history.  Essentially, usage like Windows File 
History (which uses snapshots and COW on NTFS) or Apples Time Machine. 
This is generally useful to a traditional home user, but in many cases, 
if you actually _need_ file versioning, your better off using a VCS like 
Git or Mercurial.
2. Atomic installs and updates.  This is essentially what 
apt-btrfs-snapshot and the snapper integration in Zypper do, and there 
are other cases of software that do this.  This is generally the most 
useful to traditional end-users.
3. On-system point-in-time copies of specific things.  This is one of 
the things that LVM snapshots are traditionally used for, and isn't 
often useful to end users, but can be for some middle-ware.  In essence, 
this is the same as 2, just for things other than software.
4. Getting a stable view of a live filesystem.  This is important for 
backup software and similar tools.  In essence, by taking a snapshot of 
the filesystem to be backed up, you can guarantee that nothing will 
change as your archiving it.  I don't know of any backup software that 
does this, but the custom backup scripts I use for my personal systems 
do, and we're looking at setting up our custom backup system where I 
work to do so also.


2, 3, and 4 are generally always useful.  For 2, you just need 
apt-btrfs-snapshot.  For 3 just manually creating snapshots is perfectly 
fine, and for 4 manual creation or automatic through your regular backup 
software works.  If you don't need them for the first use, I'd suggest 
just uninstalling snapper, as that's about all it's used for on Ubuntu.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Austin S. Hemmelgarn]

On 2016-11-02 07:25, Ahmed Badr wrote:

On Wed, Nov 2, 2016 at 2:19 AM, Hans van Kranenburg
 wrote:

While all of the snapshots with a matching parent_uuid are snapshots of
the same thing in linear time for btrfs, the tools may have their own
added administration on top (like snapshot names, or directories they're
placed in), which prevents them from being able to do something with the
other ones. This is not the fault of btrfs, it's because those tools
probably are designed to ignore anything they didn't cause to happen
themselves.


I understand now. Both apps seems to behave well and do not mess with
each others snapshots so I'm tempted to use both but I think I better
stick to only one as I might get confused with so many snapshots and
rollback to the wrong one.
What I would suggest doing in this case is to split out /home to a 
separate filesystem, use snapper for that, and use apt-btrfs-snapshot 
for everything else (except possibly stuff which gets served externally 
by the system, such as /var/www).  That way you get the benefits of 
snapper for the files where that type of snapshotting matters, and 
apt-btrfs-snapshot for where that actually matters, and you have no 
chance of them stepping on each others toes.


Also, make sure to check the settings for snapper, the defaults are 
pathologically bad for most end-users and will quite often put you in a 
bad situation eventually with the filesystem.  What actually works is up 
to your use case, but you almost certainly don't need snapshots more 
frequently than every few minutes.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resizing BTRFS - raw partition

[Austin S. Hemmelgarn]

On 2016-11-02 05:18, Christian Völker wrote:

Hi Hugo,

thanks for the quick reply. Regarding version- I prefer to use stable
Linux versionsand I am not going to upgrade just btrfs outside of
the verndors builds. So I am stuck happily with this version. And I run
Linux since more than 10years, so I am really fine with it, I guess :D

And thanks again for your proposal. Yes, your command worked.

I had to tell betrfs the devid!

So this did NOT work:

 btrfs fi resize  max /srv/share/

Instead the following two commands worked:

 btrfs fi resize  1:max /srv/share/
 btrfs fi resize  2:max /srv/share/

And now boths phydevices show the correct size.

This sound really strange for me that I have to tell btrfs to resize
just a single disk insteag of automatically resizing all disks...I bet
next time I have it forgotten again :-(

Responding just to this in particular.

The reasoning behind this is that by requiring the device ID to be 
specified, it reduces the amount of parsing required, which in turn 
makes the command more robust.  There are also plenty of practical 
reasons to resize just one device in a multi-device filesystem (for 
example, if you're replacing one disk at a time, or just need to free up 
some space on a single disk, etc), so we need to have support for that, 
and in turn it's easier from a software maintenance perspective to just 
require it to be that way.


That said, I would love to have 'btrfs fi resize max' special cased to 
just resize every device in the FS to max size, since increasing the 
size of a device is much more common than reducing it, and it's a lot 
more convenient than having to look up device ID's (because those can 
change as devices are added to and removed from the FS).


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Ahmed Badr]

On Wed, Nov 2, 2016 at 2:19 AM, Hans van Kranenburg
 wrote:
> While all of the snapshots with a matching parent_uuid are snapshots of
> the same thing in linear time for btrfs, the tools may have their own
> added administration on top (like snapshot names, or directories they're
> placed in), which prevents them from being able to do something with the
> other ones. This is not the fault of btrfs, it's because those tools
> probably are designed to ignore anything they didn't cause to happen
> themselves.

I understand now. Both apps seems to behave well and do not mess with
each others snapshots so I'm tempted to use both but I think I better
stick to only one as I might get confused with so many snapshots and
rollback to the wrong one.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Ahmed Badr]

On Wed, Nov 2, 2016 at 8:31 AM, Alex Powell
 wrote:
> Taking a step back as well- there is also the possibility that you
> might not need snapshots

I do need it for my root partition at least in the initial phase of
setting things up and trying out different software. Once the system
is stable, I might might not need it that much.

> If you're a noobie, I assume you're not using it to host some massive
> Oracle DB and need all these features

No I'm not hosting a DB on this server. If I'm not mistaken, I believe
Btrfs's COW functionality is not suitable for DB storage anyway.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Ahmed Badr]

On Wed, Nov 2, 2016 at 8:03 AM, Duncan <1i5t5.dun...@cox.net> wrote:
> Btrfs tends to have scaling issues if you let the number of snapshots get
> too high.  Try to keep it under 300 snapshots per subvolume (combined
> between all tools) if at all possible, and if your use-case makes it easy
> enough, try to keep it under 100 snapshots per subvolume.  Definitely
> don't let it get into the thousands of snapshots per subvolume, or
> commands such as btrfs balance and btrfs check will take MUCH longer and
> use MANY TIMES more memory.

Thanks for the heads up on that. I didn't find this info anywhere so I
will  definitely do that. I don't think I'll even need that many
snapshots in the first place. Since this is a dedicated backup server
that I'll set and forget, I'll keep a snapshot every couple of hours
for the first day and a daily snapshot for a week. I'll be backing the
root partition on another computer anyway.

Since I'll be trying out several backup software to see which one
suits me best I'll use Snapper prior to installing each software. I'm
very excited since Btrfs will make this super easy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[David Sterba]

On Wed, Nov 02, 2016 at 09:19:10AM +0800, Qu Wenruo wrote:
> 
> 
> At 11/01/2016 06:08 PM, David Sterba wrote:
> > On Tue, Nov 01, 2016 at 04:01:43PM +0800, Qu Wenruo wrote:
> >> Introduce new function, escape_string_inplace(), to escape specified
> >> characters in place.
> >
> > Sorry, the pointer to seq_path was misleading. The actual escape
> > function is mangle_path and it copies one string to another. As we just
> > print the path, we can simply switch and call putchar.
> >
> 
> Putchar() method is indeed much easier to implement.
> 
> But it makes us hard to do further formatting, like aligning the path to 
> given width. (At least we are still using 32 chars alignment for path)
> 
> So I still prefer the current full function string escaping and still 
> use %-32s for formatting.
> 
> 
> And the idea of implementing escape_string_inplace() as a pure string 
> manipulation function can make it more agile for later use.
> For example, we can reuse it for print-tree.

Reusing is fine, but I really don't like that the function modifies the
argument. What if the function is called twice on the same string? Also,
in the print-tree, this would mean the extent buffer would be modified,
potentially overwriting other items.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 2/4] btrfs-progs: introduce new send-dump object

[David Sterba]

On Wed, Nov 02, 2016 at 08:37:20AM +0800, Qu Wenruo wrote:
> > PATH_WIDTH is used only here, please hardcode it into the format string.
> >
> > The rest of the patch looks good. I think I've seen some artifacts in
> > the output, but we can tune this later.
> 
> Would you like to share the artifacts so I can handle them in next update?

>From the output of misc test 016:

snapshot: [...] parent_transid= 14
   ^ extra space

set_xattr:  ./snap_creation/xattr_file  name=user.test, len=1
  ^ comma

also set xattr does not print the attribute data.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Eryu Guan]

On Wed, Nov 02, 2016 at 06:22:58PM +0800, Wang Xiaoguang wrote:
> hi Eryu,
> 
> There has already be a generic/102 doing this test...
> Thanks for you kindly review and sorry for wasting your time.

I had impression yesterday that we have a case that does exactly the
same test, and I searched but didn't find it.. It turns out that we do
have it already :)

Thanks for writing new test cases!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Wang Xiaoguang]

hi,

On 11/02/2016 09:27 AM, Dave Chinner wrote:

On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote:

In btrfs, sometimes though the number of created files' consumed disk space
are not larger than fs's free space, we can still get some ENOSPC error, it
may be that btrfs does not try hard to reclaim disk space(I have sent kernel
patch to resolve this kind of enospc error. Note, this false enospc error
will not always happen even in kernel without my fixing patch).

Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well.

.

+RUN_TIME=$((600 * $TIME_FACTOR))
+fs_size=$((15 * 1024 * 1024 * 1024))
+_scratch_mkfs_sized $fs_size > $seqres.full 2>&1
+_scratch_mount > $seqres.full 2>&1
+
+testfile1=$SCRATCH_MNT/testfile1
+testfile2=$SCRATCH_MNT/testfile2
+filesize1=$(((fs_size * 80) / 100))
+filesize2=$(((fs_size * 5) / 100))
+
+do_test()
+{
+   while [ -f $SCRATCH_MNT/run ]; do
+   $XFS_IO_PROG -fc "pwrite 0 $filesize1" $testfile1 > /dev/null
+   $XFS_IO_PROG -fc "pwrite 0 $filesize2" $testfile2 > /dev/null
+   rm -f $testfile1 $testfile2
+   done
+}

What are you trying to test here that other ENOSPC tests
don't exercise? Why cant you just use preallocation to trigger
ENOSPC repeatedly instead of writing data? That would allow multiple
iterations per second, not one every few minutes...

Yes, generic/102 just does this job, sorry.

Regards,
Xiaoguang Wang



Cheers,

Dave.




--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Wang Xiaoguang]

hi Eryu,

There has already be a generic/102 doing this test...
Thanks for you kindly review and sorry for wasting your time.

Regards,
Xiaoguang Wang

On 11/01/2016 08:26 PM, Eryu Guan wrote:

On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote:

In btrfs, sometimes though the number of created files' consumed disk space
are not larger than fs's free space, we can still get some ENOSPC error, it
may be that btrfs does not try hard to reclaim disk space(I have sent kernel
patch to resolve this kind of enospc error. Note, this false enospc error
will not always happen even in kernel without my fixing patch).

Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well.

Signed-off-by: Wang Xiaoguang 
---
  tests/generic/389 | 78 +++
  tests/generic/389.out |  2 ++
  tests/generic/group   |  1 +
  3 files changed, 81 insertions(+)
  create mode 100755 tests/generic/389
  create mode 100644 tests/generic/389.out

diff --git a/tests/generic/389 b/tests/generic/389
new file mode 100755
index 000..96bc12e
--- /dev/null
+++ b/tests/generic/389
@@ -0,0 +1,78 @@
+#! /bin/bash
+# FS QA Test 389
+#
+# Create and delete files repeatedly to exercise ENOSPC behaviour.

Trailing whitespace in this line.


+#
+#---
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1   # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+   cd /
+   rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+
+RUN_TIME=$((600 * $TIME_FACTOR))

Hmm, does it really need 600s to run? I think it's better to limit the
runtime within 300s and make it an 'auto' test. I, personally, prefer a
"loop count" based test, I'd find out a minimum loop count that could
reproduce the ENOSPC problem more reliably on btrfs (for example, say
75%) and make the count scale with LOAD_FACTOR.


+fs_size=$((15 * 1024 * 1024 * 1024))

And does it really need 15G on SCRATCH_DEV? A smaller fs size makes test
run faster, and gives the test more chance to be run, because not
everyone has a 15G SCRATCH_DEV.


+_scratch_mkfs_sized $fs_size > $seqres.full 2>&1
+_scratch_mount > $seqres.full 2>&1

Append to $seqres.full not overwrite.


+
+testfile1=$SCRATCH_MNT/testfile1
+testfile2=$SCRATCH_MNT/testfile2
+filesize1=$(((fs_size * 80) / 100))
+filesize2=$(((fs_size * 5) / 100))

Better to have some comments on the filesizes chosen here. e.g. someone
may wonder that why it's testing ENOSPC condition with 85% full, not 99%
or 100%.


+
+do_test()
+{
+   while [ -f $SCRATCH_MNT/run ]; do
+   $XFS_IO_PROG -fc "pwrite 0 $filesize1" $testfile1 > /dev/null
+   $XFS_IO_PROG -fc "pwrite 0 $filesize2" $testfile2 > /dev/null
+   rm -f $testfile1 $testfile2

Trailing whitespace here.


+   done
+}
+
+echo "Silence is golden"
+touch $SCRATCH_MNT/run
+do_test &
+sleep $RUN_TIME
+rm -f $SCRATCH_MNT/run
+wait
+
+status=0
+exit
diff --git a/tests/generic/389.out b/tests/generic/389.out
new file mode 100644
index 000..e8c24bb
--- /dev/null
+++ b/tests/generic/389.out
@@ -0,0 +1,2 @@
+QA output created by 389
+Silence is golden

Can you please rebase on top of current master? generic/389 is already
taken, and it makes applying & testing the patch a litter harder :)


diff --git a/tests/generic/group b/tests/generic/group
index fc32cfd..b6d4013 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -391,3 +391,4 @@
  386 auto quick quota
  387 auto clone
  388 auto log metadata
+389 enospc

Perhaps we can add it to 'rw' group too.

Thanks,
Eryu






--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resizing BTRFS - raw partition

[Christian Völker]

Yohoo!

Well, slightly off-topic now.

It is a Debian derivative. Univention Corporate Server.

uname -r says: 4.1.0-ucs190-amd64

So I am pretty fine with the kernel.

But indeed, I have to think about getting a more up-to-date btrfs as I
ran on resizing in the "File too large" issue so I had to use "max"
instead of "+100G". No big deal, just a minor issue.

I use btrfs to checksum all files stored there as I had several issues
the last years where files got corrupted.

Greetings

Christian


Am 02.11.2016 um 11:14 schrieb Adam Borowski:
> On Wed, Nov 02, 2016 at 09:29:26AM +, Hugo Mills wrote:
>> On Wed, Nov 02, 2016 at 10:18:03AM +0100, Christian Völker wrote:
>>> thanks for the quick reply. Regarding version- I prefer to use stable
>>> Linux versionsand I am not going to upgrade just btrfs outside of
>>> the verndors builds. So I am stuck happily with this version. And I run
>>> Linux since more than 10years, so I am really fine with it, I guess :D
>>Well, btrfs-progs 0.19 was last released several years ago. If your
>> kernel is of the same kind of age, then you're going to be seeing a
>> whole load of really nasty data-corrupting or filesystem-breaking bugs
>> which have since been fixed. Basically, if something goes wrong with
>> your FS when you're running a kernel that old, the main rsponse you'll
>> get is, "well, that was silly of you, wasn't it?", and you'll have to
>> make a new filesystem and restore from your backups and hope it
>> doesn't happen again.
> -progs 0.19 imply kernel 2.6.32 which comes from btrfs' infancy, when it
> was hardly merged into mainline.  It's buggier than experimental features
> like RAID5/6 nowadays.
>
>>I would currently recommend running a 4.4 kernel or later. If you
>> want a "stable" kernel version from a distribution, and want some kind
>> of support for it when it goes wrong, you're probably going to have to
>> pay someone (Red Hat or SuSE, most likely) to support your
>> configuraion.
> Kernels around 3.16 or so are pretty reliable -- ones I'm using on
> production are 3.13 and 3.14, without a single issue.
>
> As 2.6.32 is for you "stable" rather than "ancient and unsupported", I guess
> you're on RHEL or a derivative.  For them, 3.10 is the next stable, which
> is on the verge of what could be reasonable (but I still second Hugo's
> advice of using at least the current LTS kernel, ie 4.4).
>
> TL;DR:
> DO NOT USE BTRFS ON ANCIENT KERNELS!!1!elebenty-one!
>
>
> Meow!

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resizing BTRFS - raw partition

[Adam Borowski]

On Wed, Nov 02, 2016 at 09:29:26AM +, Hugo Mills wrote:
> On Wed, Nov 02, 2016 at 10:18:03AM +0100, Christian Völker wrote:
> > thanks for the quick reply. Regarding version- I prefer to use stable
> > Linux versionsand I am not going to upgrade just btrfs outside of
> > the verndors builds. So I am stuck happily with this version. And I run
> > Linux since more than 10years, so I am really fine with it, I guess :D
> 
>Well, btrfs-progs 0.19 was last released several years ago. If your
> kernel is of the same kind of age, then you're going to be seeing a
> whole load of really nasty data-corrupting or filesystem-breaking bugs
> which have since been fixed. Basically, if something goes wrong with
> your FS when you're running a kernel that old, the main rsponse you'll
> get is, "well, that was silly of you, wasn't it?", and you'll have to
> make a new filesystem and restore from your backups and hope it
> doesn't happen again.

-progs 0.19 imply kernel 2.6.32 which comes from btrfs' infancy, when it
was hardly merged into mainline.  It's buggier than experimental features
like RAID5/6 nowadays.

>I would currently recommend running a 4.4 kernel or later. If you
> want a "stable" kernel version from a distribution, and want some kind
> of support for it when it goes wrong, you're probably going to have to
> pay someone (Red Hat or SuSE, most likely) to support your
> configuraion.

Kernels around 3.16 or so are pretty reliable -- ones I'm using on
production are 3.13 and 3.14, without a single issue.

As 2.6.32 is for you "stable" rather than "ancient and unsupported", I guess
you're on RHEL or a derivative.  For them, 3.10 is the next stable, which
is on the verge of what could be reasonable (but I still second Hugo's
advice of using at least the current LTS kernel, ie 4.4).

TL;DR:
DO NOT USE BTRFS ON ANCIENT KERNELS!!1!elebenty-one!


Meow!
-- 
A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol, 1kg
raspberries, 0.4kg sugar; put into a big jar for 1 month.  Filter out and
throw away the fruits (can dump them into a cake, etc), let the drink age
at least 3-6 months.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resizing BTRFS - raw partition

[Hugo Mills]

On Wed, Nov 02, 2016 at 10:18:03AM +0100, Christian Völker wrote:
> Hi Hugo,
> 
> thanks for the quick reply. Regarding version- I prefer to use stable
> Linux versionsand I am not going to upgrade just btrfs outside of
> the verndors builds. So I am stuck happily with this version. And I run
> Linux since more than 10years, so I am really fine with it, I guess :D

   Well, btrfs-progs 0.19 was last released several years ago. If your
kernel is of the same kind of age, then you're going to be seeing a
whole load of really nasty data-corrupting or filesystem-breaking bugs
which have since been fixed. Basically, if something goes wrong with
your FS when you're running a kernel that old, the main rsponse you'll
get is, "well, that was silly of you, wasn't it?", and you'll have to
make a new filesystem and restore from your backups and hope it
doesn't happen again.

   I would currently recommend running a 4.4 kernel or later. If you
want a "stable" kernel version from a distribution, and want some kind
of support for it when it goes wrong, you're probably going to have to
pay someone (Red Hat or SuSE, most likely) to support your
configuraion.

   Hugo.

> And thanks again for your proposal. Yes, your command worked.
> 
> I had to tell betrfs the devid!
> 
> So this did NOT work:
> 
>  btrfs fi resize  max /srv/share/
> 
> Instead the following two commands worked:
> 
>  btrfs fi resize  1:max /srv/share/
>  btrfs fi resize  2:max /srv/share/
> 
> And now boths phydevices show the correct size.
> 
> This sound really strange for me that I have to tell btrfs to resize
> just a single disk insteag of automatically resizing all disks...I bet
> next time I have it forgotten again :-(
> 
> 
> Greetings
> 
> 
> Christian
> 
> 
> 

-- 
Hugo Mills | In one respect at least, the Martians are a happy
hugo@... carfax.org.uk | people: they have no lawyers
http://carfax.org.uk/  |
PGP: E2AB1DE4  |   John Carter, A Princess of Mars


signature.asc
Description: Digital signature

Re: Resizing BTRFS - raw partition

[Christian Völker]

Hi Hugo,

thanks for the quick reply. Regarding version- I prefer to use stable
Linux versionsand I am not going to upgrade just btrfs outside of
the verndors builds. So I am stuck happily with this version. And I run
Linux since more than 10years, so I am really fine with it, I guess :D

And thanks again for your proposal. Yes, your command worked.

I had to tell betrfs the devid!

So this did NOT work:

 btrfs fi resize  max /srv/share/

Instead the following two commands worked:

 btrfs fi resize  1:max /srv/share/
 btrfs fi resize  2:max /srv/share/

And now boths phydevices show the correct size.

This sound really strange for me that I have to tell btrfs to resize
just a single disk insteag of automatically resizing all disks...I bet
next time I have it forgotten again :-(


Greetings


Christian



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resizing BTRFS - raw partition

[Hugo Mills]

On Wed, Nov 02, 2016 at 09:58:41AM +0100, Christian Völker wrote:
> Hi all,
> 
> I am using btrfs as follows:
> root@srv:/srv# btrfs filesystem show
> Label: none  uuid: c8f24351-ddc4-4866-843c-4e95fcb498d4
> Total devices 2 FS bytes used 1005.37GB
> devid2 size 1.00TB used 1023.98GB path /dev/sdc
> devid1 size 1.46TB used 1.00TB path /dev/sdb
> 
> Btrfs Btrfs v0.19

   That's positively antique. If your kernel is anything close to the
same age, this experience with btrfs is probably going to be extremely
painful for you.

> It is running inside a virtual machines running on VMware ESXi. I
> increased both virtual disks to 1.5TB now. I did a scsi-rescan and fdisk
> -l tells me the new size:
> Disk /dev/sdb: 1610.6 GB, 1610612736000 bytes
> Disk /dev/sdc: 1610.6 GB, 1610612736000 bytes
> 
> There are no partitions created on the disks, just raw devices used for
> BTRFS. I found several sites where they resized a btrfs filesystem based
> on LVM and the new size was immediately recognized. But how to do on raw
> partitions?

   Than those sites are mistaken. :)

> How do I tell btrfs the devices have been resized?
> I did not find a rescan command. btrfs scan does not change anything. 
> Do I really have to reboot?

   btrfs fi resize :max /mountpoint

   btrfs dev scan is just used to tell the kernel which devices
contain [parts of] which btrfs filesystems.

   Hugo.

-- 
Hugo Mills | In one respect at least, the Martians are a happy
hugo@... carfax.org.uk | people: they have no lawyers
http://carfax.org.uk/  |
PGP: E2AB1DE4  |   John Carter, A Princess of Mars


signature.asc
Description: Digital signature

Resizing BTRFS - raw partition

[Christian Völker]

Hi all,

I am using btrfs as follows:
root@srv:/srv# btrfs filesystem show
Label: none  uuid: c8f24351-ddc4-4866-843c-4e95fcb498d4
Total devices 2 FS bytes used 1005.37GB
devid2 size 1.00TB used 1023.98GB path /dev/sdc
devid1 size 1.46TB used 1.00TB path /dev/sdb

Btrfs Btrfs v0.19

It is running inside a virtual machines running on VMware ESXi. I
increased both virtual disks to 1.5TB now. I did a scsi-rescan and fdisk
-l tells me the new size:
Disk /dev/sdb: 1610.6 GB, 1610612736000 bytes
Disk /dev/sdc: 1610.6 GB, 1610612736000 bytes

There are no partitions created on the disks, just raw devices used for
BTRFS. I found several sites where they resized a btrfs filesystem based
on LVM and the new size was immediately recognized. But how to do on raw
partitions?

How do I tell btrfs the devices have been resized?
I did not find a rescan command. btrfs scan does not change anything. 
Do I really have to reboot?

Thanks!

Christian

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html