[Qemu-block] [PATCH v3 3/3] tests: Add unit tests for image locking

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 tests/Makefile.include |   2 +
 tests/test-image-locking.c | 152 +
 2 files changed, 154 insertions(+)
 create mode 100644 tests/test-image-locking.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 760a0f18b6..8cc0595b39 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -95,6 +95,7 @@ check-unit-y += tests/test-bdrv-drain$(EXESUF)
 check-unit-y += tests/test-blockjob$(EXESUF)
 check-unit-y += tests/test-blockjob-txn$(EXESUF)
 check-unit-y += tests/test-block-backend$(EXESUF)
+check-unit-y += tests/test-image-locking$(EXESUF)
 check-unit-y += tests/test-x86-cpuid$(EXESUF)
 # all code tested by test-x86-cpuid is inside topology.h
 gcov-files-test-x86-cpuid-y =
@@ -640,6 +641,7 @@ tests/test-bdrv-drain$(EXESUF): tests/test-bdrv-drain.o 
$(test-block-obj-y) $(te
 tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) 
$(test-util-obj-y)
 tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o 
$(test-block-obj-y) $(test-util-obj-y)
 tests/test-block-backend$(EXESUF): tests/test-block-backend.o 
$(test-block-obj-y) $(test-util-obj-y)
+tests/test-image-locking$(EXESUF): tests/test-image-locking.o 
$(test-block-obj-y) $(test-util-obj-y)
 tests/test-thread-pool$(EXESUF): tests/test-thread-pool.o $(test-block-obj-y)
 tests/test-iov$(EXESUF): tests/test-iov.o $(test-util-obj-y)
 tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) 
$(test-crypto-obj-y)
diff --git a/tests/test-image-locking.c b/tests/test-image-locking.c
new file mode 100644
index 00..ccfa4e2c54
--- /dev/null
+++ b/tests/test-image-locking.c
@@ -0,0 +1,152 @@
+/*
+ * Image locking tests
+ *
+ * Copyright (c) 2018 Red Hat Inc.
+ *
+ * Author: Fam Zheng 
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "sysemu/block-backend.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+
+static BlockBackend *open_image(const char *path,
+uint64_t perm, uint64_t shared_perm,
+Error **errp)
+{
+Error *local_err = NULL;
+BlockBackend *blk;
+QDict *options = qdict_new();
+
+qdict_put_str(options, "driver", "raw");
+blk = blk_new_open(path, NULL, options, BDRV_O_RDWR, _err);
+if (blk) {
+g_assert_null(local_err);
+if (blk_set_perm(blk, perm, shared_perm, errp)) {
+blk_unref(blk);
+blk = NULL;
+}
+} else {
+error_propagate(errp, local_err);
+}
+return blk;
+}
+
+static void check_locked_bytes(int fd, uint64_t perm_locks,
+   uint64_t shared_perm_locks)
+{
+int i;
+
+if (!perm_locks && !shared_perm_locks) {
+g_assert(!qemu_lock_fd_test(fd, 0, 0, true));
+return;
+}
+for (i = 0; (1ULL << i) <= BLK_PERM_ALL; i++) {
+uint64_t bit = (1ULL << i);
+bool perm_expected = !!(bit & perm_locks);
+bool shared_perm_expected = !!(bit & shared_perm_locks);
+g_assert_cmpint(perm_expected, ==,
+!!qemu_lock_fd_test(fd, 100 + i, 1, true));
+g_assert_cmpint(shared_perm_expected, ==,
+!!qemu_lock_fd_test(fd, 200 + i, 1, true));
+}
+}
+
+static void test_image_locking_basic(void)
+{
+BlockBackend *blk1, *blk2, *blk3;
+char img_path[] = "/tmp/qtest.XX";
+uint64_t perm, shared_perm;
+
+int fd = mkstemp(img_path);
+assert(fd >= 0);
+
+perm = BLK_PERM_WRITE | BLK_PERM_CONSISTENT_READ;
+shared_perm = BLK_PERM_ALL;
+blk1 = open_image(img_path, perm, shared_perm, _abort);
+g_assert(blk1);
+
+check_locked_bytes(fd, perm, ~shared_perm);
+
+/* compatible perm between blk1 and blk2 */
+blk2 = open_image(img_path, perm | BLK_PERM_RESIZE, shared_perm, NULL);
+g_assert(blk2);
+

[Qemu-block] [PATCH v3 0/3] file-posix: Simplifications on image locking

[Fam Zheng]

The first patch reduces chances of QEMU crash in unusual (but not unlikely)
cases especially when used by Libvirt (see commit message).

The second patch halves fd for images.

The third adds some more test for patch one (would have caught the regression
caused by v2).

Fam Zheng (3):
  file-posix: Skip effectiveless OFD lock operations
  file-posix: Drop s->lock_fd
  tests: Add unit tests for image locking

 block/file-posix.c |  83 
 tests/Makefile.include |   2 +
 tests/test-image-locking.c | 152 +
 3 files changed, 207 insertions(+), 30 deletions(-)
 create mode 100644 tests/test-image-locking.c

-- 
2.17.1

[Qemu-block] [PATCH v3 1/3] file-posix: Skip effectiveless OFD lock operations

[Fam Zheng]

If we know we've already locked the bytes, don't do it again; similarly
don't unlock a byte if we haven't locked it. This doesn't change the
behavior, but fixes a corner case explained below.

Libvirt had an error handling bug that an image can get its (ownership,
file mode, SELinux) permissions changed (RHBZ 1584982) by mistake behind
QEMU. Specifically, an image in use by Libvirt VM has:

$ ls -lhZ b.img
-rw-r--r--. qemu qemu system_u:object_r:svirt_image_t:s0:c600,c690 b.img

Trying to attach it a second time won't work because of image locking.
And after the error, it becomes:

$ ls -lhZ b.img
-rw-r--r--. root root system_u:object_r:virt_image_t:s0 b.img

Then, we won't be able to do OFD lock operations with the existing fd.
In other words, the code such as in blk_detach_dev:

blk_set_perm(blk, 0, BLK_PERM_ALL, _abort);

can abort() QEMU, out of environmental changes.

This patch is an easy fix to this and the change is regardlessly
reasonable, so do it.

Signed-off-by: Fam Zheng 

---

v3: Don't misuse s->perm and s->shared_perm.
v2: For s == NULL, unlock all bits. [Kevin]
---
 block/file-posix.c | 54 +-
 1 file changed, 44 insertions(+), 10 deletions(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index fe83cbf0eb..f062e477e9 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -152,6 +152,11 @@ typedef struct BDRVRawState {
 uint64_t perm;
 uint64_t shared_perm;
 
+/* The perms bits whose corresponding bytes are already locked in
+ * s->lock_fd. */
+uint64_t locked_perm;
+uint64_t locked_shared_perm;
+
 #ifdef CONFIG_XFS
 bool is_xfs:1;
 #endif
@@ -680,43 +685,72 @@ typedef enum {
  * file; if @unlock == true, also unlock the unneeded bytes.
  * @shared_perm_lock_bits is the mask of all permissions that are NOT shared.
  */
-static int raw_apply_lock_bytes(int fd,
+static int raw_apply_lock_bytes(BDRVRawState *s, int fd,
 uint64_t perm_lock_bits,
 uint64_t shared_perm_lock_bits,
 bool unlock, Error **errp)
 {
 int ret;
 int i;
+uint64_t locked_perm, locked_shared_perm;
+
+if (s) {
+locked_perm = s->locked_perm;
+locked_shared_perm = s->locked_shared_perm;
+} else {
+/*
+ * We don't have the previous bits, just lock/unlock for each of the
+ * requested bits.
+ */
+if (unlock) {
+locked_perm = BLK_PERM_ALL;
+locked_shared_perm = BLK_PERM_ALL;
+} else {
+locked_perm = 0;
+locked_shared_perm = 0;
+}
+}
 
 PERM_FOREACH(i) {
 int off = RAW_LOCK_PERM_BASE + i;
-if (perm_lock_bits & (1ULL << i)) {
+uint64_t bit = (1ULL << i);
+if ((perm_lock_bits & bit) && !(locked_perm & bit)) {
 ret = qemu_lock_fd(fd, off, 1, false);
 if (ret) {
 error_setg(errp, "Failed to lock byte %d", off);
 return ret;
+} else if (s) {
+s->locked_perm |= bit;
 }
-} else if (unlock) {
+} else if (unlock && (locked_perm & bit) && !(perm_lock_bits & bit)) {
 ret = qemu_unlock_fd(fd, off, 1);
 if (ret) {
 error_setg(errp, "Failed to unlock byte %d", off);
 return ret;
+} else if (s) {
+s->locked_perm &= ~bit;
 }
 }
 }
 PERM_FOREACH(i) {
 int off = RAW_LOCK_SHARED_BASE + i;
-if (shared_perm_lock_bits & (1ULL << i)) {
+uint64_t bit = (1ULL << i);
+if ((shared_perm_lock_bits & bit) && !(locked_shared_perm & bit)) {
 ret = qemu_lock_fd(fd, off, 1, false);
 if (ret) {
 error_setg(errp, "Failed to lock byte %d", off);
 return ret;
+} else if (s) {
+s->locked_shared_perm |= bit;
 }
-} else if (unlock) {
+} else if (unlock && (locked_shared_perm & bit) &&
+   !(shared_perm_lock_bits & bit)) {
 ret = qemu_unlock_fd(fd, off, 1);
 if (ret) {
 error_setg(errp, "Failed to unlock byte %d", off);
 return ret;
+} else if (s) {
+s->locked_shared_perm &= ~bit;
 }
 }
 }
@@ -788,7 +822,7 @@ static int raw_handle_perm_lock(BlockDriverState *bs,
 
 switch (op) {
 case RAW_PL_PREPARE:
-ret = raw_apply_lock_bytes(s->lock_fd, s->perm | new_perm,
+ret = raw_apply_lock_bytes(s, s->lock_fd, s->perm | new_perm,
~s->shared_perm | ~new_shared,
false, errp);
 if (!ret) {
@@ -800,7 +834,7 @@ static int raw_handle_perm_lock(BlockDriverState *bs,
 op = RAW_PL_ABORT;
 /* fall through to unlock 

[Qemu-block] [PATCH v3 2/3] file-posix: Drop s->lock_fd

[Fam Zheng]

The lock_fd field is not strictly necessary because transferring locked
bytes from old fd to the new one shouldn't fail anyway. This spares the
user one fd per image.

Signed-off-by: Fam Zheng 
---
 block/file-posix.c | 37 +
 1 file changed, 13 insertions(+), 24 deletions(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index f062e477e9..26d4e487d2 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -142,7 +142,6 @@ do { \
 
 typedef struct BDRVRawState {
 int fd;
-int lock_fd;
 bool use_lock;
 int type;
 int open_flags;
@@ -153,7 +152,7 @@ typedef struct BDRVRawState {
 uint64_t shared_perm;
 
 /* The perms bits whose corresponding bytes are already locked in
- * s->lock_fd. */
+ * s->fd. */
 uint64_t locked_perm;
 uint64_t locked_shared_perm;
 
@@ -542,18 +541,6 @@ static int raw_open_common(BlockDriverState *bs, QDict 
*options,
 }
 s->fd = fd;
 
-s->lock_fd = -1;
-if (s->use_lock) {
-fd = qemu_open(filename, s->open_flags);
-if (fd < 0) {
-ret = -errno;
-error_setg_errno(errp, errno, "Could not open '%s' for locking",
- filename);
-qemu_close(s->fd);
-goto fail;
-}
-s->lock_fd = fd;
-}
 s->perm = 0;
 s->shared_perm = BLK_PERM_ALL;
 
@@ -818,15 +805,13 @@ static int raw_handle_perm_lock(BlockDriverState *bs,
 return 0;
 }
 
-assert(s->lock_fd > 0);
-
 switch (op) {
 case RAW_PL_PREPARE:
-ret = raw_apply_lock_bytes(s, s->lock_fd, s->perm | new_perm,
+ret = raw_apply_lock_bytes(s, s->fd, s->perm | new_perm,
~s->shared_perm | ~new_shared,
false, errp);
 if (!ret) {
-ret = raw_check_lock_bytes(s->lock_fd, new_perm, new_shared, errp);
+ret = raw_check_lock_bytes(s->fd, new_perm, new_shared, errp);
 if (!ret) {
 return 0;
 }
@@ -834,7 +819,7 @@ static int raw_handle_perm_lock(BlockDriverState *bs,
 op = RAW_PL_ABORT;
 /* fall through to unlock bytes. */
 case RAW_PL_ABORT:
-raw_apply_lock_bytes(s, s->lock_fd, s->perm, ~s->shared_perm,
+raw_apply_lock_bytes(s, s->fd, s->perm, ~s->shared_perm,
  true, _err);
 if (local_err) {
 /* Theoretically the above call only unlocks bytes and it cannot
@@ -844,7 +829,7 @@ static int raw_handle_perm_lock(BlockDriverState *bs,
 }
 break;
 case RAW_PL_COMMIT:
-raw_apply_lock_bytes(s, s->lock_fd, new_perm, ~new_shared,
+raw_apply_lock_bytes(s, s->fd, new_perm, ~new_shared,
  true, _err);
 if (local_err) {
 /* Theoretically the above call only unlocks bytes and it cannot
@@ -956,10 +941,18 @@ static void raw_reopen_commit(BDRVReopenState *state)
 {
 BDRVRawReopenState *rs = state->opaque;
 BDRVRawState *s = state->bs->opaque;
+Error *local_err = NULL;
 
 s->check_cache_dropped = rs->check_cache_dropped;
 s->open_flags = rs->open_flags;
 
+/* Copy locks to the new fd before closing the old one. */
+raw_apply_lock_bytes(NULL, rs->fd, s->locked_perm,
+ ~s->locked_shared_perm, false, _err);
+if (local_err) {
+/* shouldn't fail in a sane host, but report it just in case. */
+error_report_err(local_err);
+}
 qemu_close(s->fd);
 s->fd = rs->fd;
 
@@ -1952,10 +1945,6 @@ static void raw_close(BlockDriverState *bs)
 qemu_close(s->fd);
 s->fd = -1;
 }
-if (s->lock_fd >= 0) {
-qemu_close(s->lock_fd);
-s->lock_fd = -1;
-}
 }
 
 /**
-- 
2.17.1

Re: [Qemu-block] [PATCH 3/4] scripts/qemu: add render_block_graph method for QEMUMachine

[Eduardo Habkost]

On Thu, Aug 16, 2018 at 08:20:26PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Render block nodes graph with help of graphviz
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

Thanks for your patch.  Comments below:

> ---
>  scripts/qemu.py | 53 +
>  1 file changed, 53 insertions(+)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index f099ce7278..cff562c713 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -460,3 +460,56 @@ class QEMUMachine(object):
>   socket.SOCK_STREAM)
>  self._console_socket.connect(self._console_address)
>  return self._console_socket
> +
> +def render_block_graph(self, filename):
> +'''
> +Render graph in text (dot) representation into "filename" and 
> graphical
> +representation into pdf file "filename".pdf
> +'''
> +
> +try:
> +from graphviz import Digraph

I'm not convinced that this belongs to qemu.py, if most code
using the qemu.py module will never use it.  Do you see any
problem in moving this code to a scripts/render_block_graph.py
script?

> +except ImportError:
> +print "Can't import graphviz. Please run 'pip install graphviz'"

If you really want to make this part of qemu.py, I suggest
raising an appropriate exception instead of printing to stdout
directly.

(But just moving this code to a separate script would probably be
simpler)

Also, keep in mind that this module needs to work with both
Python 3 and Python 2, so you need to use print("message ...")
with parenthesis.


> +return
> +
> +nodes = self.qmp('query-named-block-nodes')['return']
> +edges = self.qmp('x-query-block-nodes-relations')['return']

I suggest you use .command() instead of .qmp().  It handles
['return'] and ['error'] for you.


> +node_names = []
> +
> +graph = Digraph(comment='Block Nodes Graph')
> +graph.node('permission symbols:\l'
> +   ' w - Write\l'
> +   ' r - consistent-Read\l'
> +   ' u - write - Unchanged\l'
> +   ' g - Graph-mod\l'
> +   ' s - reSize\l'
> +   'edge label scheme:\l'
> +   ' \l'
> +   ' \l'
> +   ' \l', shape='none')
> +
> +def perm(arr):
> +s = 'w' if 'write' in arr else '_'
> +s += 'r' if 'consistent-read' in arr else '_'
> +s += 'u' if 'write-unchanged' in arr else '_'
> +s += 'g' if 'graph-mod' in arr else '_'
> +s += 's' if 'resize' in arr else '_'
> +return s
> +
> +for n in nodes:
> +node_names.append(n['node-name'])
> +label = n['node-name'] + ' [' + n['drv'] + ']'
> +if n['drv'] == 'file':
> +label = '<%s%s>' % (label, os.path.basename(n['file']))
> +graph.node(n['node-name'], label)
> +
> +for r in edges:
> +if r['parent'] not in node_names:
> +graph.node(r['parent'], shape='box')
> +
> +label = '%s\l%s\l%s\l' % (r['name'], perm(r['perm']),
> +  perm(r['shared-perm']))
> +graph.edge(r['parent'], r['child'], label=label)
> +
> +graph.render(filename)

The rest of the code looks OK to me.

-- 
Eduardo

Re: [Qemu-block] [PATCH] qemu-img.c: increase spacing between commands in documentation

[Programmingkid]

> On Aug 14, 2018, at 4:40 AM, Kevin Wolf  wrote:
> 
> Am 13.08.2018 um 20:19 hat Eric Blake geschrieben:
>> On 08/13/2018 11:56 AM, Max Reitz wrote:
>>> 
>>> Ah, hm, so much for that.  Hm...  I don't quite know what to think of
>>> this.  It does indeed improve legibility.  But the question is whether
>>> --help should be as condensed as possible, and if the user finds it hard
>>> to read, whether they should not just open the man page...
>>> 
>>> Then again, our --help text already has 102 lines.
>>> 
>>> Let me just think about it for a bit longer. :-)
>>> (And see whether others have opinions.)
>> 
>> And I've already expressed my opinion that it is already rather long, where
>> making it longer is not necessarily making it smarter.
> 
> I think if we want to improve the help text, we should split it up.
> 
> $ qemu-img --help
> qemu-img version 2.12.94 (v3.0.0-rc4-5-g4fe9c3e13d-dirty)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> usage: qemu-img [standard options] command [command options]
> QEMU disk image utility
> 
>'-h', '--help'   display this help and exit
>'-V', '--version'output version information and exit
>'-T', '--trace'  [[enable=]][,events=][,file=]
> specify tracing options
> 
> Commands:
> 
>amend   Change options of an existing disk image
>bench   Run benchmarks on a given disk image
>check   Check the disk image for consistency or repair it
>commit  Merge the disk image into its backing file
>...
> 
> Run 'qemu-img  --help' for details.
> 
> See  for how to report bugs.
> More information on the QEMU project at .
> 
> $ qemu-img check --help
> usage: qemu-img check [--object objectdef] [--image-opts] [-q] [-f fmt] 
> [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename
> 
> Command parameters:
>  -f'fmt' is the disk image format. It is guessed automatically
>in most cases.
> 
>  -quse Quiet mode - do not print any output (except errors).
> 
>  -rtries to repair any inconsistencies that are found during the
>check. '-r leaks' repairs only cluster leaks, whereas '-r
>all' fixes all kinds of errors, with a higher risk of
>choosing the wrong fix or hiding corruption that has already
>occurred.
> 
>  -T'src_cache' is the cache mode used to read input disk images,
>the valid options are the same as for the 'cache' option.
> 
>  --object  'objectdef' is a QEMU user creatable object definition. See
>the qemu(1) manual page for a description of the object
>properties. The most common object type is a 'secret', which
>is used to supply passwords and/or encryption keys.
> 
>  --output  'ofmt' is 'human' for human-readable output (default) or
>'json' for JSON output.
> 
> Examples:
> 
>...
> 
> Kevin


I am by no means an expert at qemu-img. But I did try my best to create what I 
think should be the new output for qemu-img  --help. This is just the 
text I plan on using in a future patch. It is easier to read right now than it 
will be in patch form, so please let me know if there are any errors in this 
documentation. Thank you.




usage: qemu-img amend [--object objectdef] [--image-opts] [-p] [-q] [-f fmt]
[-t cache] -o options filename

Command parameters:
 -f 'fmt' is the disk image format. It is guessed automatically
in most cases.

--image-optsTreat filename as a set of image options, instead of a plain
filename.

 -o Used with a comma separated list of format specific options in a
name=value format. Use "-o ?" for an overview of the options
supported by the used format

--object'objectdef' is a QEMU user creatable object definition. See the
qemu(1) manual page for a description of the object properties.
The most common object type is a 'secret', which is used to
supply passwords and/or encryption keys.

 -p Display progress bar. If the -p option is not used for a command
that supports it, the progress is reported when the process
receives a "SIGUSR1" signal.

 -q Quiet mode - do not print any output (except errors). There's no
progress bar in case both -q and -p options are used.

 -t Specifies the cache mode that should be used with the
destination file.

Example:
qemu-img amend -o compat=0.10 -f qcow2 image.qcow2




usage: qemu-img bench [-c count] [-d depth] [-f fmt]
[--flush-interval=flush_interval] [-n] [--no-drain] [-o offset]
[--pattern=pattern] [-q] [-s buffer_size] [-S step_size] [-t cache] [-w] [-U]
filename

Command parameters:
 -c  Number of 

Re: [Qemu-block] [Qemu-devel] [PATCH 0/2] Improve qemu-img dd

[no-reply]

Hi,

This series failed docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180815025614.53588-1-ebl...@redhat.com
Subject: [Qemu-devel] [PATCH 0/2] Improve qemu-img dd

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-quick@centos7 SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
e66d307728 qemu-img: Add dd seek= option
cdb68a8e2e qemu-img: Fix dd with skip= and count=

=== OUTPUT BEGIN ===
  BUILD   centos7
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-gu7i1x1n/src'
  GEN 
/var/tmp/patchew-tester-tmp-gu7i1x1n/src/docker-src.2018-08-16-15.58.21.12455/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-gu7i1x1n/src/docker-src.2018-08-16-15.58.21.12455/qemu.tar.vroot'...
done.
Checking out files:  46% (2932/6321)   
Checking out files:  47% (2971/6321)   
Checking out files:  48% (3035/6321)   
Checking out files:  49% (3098/6321)   
Checking out files:  50% (3161/6321)   
Checking out files:  51% (3224/6321)   
Checking out files:  52% (3287/6321)   
Checking out files:  53% (3351/6321)   
Checking out files:  54% (3414/6321)   
Checking out files:  55% (3477/6321)   
Checking out files:  56% (3540/6321)   
Checking out files:  57% (3603/6321)   
Checking out files:  58% (3667/6321)   
Checking out files:  59% (3730/6321)   
Checking out files:  60% (3793/6321)   
Checking out files:  61% (3856/6321)   
Checking out files:  62% (3920/6321)   
Checking out files:  63% (3983/6321)   
Checking out files:  64% (4046/6321)   
Checking out files:  65% (4109/6321)   
Checking out files:  66% (4172/6321)   
Checking out files:  67% (4236/6321)   
Checking out files:  68% (4299/6321)   
Checking out files:  69% (4362/6321)   
Checking out files:  70% (4425/6321)   
Checking out files:  71% (4488/6321)   
Checking out files:  72% (4552/6321)   
Checking out files:  73% (4615/6321)   
Checking out files:  74% (4678/6321)   
Checking out files:  75% (4741/6321)   
Checking out files:  76% (4804/6321)   
Checking out files:  77% (4868/6321)   
Checking out files:  78% (4931/6321)   
Checking out files:  79% (4994/6321)   
Checking out files:  80% (5057/6321)   
Checking out files:  81% (5121/6321)   
Checking out files:  82% (5184/6321)   
Checking out files:  83% (5247/6321)   
Checking out files:  84% (5310/6321)   
Checking out files:  85% (5373/6321)   
Checking out files:  86% (5437/6321)   
Checking out files:  87% (5500/6321)   
Checking out files:  88% (5563/6321)   
Checking out files:  89% (5626/6321)   
Checking out files:  90% (5689/6321)   
Checking out files:  91% (5753/6321)   
Checking out files:  92% (5816/6321)   
Checking out files:  93% (5879/6321)   
Checking out files:  94% (5942/6321)   
Checking out files:  95% (6005/6321)   
Checking out files:  96% (6069/6321)   
Checking out files:  97% (6132/6321)   
Checking out files:  98% (6195/6321)   
Checking out files:  99% (6258/6321)   
Checking out files: 100% (6321/6321)   
Checking out files: 100% (6321/6321), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-gu7i1x1n/src/docker-src.2018-08-16-15.58.21.12455/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-gu7i1x1n/src/docker-src.2018-08-16-15.58.21.12455/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-quick in qemu:centos7 
Packages installed:
SDL-devel-1.2.15-14.el7.x86_64
bison-3.0.4-1.el7.x86_64
bzip2-devel-1.0.6-13.el7.x86_64
ccache-3.3.4-1.el7.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el7.x86_64
flex-2.5.37-3.el7.x86_64
gcc-4.8.5-16.el7_4.2.x86_64
gettext-0.19.8.1-2.el7.x86_64
git-1.8.3.1-12.el7_4.x86_64
glib2-devel-2.50.3-3.el7.x86_64
libepoxy-devel-1.3.1-1.el7.x86_64
libfdt-devel-1.4.6-1.el7.x86_64
lzo-devel-2.06-8.el7.x86_64
make-3.82-23.el7.x86_64
mesa-libEGL-devel-17.0.1-6.20170307.el7.x86_64
mesa-libgbm-devel-17.0.1-6.20170307.el7.x86_64
package g++ is not installed
package librdmacm-devel is not installed
pixman-devel-0.34.0-1.el7.x86_64
spice-glib-devel-0.33-6.el7_4.1.x86_64
spice-server-devel-0.12.8-2.el7.1.x86_64
tar-1.26-32.el7.x86_64
vte-devel-0.28.2-10.el7.x86_64
xen-devel-4.6.6-10.el7.x86_64
zlib-devel-1.2.7-17.el7.x86_64

Environment variables:
PACKAGES=bison bzip2-devel ccache csnappy-devel flex g++
 gcc gettext git glib2-devel libepoxy-devel libfdt-devel
 librdmacm-devel lzo-devel make mesa-libEGL-devel 
mesa-libgbm-devel pixman-devel SDL-devel 

Re: [Qemu-block] nbd oldstyle negotiation

[Eric Blake]

On 08/16/2018 02:02 PM, Vladimir Sementsov-Ogievskiy wrote:

Hi Eric!

There is a small problem with our qemu-nbd cmdline interface: people 
forget to use option -x or don't know about it and face into problems 
with old protocol version. One of may colleagues after such situation 
(because of this option, he could not use utility nbd-client, which 
doesn't support old version) asked me to add comments about version 
selection by -x option into --help of qemu-nbd. And really, it's not an 
obvious interface solution...


Now I think, should not we use new version by default? Or, even drop old 
version support at all? Anybody uses it?


nbdkit 1.3 switched its default to newstyle (Jan 2018):
https://github.com/libguestfs/nbdkit/commit/b2a8aecc
https://github.com/libguestfs/nbdkit/commit/8158e773

and you are correct that nbd 3.10 dropped oldstyle long ago (Mar 2015):
https://github.com/NetworkBlockDevice/nbd/commit/36940193

oldstyle is severely lacking in features (it can't do TLS or efficient 
sparse file handling, for example).  Do we need a formal qemu 
deprecation period (where you still get oldstyle if -x was not used, but 
a nice big warning) before flipping the default in 3.2/3.3 [1], or are 
we okay flipping it in 3.1?


[1] Okay, I know that our new equally-arbitrary policy on when to bump 
qemu version number components means that we'll probably see 3.1 in 2018 
then 4.0 in 2019, rather than 3.2.


And, if we switch the default now (which I am in favor of), do we want 
to add a -o switch to still make it possible to select oldstyle for a 
couple more releases in case someone was relying on it?  Then again, if 
you have to amend your script to use -o to keep things from breaking, 
why not just amend your setup to use newstyle?


My personal preference: completely drop oldstyle support from qemu, for 
3.1, with no deprecation period. Do so by making '-x ""' the default for 
any qemu-nbd command line that did not otherwise specify an export name. 
 If someone still needs interoperability, they can use nbdkit in the 
middle, which will still support oldstyle, and provides a plugin for 
connecting:


newstyle client => nbdkit nbd plugin => oldstyle server
oldstyle client => nbdkit nbd plugin => newstyle server

(well, that plugin could be enhanced to support IP addresses, rather 
than just Unix sockets, but that's a topic for another mailing list)


But I'm open to counter opinions on whether we need to be more 
conservative, and will not rip things out without at least getting 
opinions from others on how aggressive or conservative the switchover 
should be.




from nbd spec:
  ... the old style negotiation is now no longer developed; starting 
with version 3.10 of the reference implementation, it is also no longer 
supported.


it's unsupported, I think it's ok to drop it. I can create patches, if 
you agree.


If you want to create patches, that's also fine by me.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] [PATCH 0/2] Improve qemu-img dd

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180815025614.53588-1-ebl...@redhat.com
Subject: [Qemu-devel] [PATCH 0/2] Improve qemu-img dd

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
e66d307728 qemu-img: Add dd seek= option
cdb68a8e2e qemu-img: Fix dd with skip= and count=

=== OUTPUT BEGIN ===
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-3unl0b1q/src'
  GEN 
/var/tmp/patchew-tester-tmp-3unl0b1q/src/docker-src.2018-08-16-15.38.04.27074/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-3unl0b1q/src/docker-src.2018-08-16-15.38.04.27074/qemu.tar.vroot'...
done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-3unl0b1q/src/docker-src.2018-08-16-15.38.04.27074/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-3unl0b1q/src/docker-src.2018-08-16-15.38.04.27074/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-mingw in qemu:fedora 
Packages installed:
SDL2-devel-2.0.8-5.fc28.x86_64
bc-1.07.1-5.fc28.x86_64
bison-3.0.4-9.fc28.x86_64
bluez-libs-devel-5.49-3.fc28.x86_64
brlapi-devel-0.6.7-12.fc28.x86_64
bzip2-1.0.6-26.fc28.x86_64
bzip2-devel-1.0.6-26.fc28.x86_64
ccache-3.4.2-2.fc28.x86_64
clang-6.0.0-5.fc28.x86_64
device-mapper-multipath-devel-0.7.4-2.git07e7bd5.fc28.x86_64
findutils-4.6.0-19.fc28.x86_64
flex-2.6.1-7.fc28.x86_64
gcc-8.1.1-1.fc28.x86_64
gcc-c++-8.1.1-1.fc28.x86_64
gettext-0.19.8.1-14.fc28.x86_64
git-2.17.1-2.fc28.x86_64
glib2-devel-2.56.1-3.fc28.x86_64
glusterfs-api-devel-4.0.2-1.fc28.x86_64
gnutls-devel-3.6.2-1.fc28.x86_64
gtk3-devel-3.22.30-1.fc28.x86_64
hostname-3.20-3.fc28.x86_64
libaio-devel-0.3.110-11.fc28.x86_64
libasan-8.1.1-1.fc28.x86_64
libattr-devel-2.4.47-23.fc28.x86_64
libcap-devel-2.25-9.fc28.x86_64
libcap-ng-devel-0.7.9-1.fc28.x86_64
libcurl-devel-7.59.0-3.fc28.x86_64
libfdt-devel-1.4.6-4.fc28.x86_64
libpng-devel-1.6.34-3.fc28.x86_64
librbd-devel-12.2.5-1.fc28.x86_64
libssh2-devel-1.8.0-7.fc28.x86_64
libubsan-8.1.1-1.fc28.x86_64
libusbx-devel-1.0.21-6.fc28.x86_64
libxml2-devel-2.9.7-4.fc28.x86_64
llvm-6.0.0-11.fc28.x86_64
lzo-devel-2.08-12.fc28.x86_64
make-4.2.1-6.fc28.x86_64
mingw32-SDL2-2.0.5-3.fc27.noarch
mingw32-bzip2-1.0.6-9.fc27.noarch
mingw32-curl-7.57.0-1.fc28.noarch
mingw32-glib2-2.54.1-1.fc28.noarch
mingw32-gmp-6.1.2-2.fc27.noarch
mingw32-gnutls-3.5.13-2.fc27.noarch
mingw32-gtk3-3.22.16-1.fc27.noarch
mingw32-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw32-libpng-1.6.29-2.fc27.noarch
mingw32-libssh2-1.8.0-3.fc27.noarch
mingw32-libtasn1-4.13-1.fc28.noarch
mingw32-nettle-3.3-3.fc27.noarch
mingw32-pixman-0.34.0-3.fc27.noarch
mingw32-pkg-config-0.28-9.fc27.x86_64
mingw64-SDL2-2.0.5-3.fc27.noarch
mingw64-bzip2-1.0.6-9.fc27.noarch
mingw64-curl-7.57.0-1.fc28.noarch
mingw64-glib2-2.54.1-1.fc28.noarch
mingw64-gmp-6.1.2-2.fc27.noarch
mingw64-gnutls-3.5.13-2.fc27.noarch
mingw64-gtk3-3.22.16-1.fc27.noarch
mingw64-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw64-libpng-1.6.29-2.fc27.noarch
mingw64-libssh2-1.8.0-3.fc27.noarch
mingw64-libtasn1-4.13-1.fc28.noarch
mingw64-nettle-3.3-3.fc27.noarch
mingw64-pixman-0.34.0-3.fc27.noarch
mingw64-pkg-config-0.28-9.fc27.x86_64
ncurses-devel-6.1-5.20180224.fc28.x86_64
nettle-devel-3.4-2.fc28.x86_64
nss-devel-3.36.1-1.1.fc28.x86_64
numactl-devel-2.0.11-8.fc28.x86_64
package PyYAML is not installed
package libjpeg-devel is not installed
perl-5.26.2-411.fc28.x86_64
pixman-devel-0.34.0-8.fc28.x86_64
python3-3.6.5-1.fc28.x86_64
snappy-devel-1.1.7-5.fc28.x86_64
sparse-0.5.2-1.fc28.x86_64
spice-server-devel-0.14.0-4.fc28.x86_64
systemtap-sdt-devel-3.2-11.fc28.x86_64
tar-1.30-3.fc28.x86_64
usbredir-devel-0.7.1-7.fc28.x86_64
virglrenderer-devel-0.6.0-4.20170210git76b3da97b.fc28.x86_64
vte3-devel-0.36.5-6.fc28.x86_64
which-2.21-8.fc28.x86_64
xen-devel-4.10.1-3.fc28.x86_64
zlib-devel-1.2.11-8.fc28.x86_64

Environment variables:
TARGET_LIST=
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname 
gcc gcc-c++ llvm clang make perl which bc findutils glib2-devel 
libaio-devel pixman-devel zlib-devel libfdt-devel libasan libubsan 
bluez-libs-devel brlapi-devel bzip2-devel device-mapper-multipath-devel 
glusterfs-api-devel gnutls-devel gtk3-devel libattr-devel libcap-devel 
libcap-ng-devel libcurl-devel libjpeg-devel libpng-devel librbd-devel 
libssh2-devel libusbx-devel 

Re: [Qemu-block] [PATCH] doc: Permit BLOCK_STATUS reply to extend beyond request

[Eric Blake]

On 08/15/2018 04:23 AM, Vladimir Sementsov-Ogievskiy wrote:

ping.

Finally, Qemu 3.0 released, and it is incompatible with current NBD 
protocol. Please, commit this patch.


Indeed, qemu 3.0 with the qemu:dirty-bitmap:NAME context does exceed the 
final length when REQ_ONE is not in force.


As I recall, there was no complaint about the concept, only the wording; 
so I think this updated wording satisfies everyone:



   If the client used the `NBD_CMD_FLAG_REQ_ONE` flag in the request,
-  then every reply chunk MUST NOT contain more than one descriptor.
+  then every reply chunk MUST contain exactly one descriptor, and that
+  descriptor MUST NOT exceed the *length* of the original request.  If
+  the client did not use the flag, and the server replies with N
+  extents, then the sum of the *length* fields of the first N-1
+  extents (if any) MUST be less than the requested length, while the
+  *length* of the final extent MAY result in a sum larger than the
+  original requested length, if the server has that information anyway
+  as a side effect of reporting the status of the requested region.

I've gone ahead and pushed this.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[Qemu-block] nbd oldstyle negotiation

[Vladimir Sementsov-Ogievskiy]

Hi Eric!

There is a small problem with our qemu-nbd cmdline interface: people 
forget to use option -x or don't know about it and face into problems 
with old protocol version. One of may colleagues after such situation 
(because of this option, he could not use utility nbd-client, which 
doesn't support old version) asked me to add comments about version 
selection by -x option into --help of qemu-nbd. And really, it's not an 
obvious interface solution...


Now I think, should not we use new version by default? Or, even drop old 
version support at all? Anybody uses it?


from nbd spec:
 ... the old style negotiation is now no longer developed; starting 
with version 3.10 of the reference implementation, it is also no longer 
supported.


it's unsupported, I think it's ok to drop it. I can create patches, if 
you agree.


--
Best regards,
Vladimir

Re: [Qemu-block] [PATCH for-3.1 v10 02/31] block: Use children list in bdrv_refresh_filename

[Eric Blake]

On 08/09/2018 04:34 PM, Max Reitz wrote:

bdrv_refresh_filename() should invoke itself recursively on all
children, not just on file.

With that change, we can remove the manual invocations in blkverify,
quorum, commit, mirror, and blklogwrites.

Signed-off-by: Max Reitz 
---
  block.c  | 9 +
  block/blklogwrites.c | 3 ---
  block/blkverify.c| 3 ---
  block/commit.c   | 1 -
  block/mirror.c   | 1 -
  block/quorum.c   | 1 -
  6 files changed, 5 insertions(+), 13 deletions(-)


Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[Qemu-block] [PATCH v2 0/2] block: for jobs, do not clear user_paused until after the resume

[Jeff Cody]

v2 changes:

Patch 1: Added r-b from John, Eric (Thanks)
Patch 2: Attached an iotest as patch 2

* cc'ed qemu-stable

For the test in patch 2, failure here is the failure output w/o patch 1:

 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "testdisk"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "running", "id": "testdisk"}}
-{"return": {}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_ERROR", "data": {"device": "testdisk", "operation": "write", 
"action": "stop"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "testdisk"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_CANCELLED", "data": {"device": "testdisk", "len": 2097152, "offset": 
1048576, "speed": 0, "type": "mirror"}}
-*** done
+QEMU_PROG: blockjob.c:460: block_job_iostatus_reset: Assertion 
`job->job.user_paused && job->job.pause_count > 0' failed.
+Wrong response matching Assertion on handle 0
Failures: 229
Failed 1 of 1 tests


git-backport-diff from v1:

[] : patches are identical
[] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/2:[] [--] 'block: for jobs, do not clear user_paused until after the 
resume'
002/2:[down] 'block: iotest to catch abort on forced blockjob cancel'


Jeff Cody (2):
  block: for jobs, do not clear user_paused until after the resume
  block: iotest to catch abort on forced blockjob cancel

 job.c  |  2 +-
 tests/qemu-iotests/229 | 95 ++
 tests/qemu-iotests/229.out | 23 +
 tests/qemu-iotests/group   |  1 +
 4 files changed, 120 insertions(+), 1 deletion(-)
 create mode 100755 tests/qemu-iotests/229
 create mode 100644 tests/qemu-iotests/229.out

-- 
2.17.1

[Qemu-block] [PATCH v2 1/2] block: for jobs, do not clear user_paused until after the resume

[Jeff Cody]

The function job_cancel_async() will always cause an assert for blockjob
user resume.  We set job->user_paused to false, and then call
job->driver->user_resume().  In the case of blockjobs, this is the
block_job_user_resume() function.

In that function, we assert that job.user_paused is set to true.
Unfortunately, right before calling this function, it has explicitly
been set to false.

The fix is pretty simple: set job->user_paused to false only after the
job user_resume() function has been called.

Reviewed-by: John Snow 
Reviewed-by: Eric Blake 
Signed-off-by: Jeff Cody 
---
 job.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/job.c b/job.c
index fa671b431a..e36ebaafd8 100644
--- a/job.c
+++ b/job.c
@@ -732,10 +732,10 @@ static void job_cancel_async(Job *job, bool force)
 {
 if (job->user_paused) {
 /* Do not call job_enter here, the caller will handle it.  */
-job->user_paused = false;
 if (job->driver->user_resume) {
 job->driver->user_resume(job);
 }
+job->user_paused = false;
 assert(job->pause_count > 0);
 job->pause_count--;
 }
-- 
2.17.1

[Qemu-block] [PATCH v2 2/2] block: iotest to catch abort on forced blockjob cancel

[Jeff Cody]

Signed-off-by: Jeff Cody 
---
 tests/qemu-iotests/229 | 95 ++
 tests/qemu-iotests/229.out | 23 +
 tests/qemu-iotests/group   |  1 +
 3 files changed, 119 insertions(+)
 create mode 100755 tests/qemu-iotests/229
 create mode 100644 tests/qemu-iotests/229.out

diff --git a/tests/qemu-iotests/229 b/tests/qemu-iotests/229
new file mode 100755
index 00..2af04c8028
--- /dev/null
+++ b/tests/qemu-iotests/229
@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Test for force canceling a running blockjob that is paused in
+# an error state.
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=jc...@redhat.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+status=1   # failure is the default!
+
+_cleanup()
+{
+_cleanup_qemu
+_cleanup_test_img
+rm -f "$TEST_IMG" "$DEST_IMG"
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+# Needs backing file and backing format support
+_supported_fmt qcow2 qed
+_supported_proto file
+_supported_os Linux
+
+
+DEST_IMG="$TEST_DIR/d.$IMGFMT"
+TEST_IMG="$TEST_DIR/b.$IMGFMT"
+
+_make_test_img 2M
+
+# destination for mirror will be too small, causing error
+TEST_IMG=$DEST_IMG _make_test_img 1M
+
+$QEMU_IO -c 'write 0 2M' "$TEST_IMG" | _filter_qemu_io
+
+_launch_qemu -drive id=testdisk,file="$TEST_IMG",format="$IMGFMT"
+
+_send_qemu_cmd $QEMU_HANDLE \
+"{'execute': 'qmp_capabilities'}" \
+'return'
+
+echo
+echo '=== Starting drive-mirror, causing error & stop  ==='
+echo
+
+_send_qemu_cmd $QEMU_HANDLE \
+"{'execute': 'drive-mirror',
+ 'arguments': {'device': 'testdisk',
+   'mode':   'absolute-paths',
+   'format': 'qcow2',
+   'target': '$DEST_IMG',
+   'sync':   'full',
+   'mode':   'existing',
+   'on-source-error': 'stop',
+   'on-target-error': 'stop' }}"\
+ "BLOCK_JOB_ERROR"
+
+echo
+echo '=== Force cancel job paused in error state  ==='
+echo
+
+success_or_failure="y" _send_qemu_cmd $QEMU_HANDLE \
+"{'execute': 'block-job-cancel',
+ 'arguments': { 'device': 'testdisk',
+'force': true}}" \
+ "BLOCK_JOB_CANCELLED" "Assertion"
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/229.out b/tests/qemu-iotests/229.out
new file mode 100644
index 00..d5dea5cdd2
--- /dev/null
+++ b/tests/qemu-iotests/229.out
@@ -0,0 +1,23 @@
+QA output created by 229
+Formatting 'TEST_DIR/b.IMGFMT', fmt=IMGFMT size=2097152
+Formatting 'TEST_DIR/d.IMGFMT', fmt=IMGFMT size=1048576
+wrote 2097152/2097152 bytes at offset 0
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+{"return": {}}
+
+=== Starting drive-mirror, causing error & stop  ===
+
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "created", "id": "testdisk"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "running", "id": "testdisk"}}
+{"return": {}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_ERROR", "data": {"device": "testdisk", "operation": "write", 
"action": "stop"}}
+
+=== Force cancel job paused in error state  ===
+
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "testdisk"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "running", "id": "testdisk"}}
+{"return": {}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_ERROR", "data": {"device": "testdisk", "operation": "write", 
"action": "stop"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "testdisk"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_CANCELLED", "data": {"device": "testdisk", "len": 2097152, "offset": 

Re: [Qemu-block] [Qemu-devel] [PATCH 00/33] Qtest driver framework

[Emanuele]

On 15/08/2018 14:38, Markus Armbruster wrote:

Emanuele Giuseppe Esposito  writes:


Qgraph API for the qtest driver framework

This series of patches introduce a different qtest driver
organization, viewing machines, drivers and tests as node in a
graph, each having one or multiple edges relations.

The idea is to have a framework where each test asks for a specific
driver, and the framework takes care of allocating the proper devices
required and passing the correct command line arguments to QEMU.

A node can be of four types:
- MACHINE:   for example "arm/raspi2"
- DRIVER:for example "generic-sdhci"
- INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers)
- TEST:  for example "sdhci-test", consumes an interface and tests
  the functions provided

An edge relation between two nodes (drivers or machines) X and Y can be:
- X CONSUMES Y: Y can be plugged into X
- X PRODUCES Y: X provides the interface Y
- X CONTAINS Y: Y is part of X component

Basic framework steps are the following:
- All nodes and edges are created in their respective machine/driver/test files
- The framework starts QEMU and asks for a list of available devices
   and machines
- The framework walks the graph starting from the available machines and
   performs a Depth First Search for tests
- Once a test is found, the path is walked again and all drivers are
   allocated accordingly and the final interface is passed to the test
- The test is executed
- Unused objects are cleaned and the path discovery is continued

Depending on the QEMU binary used, only some drivers/machines will be available
and only test that are reached by them will be executed.

This work is being done as Google Summer of Code 2018 project for QEMU,
my mentors are Paolo Bonzini and Laurent Vivier.
Additional infos on the project can be found at:
https://wiki.qemu.org/Features/qtest_driver_framework

You've likely answered this question somewhere already, but it should be
answered right here: what does this framework buy us?

I figure the main benefit is running tests in all possible contexts,
with automated context setup.  Instead, you feed the framework
declarations of how things are related (the graph stuff described
above).
You're right, that is the main benefit: running tests in all possible 
contexts, with automated context setup.
Before, tests where basically hard-coded to run on a single 
architecture, or even a single machine type.
Now, qgraph allows this kind of higher level view of the devices and 
their relationships, where tests generally consume an interface that 
hides the underneath implementation (for example, for sdhci-pci device, 
it consumes a "pci-bus", without knowing that the bus can be a pci-pc or 
pci-spapr).


This allows developers to add test without knowing or forcing the 
machine/architecture/additional required devices, they just need to know 
what interface the test can consume (and if not present, implement a 
node representing the device they want to test).
For example, if you look at virtio-9p-test, you can see that the old 
test was just testing "virtio-9p-pci", while using the qgraph I was able 
to add "virtio-9p-device" too, without any additional change to the 
test, improving test coverage.


The other benefit is the automated context setup: as Paolo said once, 
every test basically re-invented the wheel in its own way. Every test 
had a main function, a section where qemu was started with proper 
devices, and a section where everything was cleaned up and shut down. 
Qgraph handles all these operations automatically, so the developer does 
not need to care about that. Just add the test function to the graph, 
and it will take care of running the test with the proper setup.


Of course this requires a minimal knowledge of the graph and the 
relationships you are going to use, since each test will receive 
structures representing device from the other nodes in the graph.

Taking a step back: your cover letter describes a solution at a high
level.  That's good.  But it should *first* describe the problem you're
trying to solve.

Do you think this explanation of the problem is fine?

I am not very experienced with patches, so in case it is, should I 
re-write the cover letter adding the above explanations or is fine like 
this? In case I should do it, should I submit the whole series again?

v3:
- Minor fixes regarding memory leaks and naming

Signed-off-by: Emanuele Giuseppe Esposito 

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

[Eric Blake]

On 08/16/2018 12:28 PM, Vladimir Sementsov-Ogievskiy wrote:
Hmm, how should I properly set based-on, if there two series under this 
one?


Based on:
    [PATCH v3 0/8] dirty-bitmap: rewrite bdrv_dirty_iter_next_area
    and
    [PATCH 0/2] block: make .bdrv_close optional


patchew goes off mail ids, so I'll rewrite the above in a 
machine-readable format:


Based-on: <20180814121443.33114-1-vsement...@virtuozzo.com>
Based-on: <20180814124320.39481-1-vsement...@virtuozzo.com>


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [PATCH 1/4] block: improve blk_root_get_parent_desc

[Vladimir Sementsov-Ogievskiy]

Hmm, now I think, that instead of this, it is better to use pointer as 
parent id for
nod-bds parents, to be sure they will not intersect. And add special 
field to

qapi block relation info "parent-description" for such parents.

Also I'm afraid that this patch may break iotests..

16.08.2018 20:20, Vladimir Sementsov-Ogievskiy wrote:

Make blk_root_get_parent_desc return different descriptions for
different blk's in worst case.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  block/block-backend.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/block-backend.c b/block/block-backend.c
index fa120630be..e5707a0f8c 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -141,7 +141,7 @@ static char *blk_root_get_parent_desc(BdrvChild *child)
  } else {
  /* TODO Callback into the BB owner for something more detailed */
  g_free(dev_id);
-return g_strdup("a block device");
+return g_strdup_printf("a block device %p", blk);
  }
  }
  



--
Best regards,
Vladimir

[Qemu-block] [PATCH 4/4] not-for-commit: example of new command usage for debugging

[Vladimir Sementsov-Ogievskiy]

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 tests/qemu-iotests/222 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222
index 0ead56d574..6e2d96cd72 100644
--- a/tests/qemu-iotests/222
+++ b/tests/qemu-iotests/222
@@ -137,6 +137,7 @@ with iotests.FilePath('base.img') as base_img_path, \
 log('--- Cleanup ---')
 log('')
 
+vm.render_block_graph('/tmp/out.dot')
 log(vm.qmp('block-job-cancel', device=src_node))
 log(vm.event_wait('BLOCK_JOB_CANCELLED'),
 filters=[iotests.filter_qmp_event])
-- 
2.11.1

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

[Vladimir Sementsov-Ogievskiy]

Hmm, how should I properly set based-on, if there two series under this one?

Based on:
   [PATCH v3 0/8] dirty-bitmap: rewrite bdrv_dirty_iter_next_area
   and
   [PATCH 0/2] block: make .bdrv_close optional




16.08.2018 18:05, no-re...@patchew.org wrote:

Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814170126.56461-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
fede2b479e new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
   BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-axs4wus0/src'
   GEN 
/var/tmp/patchew-tester-tmp-axs4wus0/src/docker-src.2018-08-16-11.04.31.27398/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-axs4wus0/src/docker-src.2018-08-16-11.04.31.27398/qemu.tar.vroot'...
done.
Checking out files:   8% (537/6322)
Checking out files:   9% (569/6322)
Checking out files:  10% (633/6322)
Checking out files:  11% (696/6322)
Checking out files:  12% (759/6322)
Checking out files:  13% (822/6322)
Checking out files:  13% (870/6322)
Checking out files:  14% (886/6322)
Checking out files:  15% (949/6322)
Checking out files:  16% (1012/6322)
Checking out files:  17% (1075/6322)
Checking out files:  18% (1138/6322)
Checking out files:  19% (1202/6322)
Checking out files:  20% (1265/6322)
Checking out files:  21% (1328/6322)
Checking out files:  22% (1391/6322)
Checking out files:  23% (1455/6322)
Checking out files:  24% (1518/6322)
Checking out files:  25% (1581/6322)
Checking out files:  26% (1644/6322)
Checking out files:  26% (1648/6322)
Checking out files:  27% (1707/6322)
Checking out files:  28% (1771/6322)
Checking out files:  29% (1834/6322)
Checking out files:  30% (1897/6322)
Checking out files:  31% (1960/6322)
Checking out files:  32% (2024/6322)
Checking out files:  33% (2087/6322)
Checking out files:  34% (2150/6322)
Checking out files:  35% (2213/6322)
Checking out files:  36% (2276/6322)
Checking out files:  37% (2340/6322)
Checking out files:  38% (2403/6322)
Checking out files:  39% (2466/6322)
Checking out files:  40% (2529/6322)
Checking out files:  41% (2593/6322)
Checking out files:  42% (2656/6322)
Checking out files:  43% (2719/6322)
Checking out files:  44% (2782/6322)
Checking out files:  45% (2845/6322)
Checking out files:  46% (2909/6322)
Checking out files:  47% (2972/6322)
Checking out files:  48% (3035/6322)
Checking out files:  49% (3098/6322)
Checking out files:  50% (3161/6322)
Checking out files:  51% (3225/6322)
Checking out files:  52% (3288/6322)
Checking out files:  53% (3351/6322)
Checking out files:  54% (3414/6322)
Checking out files:  55% (3478/6322)
Checking out files:  56% (3541/6322)
Checking out files:  57% (3604/6322)
Checking out files:  58% (3667/6322)
Checking out files:  59% (3730/6322)
Checking out files:  60% (3794/6322)
Checking out files:  61% (3857/6322)
Checking out files:  62% (3920/6322)
Checking out files:  63% (3983/6322)
Checking out files:  64% (4047/6322)
Checking out files:  65% (4110/6322)
Checking out files:  66% (4173/6322)
Checking out files:  67% (4236/6322)
Checking out files:  68% (4299/6322)
Checking out files:  69% (4363/6322)
Checking out files:  70% (4426/6322)
Checking out files:  71% (4489/6322)
Checking out files:  72% (4552/6322)
Checking out files:  73% (4616/6322)
Checking out files:  74% (4679/6322)
Checking out files:  75% (4742/6322)
Checking out files:  76% (4805/6322)
Checking out files:  77% (4868/6322)
Checking out files:  78% (4932/6322)
Checking out files:  79% (4995/6322)
Checking out files:  80% (5058/6322)
Checking out files:  81% (5121/6322)
Checking out files:  82% (5185/6322)
Checking out files:  83% (5248/6322)
Checking out files:  84% (5311/6322)
Checking out files:  85% (5374/6322)
Checking out files:  86% (5437/6322)
Checking out files:  87% (5501/6322)
Checking out files:  88% (5564/6322)
Checking out files:  89% (5627/6322)
Checking out files:  90% (5690/6322)
Checking out files:  91% (5754/6322)
Checking out files:  92% (5817/6322)
Checking out files:  93% (5880/6322)
Checking out files:  94% (5943/6322)
Checking out files:  95% (6006/6322)
Checking out files:  96% (6070/6322)
Checking out files:  97% (6133/6322)
Checking out files:  98% (6196/6322)
Checking out files:  99% (6259/6322)
Checking out files:  99% (6321/6322)
Checking out files: 100% (6322/6322)
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-axs4wus0/src/docker-src.2018-08-16-11.04.31.27398/qemu.tar.vroot/dtc'...

[Qemu-block] [PATCH 3/4] scripts/qemu: add render_block_graph method for QEMUMachine

[Vladimir Sementsov-Ogievskiy]

Render block nodes graph with help of graphviz

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 scripts/qemu.py | 53 +
 1 file changed, 53 insertions(+)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index f099ce7278..cff562c713 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -460,3 +460,56 @@ class QEMUMachine(object):
  socket.SOCK_STREAM)
 self._console_socket.connect(self._console_address)
 return self._console_socket
+
+def render_block_graph(self, filename):
+'''
+Render graph in text (dot) representation into "filename" and graphical
+representation into pdf file "filename".pdf
+'''
+
+try:
+from graphviz import Digraph
+except ImportError:
+print "Can't import graphviz. Please run 'pip install graphviz'"
+return
+
+nodes = self.qmp('query-named-block-nodes')['return']
+edges = self.qmp('x-query-block-nodes-relations')['return']
+node_names = []
+
+graph = Digraph(comment='Block Nodes Graph')
+graph.node('permission symbols:\l'
+   ' w - Write\l'
+   ' r - consistent-Read\l'
+   ' u - write - Unchanged\l'
+   ' g - Graph-mod\l'
+   ' s - reSize\l'
+   'edge label scheme:\l'
+   ' \l'
+   ' \l'
+   ' \l', shape='none')
+
+def perm(arr):
+s = 'w' if 'write' in arr else '_'
+s += 'r' if 'consistent-read' in arr else '_'
+s += 'u' if 'write-unchanged' in arr else '_'
+s += 'g' if 'graph-mod' in arr else '_'
+s += 's' if 'resize' in arr else '_'
+return s
+
+for n in nodes:
+node_names.append(n['node-name'])
+label = n['node-name'] + ' [' + n['drv'] + ']'
+if n['drv'] == 'file':
+label = '<%s%s>' % (label, os.path.basename(n['file']))
+graph.node(n['node-name'], label)
+
+for r in edges:
+if r['parent'] not in node_names:
+graph.node(r['parent'], shape='box')
+
+label = '%s\l%s\l%s\l' % (r['name'], perm(r['perm']),
+  perm(r['shared-perm']))
+graph.edge(r['parent'], r['child'], label=label)
+
+graph.render(filename)
-- 
2.11.1

[Qemu-block] [PATCH 2/4] qapi: add x-query-block-nodes-relations

[Vladimir Sementsov-Ogievskiy]

Add new command, returning list of block nodes graph edges.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 qapi/block-core.json  | 71 +++
 include/block/block.h |  1 +
 block.c   | 57 +
 blockdev.c|  5 
 4 files changed, 134 insertions(+)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 4c7a37afdc..ad7b62c49b 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1630,6 +1630,77 @@
 { 'command': 'query-named-block-nodes', 'returns': [ 'BlockDeviceInfo' ] }
 
 ##
+# @BlockPermission:
+#
+# Enum of base block permissions.
+#
+# @consistent-read: A user that has the "permission" of consistent reads is
+#   guaranteed that their view of the contents of the block
+#   device is complete and self-consistent, representing the
+#   contents of a disk at a specific point.
+#   For most block devices (including their backing files) this
+#   is true, but the property cannot be maintained in a few
+#   situations like for intermediate nodes of a commit block
+#   job.
+#
+# @write: This permission is required to change the visible disk contents.
+#
+# @write-unchanged: This permission (which is weaker than BLK_PERM_WRITE) is
+#   both enough and required for writes to the block node when
+#   the caller promises that the visible disk content doesn't
+#   change.
+#   As the BLK_PERM_WRITE permission is strictly stronger,
+#   either is sufficient to perform an unchanging write.
+#
+# @resize: This permission is required to change the size of a block node.
+#
+# @graph-mod: This permission is required to change the node that this
+# BdrvChild points to.
+#
+# Since: 3.1
+##
+{ 'enum': 'BlockPermission',
+  'data': [ 'consistent-read', 'write', 'write-unchanged', 'resize',
+'graph-mod' ] }
+
+##
+# @BlockRelationInfo:
+#
+# Information about relation between block node and its parent.
+#
+# @parent: node name or some other parent name/description, if parent is not a
+#  block node.
+#
+# @parent-is-bds: parent is block node.
+#
+# @child: node name
+#
+# @name: name of the relation (examples are 'file' and 'backing')
+#
+# @perm: Granted permissions for the parent operating on the child.
+#
+# @shared-perm: Permissions that can still be granted to other users of the
+#   child while it is still attached this parent.
+#
+# Since: 3.1
+##
+{ 'struct': 'BlockRelationInfo',
+  'data': { 'parent': 'str', 'parent-is-bds': 'bool', 'child': 'str',
+'name': 'str', 'perm': [ 'BlockPermission' ],
+'shared-perm': [ 'BlockPermission' ] } }
+
+##
+# @x-query-block-nodes-relations:
+#
+# Get the block relations list.
+#
+# Returns: the list of BlockRelationInfo.
+#
+# Since: 3.1
+##
+{ 'command': 'x-query-block-nodes-relations', 'returns': [ 'BlockRelationInfo' 
] }
+
+##
 # @drive-mirror:
 #
 # Start mirroring a block device's writes to a new destination. target
diff --git a/include/block/block.h b/include/block/block.h
index 4e0871aaf9..04136634c2 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -448,6 +448,7 @@ void bdrv_eject(BlockDriverState *bs, bool eject_flag);
 const char *bdrv_get_format_name(BlockDriverState *bs);
 BlockDriverState *bdrv_find_node(const char *node_name);
 BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp);
+BlockRelationInfoList *bdrv_block_relations_list(Error **errp);
 BlockDriverState *bdrv_lookup_bs(const char *device,
  const char *node_name,
  Error **errp);
diff --git a/block.c b/block.c
index 6161dbe3eb..776fc714af 100644
--- a/block.c
+++ b/block.c
@@ -4003,6 +4003,63 @@ BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp)
 return list;
 }
 
+BlockRelationInfoList *bdrv_block_relations_list(Error **errp)
+{
+BlockRelationInfoList *list = NULL, *entry;
+BlockDriverState *bs;
+struct {
+unsigned int flag;
+BlockPermission num;
+} permissions[] = {
+{ BLK_PERM_CONSISTENT_READ, BLOCK_PERMISSION_CONSISTENT_READ },
+{ BLK_PERM_WRITE,   BLOCK_PERMISSION_WRITE },
+{ BLK_PERM_WRITE_UNCHANGED, BLOCK_PERMISSION_WRITE_UNCHANGED },
+{ BLK_PERM_RESIZE,  BLOCK_PERMISSION_RESIZE },
+{ BLK_PERM_GRAPH_MOD,   BLOCK_PERMISSION_GRAPH_MOD },
+{ 0, 0 }
+}, *p;
+
+QTAILQ_FOREACH(bs, _bdrv_states, node_list) {
+BdrvChild *child;
+
+QLIST_FOREACH(child, >parents, next_parent) {
+entry = g_new0(BlockRelationInfoList, 1);
+BlockRelationInfo *info = g_new0(BlockRelationInfo, 1);
+
+info->parent_is_bds = child->role->parent_is_bds;
+info->parent = 

[Qemu-block] [PATCH 1/4] block: improve blk_root_get_parent_desc

[Vladimir Sementsov-Ogievskiy]

Make blk_root_get_parent_desc return different descriptions for
different blk's in worst case.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/block-backend.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/block-backend.c b/block/block-backend.c
index fa120630be..e5707a0f8c 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -141,7 +141,7 @@ static char *blk_root_get_parent_desc(BdrvChild *child)
 } else {
 /* TODO Callback into the BB owner for something more detailed */
 g_free(dev_id);
-return g_strdup("a block device");
+return g_strdup_printf("a block device %p", blk);
 }
 }
 
-- 
2.11.1

[Qemu-block] [PATCH 0/4] block nodes graph visualization

[Vladimir Sementsov-Ogievskiy]

Hi all!

On the way of backup schemes development (and in general any complicated
developments in Qemu block layer) it would be good to have a way to print
out graph of block nodes with their permissions. Here is this way.

Instead of a lot of words I'll reply to this cover letter with example
graph, generated in patch 04.

Vladimir Sementsov-Ogievskiy (4):
  block: improve blk_root_get_parent_desc
  qapi: add x-query-block-nodes-relations
  scripts/qemu: add render_block_graph method for QEMUMachine
  not-for-commit: example of new command usage for debugging

 qapi/block-core.json   | 71 ++
 include/block/block.h  |  1 +
 block.c| 57 
 block/block-backend.c  |  2 +-
 blockdev.c |  5 
 scripts/qemu.py| 53 +
 tests/qemu-iotests/222 |  1 +
 7 files changed, 189 insertions(+), 1 deletion(-)

-- 
2.11.1

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814170126.56461-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
fede2b479e new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-axs4wus0/src'
  GEN 
/var/tmp/patchew-tester-tmp-axs4wus0/src/docker-src.2018-08-16-11.04.31.27398/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-axs4wus0/src/docker-src.2018-08-16-11.04.31.27398/qemu.tar.vroot'...
done.
Checking out files:   8% (537/6322)   
Checking out files:   9% (569/6322)   
Checking out files:  10% (633/6322)   
Checking out files:  11% (696/6322)   
Checking out files:  12% (759/6322)   
Checking out files:  13% (822/6322)   
Checking out files:  13% (870/6322)   
Checking out files:  14% (886/6322)   
Checking out files:  15% (949/6322)   
Checking out files:  16% (1012/6322)   
Checking out files:  17% (1075/6322)   
Checking out files:  18% (1138/6322)   
Checking out files:  19% (1202/6322)   
Checking out files:  20% (1265/6322)   
Checking out files:  21% (1328/6322)   
Checking out files:  22% (1391/6322)   
Checking out files:  23% (1455/6322)   
Checking out files:  24% (1518/6322)   
Checking out files:  25% (1581/6322)   
Checking out files:  26% (1644/6322)   
Checking out files:  26% (1648/6322)   
Checking out files:  27% (1707/6322)   
Checking out files:  28% (1771/6322)   
Checking out files:  29% (1834/6322)   
Checking out files:  30% (1897/6322)   
Checking out files:  31% (1960/6322)   
Checking out files:  32% (2024/6322)   
Checking out files:  33% (2087/6322)   
Checking out files:  34% (2150/6322)   
Checking out files:  35% (2213/6322)   
Checking out files:  36% (2276/6322)   
Checking out files:  37% (2340/6322)   
Checking out files:  38% (2403/6322)   
Checking out files:  39% (2466/6322)   
Checking out files:  40% (2529/6322)   
Checking out files:  41% (2593/6322)   
Checking out files:  42% (2656/6322)   
Checking out files:  43% (2719/6322)   
Checking out files:  44% (2782/6322)   
Checking out files:  45% (2845/6322)   
Checking out files:  46% (2909/6322)   
Checking out files:  47% (2972/6322)   
Checking out files:  48% (3035/6322)   
Checking out files:  49% (3098/6322)   
Checking out files:  50% (3161/6322)   
Checking out files:  51% (3225/6322)   
Checking out files:  52% (3288/6322)   
Checking out files:  53% (3351/6322)   
Checking out files:  54% (3414/6322)   
Checking out files:  55% (3478/6322)   
Checking out files:  56% (3541/6322)   
Checking out files:  57% (3604/6322)   
Checking out files:  58% (3667/6322)   
Checking out files:  59% (3730/6322)   
Checking out files:  60% (3794/6322)   
Checking out files:  61% (3857/6322)   
Checking out files:  62% (3920/6322)   
Checking out files:  63% (3983/6322)   
Checking out files:  64% (4047/6322)   
Checking out files:  65% (4110/6322)   
Checking out files:  66% (4173/6322)   
Checking out files:  67% (4236/6322)   
Checking out files:  68% (4299/6322)   
Checking out files:  69% (4363/6322)   
Checking out files:  70% (4426/6322)   
Checking out files:  71% (4489/6322)   
Checking out files:  72% (4552/6322)   
Checking out files:  73% (4616/6322)   
Checking out files:  74% (4679/6322)   
Checking out files:  75% (4742/6322)   
Checking out files:  76% (4805/6322)   
Checking out files:  77% (4868/6322)   
Checking out files:  78% (4932/6322)   
Checking out files:  79% (4995/6322)   
Checking out files:  80% (5058/6322)   
Checking out files:  81% (5121/6322)   
Checking out files:  82% (5185/6322)   
Checking out files:  83% (5248/6322)   
Checking out files:  84% (5311/6322)   
Checking out files:  85% (5374/6322)   
Checking out files:  86% (5437/6322)   
Checking out files:  87% (5501/6322)   
Checking out files:  88% (5564/6322)   
Checking out files:  89% (5627/6322)   
Checking out files:  90% (5690/6322)   
Checking out files:  91% (5754/6322)   
Checking out files:  92% (5817/6322)   
Checking out files:  93% (5880/6322)   
Checking out files:  94% (5943/6322)   
Checking out files:  95% (6006/6322)   
Checking out files:  96% (6070/6322)   
Checking out files:  97% (6133/6322)   
Checking out files:  98% (6196/6322)   
Checking out files:  99% (6259/6322)   
Checking out files:  99% (6321/6322)   
Checking out files: 100% (6322/6322)   
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 

Re: [Qemu-block] [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814165349.56344-2-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-quick@centos7 SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
d0a4cf13bb new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   centos7
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-qje15dz0/src'
  GEN 
/var/tmp/patchew-tester-tmp-qje15dz0/src/docker-src.2018-08-16-11.13.18.22968/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-qje15dz0/src/docker-src.2018-08-16-11.13.18.22968/qemu.tar.vroot'...
done.
Checking out files:  48% (3070/6322)   
Checking out files:  49% (3098/6322)   
Checking out files:  50% (3161/6322)   
Checking out files:  51% (3225/6322)   
Checking out files:  52% (3288/6322)   
Checking out files:  53% (3351/6322)   
Checking out files:  54% (3414/6322)   
Checking out files:  55% (3478/6322)   
Checking out files:  56% (3541/6322)   
Checking out files:  57% (3604/6322)   
Checking out files:  58% (3667/6322)   
Checking out files:  59% (3730/6322)   
Checking out files:  60% (3794/6322)   
Checking out files:  61% (3857/6322)   
Checking out files:  62% (3920/6322)   
Checking out files:  63% (3983/6322)   
Checking out files:  64% (4047/6322)   
Checking out files:  65% (4110/6322)   
Checking out files:  66% (4173/6322)   
Checking out files:  67% (4236/6322)   
Checking out files:  68% (4299/6322)   
Checking out files:  69% (4363/6322)   
Checking out files:  70% (4426/6322)   
Checking out files:  71% (4489/6322)   
Checking out files:  72% (4552/6322)   
Checking out files:  73% (4616/6322)   
Checking out files:  74% (4679/6322)   
Checking out files:  75% (4742/6322)   
Checking out files:  76% (4805/6322)   
Checking out files:  77% (4868/6322)   
Checking out files:  78% (4932/6322)   
Checking out files:  79% (4995/6322)   
Checking out files:  80% (5058/6322)   
Checking out files:  81% (5121/6322)   
Checking out files:  82% (5185/6322)   
Checking out files:  83% (5248/6322)   
Checking out files:  84% (5311/6322)   
Checking out files:  85% (5374/6322)   
Checking out files:  86% (5437/6322)   
Checking out files:  87% (5501/6322)   
Checking out files:  88% (5564/6322)   
Checking out files:  89% (5627/6322)   
Checking out files:  90% (5690/6322)   
Checking out files:  91% (5754/6322)   
Checking out files:  92% (5817/6322)   
Checking out files:  93% (5880/6322)   
Checking out files:  94% (5943/6322)   
Checking out files:  95% (6006/6322)   
Checking out files:  96% (6070/6322)   
Checking out files:  97% (6133/6322)   
Checking out files:  98% (6196/6322)   
Checking out files:  99% (6259/6322)   
Checking out files: 100% (6322/6322)   
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-qje15dz0/src/docker-src.2018-08-16-11.13.18.22968/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-qje15dz0/src/docker-src.2018-08-16-11.13.18.22968/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-quick in qemu:centos7 
Packages installed:
SDL-devel-1.2.15-14.el7.x86_64
bison-3.0.4-1.el7.x86_64
bzip2-devel-1.0.6-13.el7.x86_64
ccache-3.3.4-1.el7.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el7.x86_64
flex-2.5.37-3.el7.x86_64
gcc-4.8.5-16.el7_4.2.x86_64
gettext-0.19.8.1-2.el7.x86_64
git-1.8.3.1-12.el7_4.x86_64
glib2-devel-2.50.3-3.el7.x86_64
libepoxy-devel-1.3.1-1.el7.x86_64
libfdt-devel-1.4.6-1.el7.x86_64
lzo-devel-2.06-8.el7.x86_64
make-3.82-23.el7.x86_64
mesa-libEGL-devel-17.0.1-6.20170307.el7.x86_64
mesa-libgbm-devel-17.0.1-6.20170307.el7.x86_64
package g++ is not installed
package librdmacm-devel is not installed
pixman-devel-0.34.0-1.el7.x86_64
spice-glib-devel-0.33-6.el7_4.1.x86_64
spice-server-devel-0.12.8-2.el7.1.x86_64
tar-1.26-32.el7.x86_64
vte-devel-0.28.2-10.el7.x86_64
xen-devel-4.6.6-10.el7.x86_64
zlib-devel-1.2.7-17.el7.x86_64

Environment variables:
PACKAGES=bison bzip2-devel ccache csnappy-devel flex g++
 gcc gettext git glib2-devel libepoxy-devel libfdt-devel
 librdmacm-devel lzo-devel make mesa-libEGL-devel 
mesa-libgbm-devel pixman-devel SDL-devel spice-glib-devel 
spice-server-devel tar vte-devel xen-devel zlib-devel

Re: [Qemu-block] [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814165349.56344-2-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
d0a4cf13bb new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-nry751rd/src'
  GEN 
/var/tmp/patchew-tester-tmp-nry751rd/src/docker-src.2018-08-16-11.09.35.11313/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-nry751rd/src/docker-src.2018-08-16-11.09.35.11313/qemu.tar.vroot'...
done.
Checking out files:   7% (487/6322)   
Checking out files:   8% (506/6322)   
Checking out files:   9% (569/6322)   
Checking out files:  10% (633/6322)   
Checking out files:  11% (696/6322)   
Checking out files:  12% (759/6322)   
Checking out files:  13% (822/6322)   
Checking out files:  14% (886/6322)   
Checking out files:  15% (949/6322)   
Checking out files:  16% (1012/6322)   
Checking out files:  17% (1075/6322)   
Checking out files:  18% (1138/6322)   
Checking out files:  19% (1202/6322)   
Checking out files:  20% (1265/6322)   
Checking out files:  21% (1328/6322)   
Checking out files:  22% (1391/6322)   
Checking out files:  23% (1455/6322)   
Checking out files:  24% (1518/6322)   
Checking out files:  25% (1581/6322)   
Checking out files:  26% (1644/6322)   
Checking out files:  27% (1707/6322)   
Checking out files:  27% (1718/6322)   
Checking out files:  28% (1771/6322)   
Checking out files:  29% (1834/6322)   
Checking out files:  30% (1897/6322)   
Checking out files:  31% (1960/6322)   
Checking out files:  32% (2024/6322)   
Checking out files:  33% (2087/6322)   
Checking out files:  34% (2150/6322)   
Checking out files:  35% (2213/6322)   
Checking out files:  36% (2276/6322)   
Checking out files:  37% (2340/6322)   
Checking out files:  38% (2403/6322)   
Checking out files:  39% (2466/6322)   
Checking out files:  40% (2529/6322)   
Checking out files:  41% (2593/6322)   
Checking out files:  42% (2656/6322)   
Checking out files:  43% (2719/6322)   
Checking out files:  44% (2782/6322)   
Checking out files:  45% (2845/6322)   
Checking out files:  46% (2909/6322)   
Checking out files:  47% (2972/6322)   
Checking out files:  48% (3035/6322)   
Checking out files:  49% (3098/6322)   
Checking out files:  50% (3161/6322)   
Checking out files:  51% (3225/6322)   
Checking out files:  52% (3288/6322)   
Checking out files:  53% (3351/6322)   
Checking out files:  54% (3414/6322)   
Checking out files:  55% (3478/6322)   
Checking out files:  56% (3541/6322)   
Checking out files:  57% (3604/6322)   
Checking out files:  58% (3667/6322)   
Checking out files:  59% (3730/6322)   
Checking out files:  60% (3794/6322)   
Checking out files:  61% (3857/6322)   
Checking out files:  62% (3920/6322)   
Checking out files:  63% (3983/6322)   
Checking out files:  64% (4047/6322)   
Checking out files:  65% (4110/6322)   
Checking out files:  66% (4173/6322)   
Checking out files:  67% (4236/6322)   
Checking out files:  68% (4299/6322)   
Checking out files:  69% (4363/6322)   
Checking out files:  70% (4426/6322)   
Checking out files:  71% (4489/6322)   
Checking out files:  72% (4552/6322)   
Checking out files:  73% (4616/6322)   
Checking out files:  74% (4679/6322)   
Checking out files:  75% (4742/6322)   
Checking out files:  76% (4805/6322)   
Checking out files:  77% (4868/6322)   
Checking out files:  78% (4932/6322)   
Checking out files:  79% (4995/6322)   
Checking out files:  80% (5058/6322)   
Checking out files:  81% (5121/6322)   
Checking out files:  82% (5185/6322)   
Checking out files:  83% (5248/6322)   
Checking out files:  84% (5311/6322)   
Checking out files:  85% (5374/6322)   
Checking out files:  86% (5437/6322)   
Checking out files:  87% (5501/6322)   
Checking out files:  88% (5564/6322)   
Checking out files:  89% (5627/6322)   
Checking out files:  90% (5690/6322)   
Checking out files:  91% (5754/6322)   
Checking out files:  92% (5817/6322)   
Checking out files:  93% (5880/6322)   
Checking out files:  94% (5943/6322)   
Checking out files:  95% (6006/6322)   
Checking out files:  96% (6070/6322)   
Checking out files:  97% (6133/6322)   
Checking out files:  98% (6196/6322)   
Checking out files:  99% (6259/6322)   
Checking out files: 100% (6322/6322)   
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-nry751rd/src/docker-src.2018-08-16-11.09.35.11313/qemu.tar.vroot/dtc'...
Submodule path 

Re: [Qemu-block] [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814165349.56344-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-quick@centos7 SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]   patchew/20180816145554.9814-1-stefa...@redhat.com -> 
patchew/20180816145554.9814-1-stefa...@redhat.com
Switched to a new branch 'test'
8dd473d656 new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   centos7
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-axs8vnd6/src'
  GEN 
/var/tmp/patchew-tester-tmp-axs8vnd6/src/docker-src.2018-08-16-11.02.24.22115/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-axs8vnd6/src/docker-src.2018-08-16-11.02.24.22115/qemu.tar.vroot'...
done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-axs8vnd6/src/docker-src.2018-08-16-11.02.24.22115/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-axs8vnd6/src/docker-src.2018-08-16-11.02.24.22115/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-quick in qemu:centos7 
Packages installed:
SDL-devel-1.2.15-14.el7.x86_64
bison-3.0.4-1.el7.x86_64
bzip2-devel-1.0.6-13.el7.x86_64
ccache-3.3.4-1.el7.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el7.x86_64
flex-2.5.37-3.el7.x86_64
gcc-4.8.5-16.el7_4.2.x86_64
gettext-0.19.8.1-2.el7.x86_64
git-1.8.3.1-12.el7_4.x86_64
glib2-devel-2.50.3-3.el7.x86_64
libepoxy-devel-1.3.1-1.el7.x86_64
libfdt-devel-1.4.6-1.el7.x86_64
lzo-devel-2.06-8.el7.x86_64
make-3.82-23.el7.x86_64
mesa-libEGL-devel-17.0.1-6.20170307.el7.x86_64
mesa-libgbm-devel-17.0.1-6.20170307.el7.x86_64
package g++ is not installed
package librdmacm-devel is not installed
pixman-devel-0.34.0-1.el7.x86_64
spice-glib-devel-0.33-6.el7_4.1.x86_64
spice-server-devel-0.12.8-2.el7.1.x86_64
tar-1.26-32.el7.x86_64
vte-devel-0.28.2-10.el7.x86_64
xen-devel-4.6.6-10.el7.x86_64
zlib-devel-1.2.7-17.el7.x86_64

Environment variables:
PACKAGES=bison bzip2-devel ccache csnappy-devel flex g++
 gcc gettext git glib2-devel libepoxy-devel libfdt-devel
 librdmacm-devel lzo-devel make mesa-libEGL-devel 
mesa-libgbm-devel pixman-devel SDL-devel spice-glib-devel 
spice-server-devel tar vte-devel xen-devel zlib-devel
HOSTNAME=061e6499f5cc
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
SHLVL=1
HOME=/home/patchew
TEST_DIR=/tmp/qemu-test
FEATURES= dtc
DEBUG=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu 
--prefix=/tmp/qemu-test/install
No C++ compiler available; disabling C++ specific optional code
Install prefix/tmp/qemu-test/install
BIOS directory/tmp/qemu-test/install/share/qemu
firmware path /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install/bin
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib/qemu
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install/etc
local state directory   /tmp/qemu-test/install/var
Manual directory  /tmp/qemu-test/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path   /tmp/qemu-test/src
GIT binarygit
GIT submodules
C compilercc
Host C compiler   cc
C++ compiler  
Objective-C compiler cc
ARFLAGS   rv
CFLAGS-O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS   -I/usr/include/pixman-1-Werror -pthread 
-I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -mcx16 
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes 
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes 
-fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels 
-Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security 
-Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration 
-Wold-style-definition -Wtype-limits -fstack-protector-strong 
-Wno-missing-braces -I/usr/include/libpng15 -I/usr/include/spice-server 
-I/usr/include/cacard 

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814170126.56461-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-quick@centos7 SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
fede2b479e new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   centos7
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-5cpgkzz3/src'
  GEN 
/var/tmp/patchew-tester-tmp-5cpgkzz3/src/docker-src.2018-08-16-11.07.56.6514/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-5cpgkzz3/src/docker-src.2018-08-16-11.07.56.6514/qemu.tar.vroot'...
done.
Checking out files:  46% (2933/6322)   
Checking out files:  47% (2972/6322)   
Checking out files:  48% (3035/6322)   
Checking out files:  49% (3098/6322)   
Checking out files:  50% (3161/6322)   
Checking out files:  51% (3225/6322)   
Checking out files:  52% (3288/6322)   
Checking out files:  53% (3351/6322)   
Checking out files:  54% (3414/6322)   
Checking out files:  55% (3478/6322)   
Checking out files:  56% (3541/6322)   
Checking out files:  57% (3604/6322)   
Checking out files:  58% (3667/6322)   
Checking out files:  59% (3730/6322)   
Checking out files:  60% (3794/6322)   
Checking out files:  61% (3857/6322)   
Checking out files:  62% (3920/6322)   
Checking out files:  63% (3983/6322)   
Checking out files:  64% (4047/6322)   
Checking out files:  65% (4110/6322)   
Checking out files:  66% (4173/6322)   
Checking out files:  67% (4236/6322)   
Checking out files:  68% (4299/6322)   
Checking out files:  69% (4363/6322)   
Checking out files:  70% (4426/6322)   
Checking out files:  71% (4489/6322)   
Checking out files:  72% (4552/6322)   
Checking out files:  73% (4616/6322)   
Checking out files:  74% (4679/6322)   
Checking out files:  75% (4742/6322)   
Checking out files:  76% (4805/6322)   
Checking out files:  77% (4868/6322)   
Checking out files:  78% (4932/6322)   
Checking out files:  79% (4995/6322)   
Checking out files:  80% (5058/6322)   
Checking out files:  81% (5121/6322)   
Checking out files:  82% (5185/6322)   
Checking out files:  83% (5248/6322)   
Checking out files:  84% (5311/6322)   
Checking out files:  85% (5374/6322)   
Checking out files:  85% (5408/6322)   
Checking out files:  86% (5437/6322)   
Checking out files:  87% (5501/6322)   
Checking out files:  88% (5564/6322)   
Checking out files:  89% (5627/6322)   
Checking out files:  90% (5690/6322)   
Checking out files:  91% (5754/6322)   
Checking out files:  92% (5817/6322)   
Checking out files:  93% (5880/6322)   
Checking out files:  94% (5943/6322)   
Checking out files:  95% (6006/6322)   
Checking out files:  96% (6070/6322)   
Checking out files:  97% (6133/6322)   
Checking out files:  98% (6196/6322)   
Checking out files:  99% (6259/6322)   
Checking out files: 100% (6322/6322)   
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 
'/var/tmp/patchew-tester-tmp-5cpgkzz3/src/docker-src.2018-08-16-11.07.56.6514/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered 
for path 'ui/keycodemapdb'
Cloning into 
'/var/tmp/patchew-tester-tmp-5cpgkzz3/src/docker-src.2018-08-16-11.07.56.6514/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out 
'6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPYRUNNER
RUN test-quick in qemu:centos7 
Packages installed:
SDL-devel-1.2.15-14.el7.x86_64
bison-3.0.4-1.el7.x86_64
bzip2-devel-1.0.6-13.el7.x86_64
ccache-3.3.4-1.el7.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el7.x86_64
flex-2.5.37-3.el7.x86_64
gcc-4.8.5-16.el7_4.2.x86_64
gettext-0.19.8.1-2.el7.x86_64
git-1.8.3.1-12.el7_4.x86_64
glib2-devel-2.50.3-3.el7.x86_64
libepoxy-devel-1.3.1-1.el7.x86_64
libfdt-devel-1.4.6-1.el7.x86_64
lzo-devel-2.06-8.el7.x86_64
make-3.82-23.el7.x86_64
mesa-libEGL-devel-17.0.1-6.20170307.el7.x86_64
mesa-libgbm-devel-17.0.1-6.20170307.el7.x86_64
package g++ is not installed
package librdmacm-devel is not installed
pixman-devel-0.34.0-1.el7.x86_64
spice-glib-devel-0.33-6.el7_4.1.x86_64
spice-server-devel-0.12.8-2.el7.1.x86_64
tar-1.26-32.el7.x86_64
vte-devel-0.28.2-10.el7.x86_64
xen-devel-4.6.6-10.el7.x86_64
zlib-devel-1.2.7-17.el7.x86_64

Environment variables:
PACKAGES=bison bzip2-devel ccache csnappy-devel flex g++
 gcc gettext git glib2-devel libepoxy-devel libfdt-devel
 librdmacm-devel lzo-devel make mesa-libEGL-devel 
mesa-libgbm-devel 

Re: [Qemu-block] [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180814165349.56344-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [RFC] new, node-graph-based fleecing and backup

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=8
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
8dd473d656 new, node-graph-based fleecing and backup

=== OUTPUT BEGIN ===
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-rg8mj_mn/src'
  GEN 
/var/tmp/patchew-tester-tmp-rg8mj_mn/src/docker-src.2018-08-16-10.57.23.9820/qemu.tar
Cloning into 
'/var/tmp/patchew-tester-tmp-rg8mj_mn/src/docker-src.2018-08-16-10.57.23.9820/qemu.tar.vroot'...
done.
Checking out files:   8% (518/6322)   
Checking out files:   9% (569/6322)   
Checking out files:  10% (633/6322)   
Checking out files:  11% (696/6322)   
Checking out files:  11% (740/6322)   
Checking out files:  12% (759/6322)   
Checking out files:  13% (822/6322)   
Checking out files:  14% (886/6322)   
Checking out files:  15% (949/6322)   
Checking out files:  16% (1012/6322)   
Checking out files:  16% (1046/6322)   
Checking out files:  17% (1075/6322)   
Checking out files:  18% (1138/6322)   
Checking out files:  18% (1194/6322)   
Checking out files:  19% (1202/6322)   
Checking out files:  20% (1265/6322)   
Checking out files:  21% (1328/6322)   
Checking out files:  22% (1391/6322)   
Checking out files:  23% (1455/6322)   
Checking out files:  24% (1518/6322)   
Checking out files:  25% (1581/6322)   
Checking out files:  26% (1644/6322)   
Checking out files:  27% (1707/6322)   
Checking out files:  28% (1771/6322)   
Checking out files:  29% (1834/6322)   
Checking out files:  30% (1897/6322)   
Checking out files:  31% (1960/6322)   
Checking out files:  32% (2024/6322)   
Checking out files:  33% (2087/6322)   
Checking out files:  34% (2150/6322)   
Checking out files:  35% (2213/6322)   
Checking out files:  36% (2276/6322)   
Checking out files:  37% (2340/6322)   
Checking out files:  38% (2403/6322)   
Checking out files:  39% (2466/6322)   
Checking out files:  40% (2529/6322)   
Checking out files:  41% (2593/6322)   
Checking out files:  42% (2656/6322)   
Checking out files:  42% (2705/6322)   
Checking out files:  43% (2719/6322)   
Checking out files:  44% (2782/6322)   
Checking out files:  45% (2845/6322)   
Checking out files:  46% (2909/6322)   
Checking out files:  47% (2972/6322)   
Checking out files:  48% (3035/6322)   
Checking out files:  49% (3098/6322)   
Checking out files:  50% (3161/6322)   
Checking out files:  51% (3225/6322)   
Checking out files:  52% (3288/6322)   
Checking out files:  53% (3351/6322)   
Checking out files:  54% (3414/6322)   
Checking out files:  55% (3478/6322)   
Checking out files:  56% (3541/6322)   
Checking out files:  57% (3604/6322)   
Checking out files:  58% (3667/6322)   
Checking out files:  59% (3730/6322)   
Checking out files:  60% (3794/6322)   
Checking out files:  61% (3857/6322)   
Checking out files:  62% (3920/6322)   
Checking out files:  63% (3983/6322)   
Checking out files:  64% (4047/6322)   
Checking out files:  65% (4110/6322)   
Checking out files:  66% (4173/6322)   
Checking out files:  67% (4236/6322)   
Checking out files:  68% (4299/6322)   
Checking out files:  69% (4363/6322)   
Checking out files:  70% (4426/6322)   
Checking out files:  71% (4489/6322)   
Checking out files:  72% (4552/6322)   
Checking out files:  73% (4616/6322)   
Checking out files:  74% (4679/6322)   
Checking out files:  75% (4742/6322)   
Checking out files:  76% (4805/6322)   
Checking out files:  77% (4868/6322)   
Checking out files:  78% (4932/6322)   
Checking out files:  79% (4995/6322)   
Checking out files:  80% (5058/6322)   
Checking out files:  81% (5121/6322)   
Checking out files:  82% (5185/6322)   
Checking out files:  83% (5248/6322)   
Checking out files:  84% (5311/6322)   
Checking out files:  85% (5374/6322)   
Checking out files:  86% (5437/6322)   
Checking out files:  87% (5501/6322)   
Checking out files:  88% (5564/6322)   
Checking out files:  89% (5627/6322)   
Checking out files:  90% (5690/6322)   
Checking out files:  91% (5754/6322)   
Checking out files:  92% (5817/6322)   
Checking out files:  93% (5880/6322)   
Checking out files:  94% (5943/6322)   
Checking out files:  95% (6006/6322)   
Checking out files:  96% (6070/6322)   
Checking out files:  97% (6133/6322)   
Checking out files:  98% (6196/6322)   
Checking out files:  99% (6259/6322)   
Checking out files: 100% (6322/6322)   
Checking out files: 100% (6322/6322), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 

[Qemu-block] [PATCH v2 4/4] dmg: exchanging hardcoded dmg UDIF block types to enum.

[Julio Faracco]

This change is better to understand what kind of block type is being
handled by the code. Using a syntax similar to the DMG documentation is
easier than tracking all hex values assigned to a block type.

Signed-off-by: Julio Faracco 
---
 block/dmg.c | 43 ---
 1 file changed, 28 insertions(+), 15 deletions(-)

diff --git a/block/dmg.c b/block/dmg.c
index 615f818c5a..6d055594df 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -44,6 +44,19 @@ enum {
 DMG_SECTORCOUNTS_MAX = DMG_LENGTHS_MAX / 512,
 };
 
+enum {
+/* DMG Block Type */
+UDZE = 0, /* Zeroes */
+UDRW, /* RAW type */
+UDIG, /* Ignore */
+UDCO = 0x8004,
+UDZO,
+UDBZ,
+ULFO,
+UDCM = 0x7ffe, /* Comments */
+UDLE   /* Last Entry */
+};
+
 static int dmg_probe(const uint8_t *buf, int buf_size, const char *filename)
 {
 int len;
@@ -108,16 +121,16 @@ static void update_max_chunk_size(BDRVDMGState *s, 
uint32_t chunk,
 uint32_t uncompressed_sectors = 0;
 
 switch (s->types[chunk]) {
-case 0x8005: /* zlib compressed */
-case 0x8006: /* bzip2 compressed */
-case 0x8007: /* lzfse compressed */
+case UDZO: /* zlib compressed */
+case UDBZ: /* bzip2 compressed */
+case ULFO: /* lzfse compressed */
 compressed_size = s->lengths[chunk];
 uncompressed_sectors = s->sectorcounts[chunk];
 break;
-case 1: /* copy */
+case UDRW: /* copy */
 uncompressed_sectors = DIV_ROUND_UP(s->lengths[chunk], 512);
 break;
-case 2: /* zero */
+case UDIG: /* zero */
 /* as the all-zeroes block may be large, it is treated specially: the
  * sector is not copied from a large buffer, a simple memset is used
  * instead. Therefore uncompressed_sectors does not need to be set. */
@@ -186,13 +199,13 @@ typedef struct DmgHeaderState {
 static bool dmg_is_known_block_type(uint32_t entry_type)
 {
 switch (entry_type) {
-case 0x0001:/* uncompressed */
-case 0x0002:/* zeroes */
-case 0x8005:/* zlib */
+case UDRW:/* uncompressed */
+case UDIG:/* zeroes */
+case UDZO:/* zlib */
 return true;
-case 0x8006:/* bzip2 */
+case UDBZ:/* bzip2 */
 return !!dmg_uncompress_bz2;
-case 0x8007:/* lzfse */
+case ULFO:/* lzfse */
 return !!dmg_uncompress_lzfse;
 default:
 return false;
@@ -592,7 +605,7 @@ static inline int dmg_read_chunk(BlockDriverState *bs, 
uint64_t sector_num)
 
 s->current_chunk = s->n_chunks;
 switch (s->types[chunk]) { /* block entry type */
-case 0x8005: { /* zlib compressed */
+case UDZO: { /* zlib compressed */
 /* we need to buffer, because only the chunk as whole can be
  * inflated. */
 ret = bdrv_pread(bs->file, s->offsets[chunk],
@@ -615,7 +628,7 @@ static inline int dmg_read_chunk(BlockDriverState *bs, 
uint64_t sector_num)
 return -1;
 }
 break; }
-case 0x8006: /* bzip2 compressed */
+case UDBZ: /* bzip2 compressed */
 if (!dmg_uncompress_bz2) {
 break;
 }
@@ -636,7 +649,7 @@ static inline int dmg_read_chunk(BlockDriverState *bs, 
uint64_t sector_num)
 return ret;
 }
 break;
-case 0x8007:
+case ULFO:
 if (!dmg_uncompress_lzfse) {
 break;
 }
@@ -657,14 +670,14 @@ static inline int dmg_read_chunk(BlockDriverState *bs, 
uint64_t sector_num)
 return ret;
 }
 break;
-case 1: /* copy */
+case UDRW: /* copy */
 ret = bdrv_pread(bs->file, s->offsets[chunk],
  s->uncompressed_chunk, s->lengths[chunk]);
 if (ret != s->lengths[chunk]) {
 return -1;
 }
 break;
-case 2: /* zero */
+case UDIG: /* zero */
 /* see dmg_read, it is treated specially. No buffer needs to be
  * pre-filled, the zeroes can be set directly. */
 break;
-- 
2.17.1

[Qemu-block] [PATCH v2 0/4] Adding LZFSE compression support for DMG block driver.

[Julio Faracco]

Since Mac OS X El Capitain (v10.11), Apple uses LZFSE compression to 
generate compressed DMGs as an alternative to BZIP2. Possible, Apple
want to keep this algorithm as default in long term. Some years ago, 
Apple opened the LZFSE algorithm to opensource and the main source (or 
the most active repo), can be found at: https://github.com/lzfse/lzfse

Julio Faracco (4):
  block: adding lzfse decompressing support as a module.
  configure: adding support to lzfse library.
  dmg: including dmg-lzfse module inside dmg block driver.
  dmg: exchanging hardcoded dmg UDIF block types to enum.

 block/Makefile.objs |  2 ++
 block/dmg-lzfse.c   | 49 ++
 block/dmg.c | 65 -
 block/dmg.h |  3 +++
 configure   | 31 +
 5 files changed, 138 insertions(+), 12 deletions(-)
 create mode 100644 block/dmg-lzfse.c

-- 
2.17.1

Re: [Qemu-block] [PATCH 0/7] qcow2: async handling of fragmented io

[Vladimir Sementsov-Ogievskiy]

16.08.2018 03:51, Max Reitz wrote:

On 2018-08-07 19:43, Vladimir Sementsov-Ogievskiy wrote:

Hi all!

Here is an asynchronous scheme for handling fragmented qcow2
reads and writes. Both qcow2 read and write functions loops through
sequential portions of data. The series aim it to parallelize these
loops iterations.

It improves performance for fragmented qcow2 images, I've tested it
as follows:

I have four 4G qcow2 images (with default 64k block size) on my ssd disk:
t-seq.qcow2 - sequentially written qcow2 image
t-reverse.qcow2 - filled by writing 64k portions from end to the start
t-rand.qcow2 - filled by writing 64k portions (aligned) in random order
t-part-rand.qcow2 - filled by shuffling order of 64k writes in 1m clusters
(see source code of image generation in the end for details)

and the test (sequential io by 1mb chunks):

test write:
 for t in /ssd/t-*; \
 do sync; echo 1 > /proc/sys/vm/drop_caches; echo ===  $t  ===; \
 ./qemu-img bench -c 4096 -d 1 -f qcow2 -n -s 1m -t none -w $t; \
 done

test read (same, just drop -w parameter):
 for t in /ssd/t-*; \
 do sync; echo 1 > /proc/sys/vm/drop_caches; echo ===  $t  ===; \
 ./qemu-img bench -c 4096 -d 1 -f qcow2 -n -s 1m -t none $t; \
 done

short info about parameters:
   -w - do writes (otherwise do reads)
   -c - count of blocks
   -s - block size
   -t none - disable cache
   -n - native aio
   -d 1 - don't use parallel requests provided by qemu-img bench itself

Hm, actually, why not?  And how does a guest behave?

If parallel requests on an SSD perform better, wouldn't a guest issue
parallel requests to the virtual device and thus to qcow2 anyway?


Guest knows nothing about qcow2 fragmentation, so this kind of 
"asynchronization" could be done only at qcow2 level.
However, if guest do async io, send a lot of parallel requests, it 
behave like qemu-img without -d 1 option, and in this case,
parallel loop iterations in qcow2 doesn't have such great sense. 
However, I think that async parallel requests are better in
general than sequential, because if device have some unused opportunity 
of parallelization, it will be utilized. We've already
use this approach in mirror and qemu-img convert. In Virtuozzo we have 
backup, improved by parallelization of requests
loop too. I think, it would be good to have some general code for such 
things in future.




(I suppose the global qcow2 lock could be an issue here, but then your
benchmark should work even without -d 1.)

Max




--
Best regards,
Vladimir

[Qemu-block] [PATCH v2 2/4] configure: adding support to lzfse library.

[Julio Faracco]

This commit includes the support to lzfse opensource library. With this
library dmg block driver can decompress images with this type of
compression inside.

Signed-off-by: Julio Faracco 
---
 block/Makefile.objs |  2 ++
 configure   | 31 +++
 2 files changed, 33 insertions(+)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index c8337bf186..f4ddbb9c7b 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -47,6 +47,8 @@ ssh.o-cflags   := $(LIBSSH2_CFLAGS)
 ssh.o-libs := $(LIBSSH2_LIBS)
 block-obj-$(if $(CONFIG_BZIP2),m,n) += dmg-bz2.o
 dmg-bz2.o-libs := $(BZIP2_LIBS)
+block-obj-$(if $(CONFIG_LZFSE),m,n) += dmg-lzfse.o
+dmg-lzfse.o-libs   := $(LZFSE_LIBS)
 qcow.o-libs:= -lz
 linux-aio.o-libs   := -laio
 parallels.o-cflags := $(LIBXML2_CFLAGS)
diff --git a/configure b/configure
index 2a7796ea80..c5a4628f98 100755
--- a/configure
+++ b/configure
@@ -432,6 +432,7 @@ capstone=""
 lzo=""
 snappy=""
 bzip2=""
+lzfse=""
 guest_agent=""
 guest_agent_with_vss="no"
 guest_agent_ntddscsi="no"
@@ -1300,6 +1301,10 @@ for opt do
   ;;
   --enable-bzip2) bzip2="yes"
   ;;
+  --enable-lzfse) lzfse="yes"
+  ;;
+  --disable-lzfse) lzfse="no"
+  ;;
   --enable-guest-agent) guest_agent="yes"
   ;;
   --disable-guest-agent) guest_agent="no"
@@ -1689,6 +1694,8 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   snappy  support of snappy compression library
   bzip2   support of bzip2 compression library
   (for reading bzip2-compressed dmg images)
+  lzfse   support of lzfse compression library
+  (for reading lzfse-compressed dmg images)
   seccomp seccomp support
   coroutine-pool  coroutine freelist (better performance)
   glusterfs   GlusterFS backend
@@ -2213,6 +2220,24 @@ EOF
 fi
 fi
 
+##
+# lzfse check
+
+if test "$lzfse" != "no" ; then
+cat > $TMPC << EOF
+#include 
+int main(void) { lzfse_decode_scratch_size(); return 0; }
+EOF
+if compile_prog "" "-llzfse" ; then
+lzfse="yes"
+else
+if test "$lzfse" = "yes"; then
+feature_not_found "lzfse" "Install lzfse devel"
+fi
+lzfse="no"
+fi
+fi
+
 ##
 # libseccomp check
 
@@ -6001,6 +6026,7 @@ echo "Live block migration $live_block_migration"
 echo "lzo support   $lzo"
 echo "snappy support$snappy"
 echo "bzip2 support $bzip2"
+echo "lzfse support $lzfse"
 echo "NUMA host support $numa"
 echo "libxml2   $libxml2"
 echo "tcmalloc support  $tcmalloc"
@@ -6525,6 +6551,11 @@ if test "$bzip2" = "yes" ; then
   echo "BZIP2_LIBS=-lbz2" >> $config_host_mak
 fi
 
+if test "$lzfse" = "yes" ; then
+  echo "CONFIG_LZFSE=y" >> $config_host_mak
+  echo "LZFSE_LIBS=-llzfse" >> $config_host_mak
+fi
+
 if test "$libiscsi" = "yes" ; then
   echo "CONFIG_LIBISCSI=m" >> $config_host_mak
   echo "LIBISCSI_CFLAGS=$libiscsi_cflags" >> $config_host_mak
-- 
2.17.1

[Qemu-block] [PATCH v2 3/4] dmg: including dmg-lzfse module inside dmg block driver.

[Julio Faracco]

This commit includes the support to new module dmg-lzfse into dmg block
driver. It includes the support for block type ULFO (0x8007).

Signed-off-by: Julio Faracco 
---
 block/dmg.c | 28 
 block/dmg.h |  3 +++
 2 files changed, 31 insertions(+)

diff --git a/block/dmg.c b/block/dmg.c
index c9b3c519c4..615f818c5a 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -33,6 +33,9 @@
 int (*dmg_uncompress_bz2)(char *next_in, unsigned int avail_in,
   char *next_out, unsigned int avail_out);
 
+int (*dmg_uncompress_lzfse)(char *next_in, unsigned int avail_in,
+char *next_out, unsigned int avail_out);
+
 enum {
 /* Limit chunk sizes to prevent unreasonable amounts of memory being used
  * or truncating when converting to 32-bit types
@@ -107,6 +110,7 @@ static void update_max_chunk_size(BDRVDMGState *s, uint32_t 
chunk,
 switch (s->types[chunk]) {
 case 0x8005: /* zlib compressed */
 case 0x8006: /* bzip2 compressed */
+case 0x8007: /* lzfse compressed */
 compressed_size = s->lengths[chunk];
 uncompressed_sectors = s->sectorcounts[chunk];
 break;
@@ -188,6 +192,8 @@ static bool dmg_is_known_block_type(uint32_t entry_type)
 return true;
 case 0x8006:/* bzip2 */
 return !!dmg_uncompress_bz2;
+case 0x8007:/* lzfse */
+return !!dmg_uncompress_lzfse;
 default:
 return false;
 }
@@ -431,6 +437,7 @@ static int dmg_open(BlockDriverState *bs, QDict *options, 
int flags,
 }
 
 block_module_load_one("dmg-bz2");
+block_module_load_one("dmg-lzfse");
 
 s->n_chunks = 0;
 s->offsets = s->lengths = s->sectors = s->sectorcounts = NULL;
@@ -629,6 +636,27 @@ static inline int dmg_read_chunk(BlockDriverState *bs, 
uint64_t sector_num)
 return ret;
 }
 break;
+case 0x8007:
+if (!dmg_uncompress_lzfse) {
+break;
+}
+/* we need to buffer, because only the chunk as whole can be
+ * inflated. */
+ret = bdrv_pread(bs->file, s->offsets[chunk],
+ s->compressed_chunk, s->lengths[chunk]);
+if (ret != s->lengths[chunk]) {
+return -1;
+}
+
+ret = dmg_uncompress_lzfse((char *)s->compressed_chunk,
+   (unsigned int) s->lengths[chunk],
+   (char *)s->uncompressed_chunk,
+   (unsigned int)
+   (512 * s->sectorcounts[chunk]));
+if (ret < 0) {
+return ret;
+}
+break;
 case 1: /* copy */
 ret = bdrv_pread(bs->file, s->offsets[chunk],
  s->uncompressed_chunk, s->lengths[chunk]);
diff --git a/block/dmg.h b/block/dmg.h
index 2ecf239ba5..f28929998f 100644
--- a/block/dmg.h
+++ b/block/dmg.h
@@ -55,4 +55,7 @@ typedef struct BDRVDMGState {
 extern int (*dmg_uncompress_bz2)(char *next_in, unsigned int avail_in,
  char *next_out, unsigned int avail_out);
 
+extern int (*dmg_uncompress_lzfse)(char *next_in, unsigned int avail_in,
+   char *next_out, unsigned int avail_out);
+
 #endif
-- 
2.17.1

[Qemu-block] [PATCH v2 1/4] block: adding lzfse decompressing support as a module.

[Julio Faracco]

QEMU dmg support includes zlib and bzip2, but it does not contains lzfse
support. This commit adds the source file to extend compression support
for new DMGs.

Signed-off-by: Julio Faracco 
---
 block/dmg-lzfse.c | 49 +++
 1 file changed, 49 insertions(+)
 create mode 100644 block/dmg-lzfse.c

diff --git a/block/dmg-lzfse.c b/block/dmg-lzfse.c
new file mode 100644
index 00..19d25bc646
--- /dev/null
+++ b/block/dmg-lzfse.c
@@ -0,0 +1,49 @@
+/*
+ * DMG lzfse uncompression
+ *
+ * Copyright (c) 2018 Julio Cesar Faracco
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "dmg.h"
+#include 
+
+static int dmg_uncompress_lzfse_do(char *next_in, unsigned int avail_in,
+   char *next_out, unsigned int avail_out)
+{
+size_t out_size = lzfse_decode_buffer((uint8_t *) next_out, avail_out,
+  (uint8_t *) next_in, avail_in,
+  NULL);
+
+/* We need to decode the single chunk only. */
+/* So, out_size == avail_out is not an error here. */
+if (out_size > 0) {
+return out_size;
+}
+return -1;
+}
+
+__attribute__((constructor))
+static void dmg_lzfse_init(void)
+{
+assert(!dmg_uncompress_lzfse);
+dmg_uncompress_lzfse = dmg_uncompress_lzfse_do;
+}
-- 
2.17.1

Re: [Qemu-block] [Qemu-devel] [PATCH for-3.1 v10 00/31] block: Fix some filename generation issues

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180809213528.14738-1-mre...@redhat.com
Subject: [Qemu-devel] [PATCH for-3.1 v10 00/31] block: Fix some filename 
generation issues

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
4d2a6801e3 iotests: Test json:{} filenames of internal BDSs
92861b0d23 block: BDS options may lack the "driver" option
5e539a99ee block/null: Generate filename even with latency-ns
3011e18097 block/curl: Implement bdrv_refresh_filename()
c3d3ba04c4 block/curl: Harmonize option defaults
0f00168fdc block/nvme: Fix bdrv_refresh_filename()
10fd7a823b block: Do not copy exact_filename from format file
1705480785 block: Purify .bdrv_refresh_filename()
d5face8a37 block: Generically refresh runtime options
d83b993208 block: Add BlockDriver.bdrv_gather_child_options
38384107dc block: Add strong_runtime_opts to BlockDriver
b9b3310f5d iotests: Add quorum case to test 110
292e6e5c96 block: Use bdrv_dirname() for relative filenames
f1fd748aca block/nfs: Implement bdrv_dirname()
8dfbf896b6 block/nbd: Make bdrv_dirname() return NULL
a5187853f6 quorum: Make bdrv_dirname() return NULL
4f62d050cd blkverify: Make bdrv_dirname() return NULL
f4ae7225b0 block: Add bdrv_dirname()
9b5ad84b84 block: Fix bdrv_find_backing_image()
324ade1026 block: Add bdrv_make_absolute_filename()
68def7295f block: bdrv_get_full_backing_filename's ret. val.
e5e55fcbb3 block: bdrv_get_full_backing_filename_from_...'s ret. val.
b434ec5b34 block: Make path_combine() return the path
a3542a966b iotests: Add test for backing file overrides
6affaa70a0 iotests.py: Add node_info()
13d2e98fe3 iotests.py: Add filter_imgfmt()
a46abb22e3 block: Respect backing bs in bdrv_refresh_filename
8b0cd321f7 block: Add BDS.auto_backing_file
0b9c999e16 block: Skip implicit nodes for filename info
b4b7a10cc2 block: Use children list in bdrv_refresh_filename
92210c34b4 block: Use bdrv_refresh_filename() to pull

=== OUTPUT BEGIN ===
Checking PATCH 1/31: block: Use bdrv_refresh_filename() to pull...
Checking PATCH 2/31: block: Use children list in bdrv_refresh_filename...
Checking PATCH 3/31: block: Skip implicit nodes for filename info...
Checking PATCH 4/31: block: Add BDS.auto_backing_file...
Checking PATCH 5/31: block: Respect backing bs in bdrv_refresh_filename...
ERROR: return is not a function, parentheses are not required
#46: FILE: block.c:5192:
+return (bs->auto_backing_file[0] != '\0');

total: 1 errors, 0 warnings, 136 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 6/31: iotests.py: Add filter_imgfmt()...
Checking PATCH 7/31: iotests.py: Add node_info()...
Checking PATCH 8/31: iotests: Add test for backing file overrides...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#10: 
new file mode 100755

total: 0 errors, 1 warnings, 323 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 9/31: block: Make path_combine() return the path...
Checking PATCH 10/31: block: bdrv_get_full_backing_filename_from_...'s ret. 
val
Checking PATCH 11/31: block: bdrv_get_full_backing_filename's ret. val
Checking PATCH 12/31: block: Add bdrv_make_absolute_filename()...
Checking PATCH 13/31: block: Fix bdrv_find_backing_image()...
Checking PATCH 14/31: block: Add bdrv_dirname()...
Checking PATCH 15/31: blkverify: Make bdrv_dirname() return NULL...
Checking PATCH 16/31: quorum: Make bdrv_dirname() return NULL...
Checking PATCH 17/31: block/nbd: Make bdrv_dirname() return NULL...
Checking PATCH 18/31: block/nfs: Implement bdrv_dirname()...
Checking PATCH 19/31: block: Use bdrv_dirname() for relative filenames...
Checking PATCH 20/31: iotests: Add quorum case to test 110...
Checking PATCH 21/31: block: Add strong_runtime_opts to BlockDriver...
Checking PATCH 22/31: block: Add BlockDriver.bdrv_gather_child_options...
Checking PATCH 23/31: block: Generically refresh runtime options...
Checking PATCH 24/31: block: Purify .bdrv_refresh_filename()...
Checking PATCH 25/31: block: Do not copy exact_filename from format file...
Checking PATCH 26/31: block/nvme: Fix bdrv_refresh_filename()...
Checking PATCH 27/31: block/curl: 

Re: [Qemu-block] [PULL 00/21] Block layer patches

[Peter Maydell]

On 15 August 2018 at 13:55, Kevin Wolf  wrote:
> The following changes since commit 38441756b70eec5807b5f60dad11a93a91199866:
>
>   Update version for v3.0.0 release (2018-08-14 16:38:43 +0100)
>
> are available in the git repository at:
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to b5fc2d306664c0c1c6c5cf8e164ffa7b8892283e:
>
>   qapi: block: Remove mentions of error types which were removed (2018-08-15 
> 12:50:39 +0200)
>
> 
> Block layer patches:
>
> - Remove deprecated -drive options for geometry/serial/addr
> - luks: Allow shared writers if the parents allow them (share-rw=on)
> - qemu-img: Fix error when trying to convert to encrypted target image
> - mirror: Fail gracefully for source == target
> - I/O throttling: Fix behaviour during drain (always ignore the limits)
> - bdrv_reopen() related fixes for bs->options/explicit_options content
> - Documentation improvements
>

Applied, thanks.

-- PMM

Re: [Qemu-block] [Qemu-devel] [PATCH v3 0/8] dirty-bitmap: rewrite bdrv_dirty_iter_next_area

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180814121443.33114-1-vsement...@virtuozzo.com
Subject: [Qemu-devel] [PATCH v3 0/8] dirty-bitmap: rewrite 
bdrv_dirty_iter_next_area

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
e8d4951adf Revert "hbitmap: Add @advance param to hbitmap_iter_next()"
f69ec3244d Revert "test-hbitmap: Add non-advancing iter_next tests"
c924e13bbe Revert "block/dirty-bitmap: Add bdrv_dirty_iter_next_area"
da911f4201 block/mirror: fix and improve do_sync_target_write
e4699ed90e tests: add tests for hbitmap_next_dirty_area
ad2af3fae8 dirty-bitmap: add bdrv_dirty_bitmap_next_dirty_area
b85dbbed91 tests: add tests for hbitmap_next_zero with specified end parameter
78a802d4d3 dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

=== OUTPUT BEGIN ===
Checking PATCH 1/8: dirty-bitmap: improve bdrv_dirty_bitmap_next_zero...
Checking PATCH 2/8: tests: add tests for hbitmap_next_zero with specified end 
parameter...
Checking PATCH 3/8: dirty-bitmap: add bdrv_dirty_bitmap_next_dirty_area...
Checking PATCH 4/8: tests: add tests for hbitmap_next_dirty_area...
ERROR: suspect code indent for conditional statements (4, 4)
#36: FILE: tests/test-hbitmap.c:1038:
+for (len2 = 1; off2 + len2 < end && hbitmap_get(data->hb, off2 + len2);
[...]
+{

total: 1 errors, 0 warnings, 118 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 5/8: block/mirror: fix and improve do_sync_target_write...
Checking PATCH 6/8: Revert "block/dirty-bitmap: Add 
bdrv_dirty_iter_next_area"...
Checking PATCH 7/8: Revert "test-hbitmap: Add non-advancing iter_next tests"...
Checking PATCH 8/8: Revert "hbitmap: Add @advance param to 
hbitmap_iter_next()"...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [Qemu-block] [PATCH 2/2] qemu-img: Add dd seek= option

[Kevin Wolf]

Am 16.08.2018 um 04:49 hat Max Reitz geschrieben:
> On 2018-08-16 04:39, Eric Blake wrote:
> > If convert were more powerful, I'd be fine dropping 'qemu-img dd' after
> > a proper deprecation period.
> 
> Technically it has those features already, with the raw block driver's
> offset and size parameters.

In a way, yes. It requires you to precreate the target image, though,
because --target-image-opts requires -n.

We'll probably want to fix something in this area anyway because in the
common case, you already need those options to even precreate the image,
and qemu-img create doesn't support open options for the protocol layer
either (unlike blockdev-create).

> >> ((That gave me a good idea.  Actually, it's probably not such a good
> >> idea, but I guess I'll do it in my spare time anyway.  A qemu-img fuse
> >> might be nice which represents an image as a raw image at some mount
> >> point.  Benefits over qemu-nbd: (1) You don't need root, (2) you don't
> >> need to type modprobe nbd.))
> > 
> > So the kernel->userspace translation would be happening via the FUSE
> > interface instead of the NBD interface.  Data still bounces around just
> > as much, but it might be a fun project.  Does fuse behave well when
> > serving exactly one file at the mountpoint, rather than the more typical
> > file system rooted in a directory?  NBD at least has the benefit of
> > claiming to be a block device all along, rather than complicating the
> > user interface with POSIX file system rules (which you'll be bending,
> > because you are serving exactly one file instead of a system).

To make it more real, you could expose a snapshots/ subdirectory. Or
maybe better not.

> Well, but I can just pretend my FUSE file is a block device, no?
> 
> Also, I just discovered something really interesting: FUSE allows you to
> specify a single file as a mountpoint.
> 
> And you know what?  You can open the original file before you replace it
> by the FUSE "filesystem".
> 
> So my fun interface is going to looks like this:
> 
> $ qemu-img fuse foo.qcow2
> 
> And then your foo.qcow2 is a raw image until the next "fusermount -u
> foo.qcow2"!  Isn't that fun?

Yes, sounds fun. Until you realise that I'd actually like to merge
something like this when it exists. ;-)

For a more serious implementation, maybe it should even be a separate
qemu-fuse rather than a qemu-img subcommand.

Kevin


signature.asc
Description: PGP signature

Re: [Qemu-block] [PATCH 16/21] block/commit: refactor commit to use job callbacks

[Kevin Wolf]

Am 15.08.2018 um 23:17 hat John Snow geschrieben:
> On 08/09/2018 11:12 AM, Kevin Wolf wrote:
> > Am 07.08.2018 um 06:33 hat John Snow geschrieben:
> >> Use the component callbacks; prepare, commit, abort, and clean.
> >>
> >> NB: prepare is only called when the job has not yet failed;
> >> and abort can be called after prepare.
> >>
> >> complete -> prepare -> abort -> clean
> >> complete -> abort -> clean
> > 
> > I always found this confusing. After converting all jobs to use the
> > infrastructure, do you think that this design is helpful? You seem to be
> > calling *_common function from commit and abort, with an almost empty
> > prepare, which looks like a hint that maybe we should reorganise things.
> > 
> 
> After rewriting things a bit, I think it would be nicer to call prepare
> regardless of circumstances. The callbacks will be nicer for it.
> 
> When I wrote it the first time, the thought process was something like:
> 
> "Well, we won't need to prepare anything if we've already failed, we
> just need to speed along to abort."
> 
> I wasn't considering so carefully the common cleanup case that needs to
> occur post-finalization which appears to actually happen in a good
> number of jobs.

Maybe let's see how things turn out when we actually make some more jobs
transactionable. For now, it seems that the *_common function can go
away at least for commit; and we didn't even try to properly split the
completion of the other two jobs.

> >> Signed-off-by: John Snow 
> >> ---
> >>  block/commit.c | 94 
> >> +-
> >>  1 file changed, 61 insertions(+), 33 deletions(-)
> >>
> >> diff --git a/block/commit.c b/block/commit.c
> >> index 1a4a56795f..6673a0544e 100644
> >> --- a/block/commit.c
> >> +++ b/block/commit.c
> >> @@ -35,7 +35,9 @@ typedef struct CommitBlockJob {
> >>  BlockJob common;
> >>  BlockDriverState *commit_top_bs;
> >>  BlockBackend *top;
> >> +BlockDriverState *top_bs;
> >>  BlockBackend *base;
> >> +BlockDriverState *base_bs;
> >>  BlockdevOnError on_error;
> >>  int base_flags;
> >>  char *backing_file_str;
> > 
> > More state. :-(
> > 
> > Can we at least move the new fields to the end of the struct with a
> > comment that they are only valid after .exit()?
> > 
> 
> Sure ... admittedly this is just a crutch because I was not precisely
> sure exactly when these might change out from underneath me. If there
> are some clever ways to avoid needing the state, feel free to suggest them.

I did, in the reply to my own mail. Everything that would need the state
can actually live in .abort, so it can be local.

> >> +}
> >> +
> >> +static void commit_commit(Job *job)
> >> +{
> >> +commit_exit_common(job);
> >> +}
> > 
> > (I think I've answered this question now, by effectively proposing to do
> > away with .commit, but I'll leave it there anyway.)
> > 
> > Is there any reason for the split between .prepare and .commit as it is
> > or is this mostly arbitrary? Probably the latter because commit isn't
> > even transactionable?
> > 
> 
> Just historical, yeah -- we only had commit and abort for a while, and
> prepare didn't join the party until we wanted finalize and it became
> apparent we needed a way to check the return code and still be able to
> fail the transaction in time to be able to do a final commit or still
> abort very, very late in the process.
> 
> Probably there's no real meaningful reason that prepare and commit need
> to be separate callbacks.
> 
> It is possible that:
> 
> prepare --> [abort] --> clean
> 
> would be entirely sufficient for all current cases.

I think jobs that actually support transactions (i.e. backup currently)
do in fact need commit. The other ones might not.

Kevin

Re: [Qemu-block] [PATCH 04/21] block/commit: utilize job_exit shim

[Kevin Wolf]

Am 15.08.2018 um 22:52 hat John Snow geschrieben:
> On 08/08/2018 12:29 PM, Kevin Wolf wrote:
> > Am 07.08.2018 um 06:33 hat John Snow geschrieben:
> >> Change the manual deferment to commit_complete into the implicit
> >> callback to job_exit.
> >>
> >> Signed-off-by: John Snow 
> > 
> > There is one tricky thing in this patch that the commit message could be
> > a bit more explicit about, which is moving job_completed() to a later
> > point.
> > 
> > This is the code that happens between the old call of job_completed()
> > and the new one:
> > 
> > /* If bdrv_drop_intermediate() didn't already do that, remove the commit
> >  * filter driver from the backing chain. Do this as the final step so 
> > that
> >  * the 'consistent read' permission can be granted.  */
> > if (remove_commit_top_bs) {
> > bdrv_child_try_set_perm(commit_top_bs->backing, 0, BLK_PERM_ALL,
> > _abort);
> > bdrv_replace_node(commit_top_bs, backing_bs(commit_top_bs),
> >   _abort);
> > }
> > 
> > bdrv_unref(commit_top_bs);
> > bdrv_unref(top);
> > 
> > As the comment states, bdrv_replace_node() requires that the permission
> > restrictions that the commit job made are already lifted. The most
> > important part is done by the explicit block_job_remove_all_bdrv() call
> > right before this hunk. It still leaves bjob->blk around, which could
> > have implications, but luckily we didn't take any permissions for that
> > one:
> > 
> > s = block_job_create(job_id, _job_driver, NULL, bs, 0, 
> > BLK_PERM_ALL,
> >  speed, JOB_DEFAULT, NULL, NULL, errp);
> > 
> > So I think we got everything out of the way and bdrv_replace_node() can
> > do what it wants to do.
> > 
> > Kevin
> > 
> 
> I suppose it will be up to the author of a job to be aware of any
> permissions they pick up at creation time that might have an effect on
> the cleanup they wish to do during completion time.

I'm really only talking about the commit job, the specific conversion
in this patch and the terse commit message.

Kevin

Re: [Qemu-block] [Qemu-devel] [RFC PATCH] rbd: Don't convert keypairs to JSON and back

[Markus Armbruster]

Max Reitz  writes:

> On 2018-08-15 10:12, Markus Armbruster wrote:
>> Max Reitz  writes:
>
> [...]
>
>>> To me personally the issue is that if you can specify a plain filename,
>>> bdrv_refresh_filename() should give you that plain filename back.  So
>>> rbd's implementation of that is lacking.  Well, it just doesn't exist.
>> 
>> I'm not even sure I understand what you're talking about.
>
> We have this bdrv_refresh_filename() thing which can do this:
>
> $ qemu-img info \
> "json:{'driver':'raw',
>'file':{'driver':'nbd','host':'localhost'}}"
> image: nbd://localhost:10809
> [...]
>
> So it can reconstruct a plain filename even if you specify it as options
> instead of just using a plain filename.
>
>
> Now here's my fault: I thought it might be necessary for a driver to
> implement that function (which rbd doesn't) so that you'd get a nice
> filename back (instead of just json:{} garbled things).   But you don't.
>  For protocol drivers, you'll just get the initial filename back.  (So
> my comment was just wrong.)
>
> So what I was thinking about was some case where you specified a normal
> plain filename and qemu would give you back json:{}.  (If rbd
> implemented bdrv_refresh_filename(), that wouldn't happen, because it
> would reconstruct a nice normal filename.)  It turns out, I don't think
> that can happen so easily.  You'll just get your filename back.
>
> Because here's what I'm thinking: If someone uses an option that is
> undocumented and starts with =, well, too bad.  If someone uses a normal
> filename, but gets back a json:{} filename...  Then they are free to use
> that anywhere, and their use of "=" is legitimized.
>
>
> Now that issue kind of reappears when you open an RBD volume, and then
> e.g. take a blockdev-snapshot.  Then your overlay has an overridden
> backing file (one that may be different from what its image header says)
> and its filename may well become a json:{} one (to arrange for the
> overridden backing file options).  Of course, if you opened the RBD
> volume with a filename with some of the options warranting
> =keyvalue-pairs, then your json:{} filename will contain those options
> under =keyvalue-pairs.
>
> So...  I'm not quite sure what I want to say?  I think there are edge
> cases where the user may not have put any weird option into qemu, but
> they do get a json:{} filename with =keyvalue-pairs out of it.  And I
> think users are free to use json:{} filenames qemu spews at them, and we
> can't blame them for it.

Makes sense.

More reason to deprecate key-value pairs in pseudo-filenames.

The only alternative would be to also provide them in QAPI
BlockdevOptionsRbd.  I find that as distasteful as ever, but if the
block maintainers decide we need it, I'll hold my nose.

 If so, and we are comfortable changing the output the way this patch does
 (technically altering ABI anyway), we might as well go all the way and
 filter it out completely.  That would be preferable to cleaning up the json
 output of the internal key/value pairs, IMO.
>>>
>>> Well, this filtering at least is done by my "Fix some filename
>>> generation issues" series.
>> 
>> Likewise.
>
> The series overhauls quite a bit of the bdrv_refresh_filename()
> infrastructure.  That function is also responsible for generating
> json:{} filenames.
>
> One thing it introduces is a BlockDriver field where a driver can
> specify which of the runtime options are actually important.  The rest
> is omitted from the generated json:{} filename.
>
> I may have taken the liberty not to include =keyvalue-pairs in RBD's
> "strong runtime options" list.

I see.

Permit me to digress a bit.

I understand one application for generating a json: filename is for
putting it into an image file header (say as a COW's backing image).

Having image file headers point to other images is not as simple as it
may look at first glance.  The basic case of image format plus plain
filename (not containing '/') is straightforward enough.  But if I make
the filename absolute (with a leading '/'), the image becomes less easy
to move to another machine.

Similarly, certain Ceph configuration bits may make no sense on another
machine, and putting them into an image file header may not be a good
idea.

Rather vague, I'm afraid.