[PATCH 1/3] qsd: Add pre-init argument parsing pass

2021-12-22 Thread Hanna Reitz
We want to add a --daemonize argument to QSD's command line. This will require forking the process before we do any complex initialization steps, like setting up the block layer or QMP. Therefore, we must scan the command line for it long before our current process_options() call. Instead of add

[PATCH 2/3] qsd: Add --daemonize

2021-12-22 Thread Hanna Reitz
This option does basically the same as --fork does for qemu-nbd: - We fork off a child process - The child process is daemonized (closing its stdin and stdout) - stderr of the child is routed through the parent, so the parent can see errors and adjust its exit code accordingly - Once the child cl

[PATCH 0/3] qsd: Add --daemonize; and add job quit tests

2021-12-22 Thread Hanna Reitz
Hi, This series began as an attempt to write a reproducer for the following case: You have a mirror job in READY state with a target that’s slow. There is still a lot of data to be written (because of active I/O in the guest). You quit qemu, and you expect quitting to be reasonably quic

[PATCH 3/3] iotests/185: Add post-READY quit tests

2021-12-22 Thread Hanna Reitz
185 tests quitting qemu while a block job is active. It does not specifically test quitting qemu while a mirror or active commit job is in its READY phase. Add two test cases for this, where we respectively mirror or commit to an external QSD instance, which provides a throttled block device. qe

[PATCH] tests/qtest/hd-geo-test: Check for the lsi53c895a controller before using it

2021-12-22 Thread Thomas Huth
The lsi53c895a SCSI controller might have been disabled in the target binary, so let's check for its availability first before using it. Signed-off-by: Thomas Huth --- tests/qtest/hd-geo-test.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/qtest/hd-geo-test.c

[PATCH RFC v2 0/4] blockdev-replace

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Hi all! v2: 1. Let's start with minimal functionality: replace only one specific child. 2. Explicit type of parent: qdev / export / driver Hmm, 'driver' is for BlockDriverState.. Any better idea? I don't want to call it "node" or even "block-node", as finally, devices and exports are kind of no

[PATCH v2 1/4] block-backend: blk_root(): drop const specifier on return type

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
We'll need get non-const child pointer for graph modifications in further commits. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 2 +- block/block-backend.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/sysemu/block-bac

[PATCH v2 2/4] block/export: add blk_by_export_id()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 1 + block/export/export.c | 18 ++ 2 files changed, 19 insertions(+) diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h index 904d70f49c..250c7465a5 100644 --- a/

[PATCH v2 4/4] qapi: add blockdev-replace command

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add a command that can replace bs in following BdrvChild structures: - qdev blk root child - block-export blk root child - any child BlockDriverState selected by child-name Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block.json | 61 +

[PATCH v2 3/4] block: make bdrv_find_child() function public

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
To be reused soon. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block_int.h | 1 + block.c | 13 + blockdev.c| 14 -- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/block/block_int.h b/include/b

[PULL 0/3] Block patches

2021-12-22 Thread Hanna Reitz
The following changes since commit 8c5f94cd4182753959c8be8de415120dc879d8f0: Merge tag 'pull-loong-20211221-2' of https://gitlab.com/rth7680/qemu into staging (2021-12-21 13:30:35 -0800) are available in the Git repository at: https://gitlab.com/hreitz/qemu.git tags/pull-block-2021-12-22 f

[PULL 1/3] iotests/testrunner.py: add doc string for run_test()

2021-12-22 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to modify these methods and will add more documentation in further commit. As a preparation add basic documentation. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <2021120313.2780098-2-vsement...@virtuozzo.com> Reviewed-by: John Snow

[PULL 2/3] iotests/testrunner.py: move updating last_elapsed to run_tests

2021-12-22 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to use do_run_test() in multiprocessing environment, where we'll not be able to change original runner object. Happily, the only thing we change is that last_elapsed and it's simple to do it in run_tests() instead. All other accesses to self in do_

[PULL 3/3] iotests: check: multiprocessing support

2021-12-22 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Add -j parameter, to run tests in several jobs simultaneously. For realization - simply utilize multiprocessing.Pool class. Notes: 1. Of course, tests can't run simultaneously in same TEST_DIR. So, use subdirectories TEST_DIR/testname/ and SOCK_DIR/testnam

[PATCH v3 00/19] Make image fleecing more usable

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
v3: Rebase on master, small qapi docs improvements suggested by Markus There several improvements to fleecing scheme: 1. support bitmap in copy-before-write filter 2. introduce fleecing block driver, which opens the door for a lot of image fleecing improvements. See "block: introduce fleec

[PATCH v3 07/19] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 11 +++ block/dirty-bitmap.c | 6 ++ util/hbitmap.c |

[PATCH v3 03/19] block/block-copy: block_copy_state_new(): add bitmap parameter

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
This will be used in the following commit to bring "incremental" mode to copy-before-write filter. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 2 +- block/block-copy.c | 14 -- block/copy-before-write.c | 2 +- 3 files changed, 14 insertion

[PATCH v3 02/19] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
That simplifies handling failure in existing code and in further new usage of bdrv_merge_dirty_bitmap(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h| 2 +- block/dirty-bitmap.c| 9 +++-- block/monitor/bitmap-qmp-cmds.c | 5 + 3 files chang

[PATCH v3 10/19] block: introduce fleecing block driver

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Introduce a new driver, that works in pair with copy-before-write to improve fleecing. Without fleecing driver, old fleecing scheme looks as follows: [guest] | |root v [copy-before-write] -> [temp.qcow2] <--- [nbd export] | target | |file |backin

[PATCH v3 05/19] block/block-copy: add block_copy_reset()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Split block_copy_reset() out of block_copy_reset_unallocated() to be used separately later. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 1 + block/block-copy.c | 21 + 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/in

[PATCH v3 15/19] iotests.py: add qemu_io_pipe_and_status()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 4 1 file changed, 4 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 83bfedb902.

[PATCH v3 17/19] block: blk_root(): return non-const pointer

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
In the following patch we'll want to pass blk children to block-copy. Const pointers are not enough. So, return non const pointer from blk_root(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 2 +- block/block-backend.c | 2 +- 2 files changed, 2 inse

[PATCH v3 01/19] block/block-copy: move copy_bitmap initialization to block_copy_state_new()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
We are going to complicate bitmap initialization in the further commit. And in future, backup job will be able to work without filter (when source is immutable), so we'll need same bitmap initialization in copy-before-write filter and in backup job. So, it's reasonable to do it in block-copy. Note

[PATCH v3 08/19] block/reqlist: add reqlist_wait_all()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add function to wait for all intersecting requests. To be used in the further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/reqlist.h | 8 block/reqlist.c | 8 2 files changed, 16 insertions(+) diff --git a/include/block/reqlist.h b/include/bloc

[PATCH v3 11/19] block/copy-before-write: support fleecing block driver

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
The last step to make new fleecing scheme work (see block/fleecing.h for descritption) is to update copy-before-write filter: If we detect that unfiltered target child is fleecing block driver, we do: - initialize shared FleecingState - activate fleecing block driver with it - do guest write sy

[PATCH v3 06/19] block: intoduce reqlist

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Split intersecting-requests functionality out of block-copy to be reused in copy-before-write filter. Note: while being here, fix tiny typo in MAINTAINERS. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/reqlist.h | 67 +++ block/block-copy.c | 116 ++

[PATCH v3 09/19] block: introduce FleecingState class

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
FleecingState represents state shared between copy-before-write filter and upcoming fleecing block driver. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/fleecing.h | 135 ++ block/fleecing.c | 182 ++ MAINTAINE

[PATCH v3 04/19] block/copy-before-write: add bitmap open parameter

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 10 +- block/copy-before-write.c | 30 +- 2 files chang

[PATCH v3 19/19] iotests/image-fleecing: test push backup with fleecing

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add test for push backup with fleecing: - start fleecing with copy-before-write filter - start a backup job from temporary fleecing node to actual backup target Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 121 ++-- tests/qem

[PATCH v3 13/19] block/copy-before-write: use write-unchanged in fleecing mode

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
As announced in previous commit, we need use write-unchanged operations for fleecing, so that fleecing client may unshare writes if needed. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a

[PATCH v3 16/19] iotests/image-fleecing: add test case with bitmap

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 32 ++-- tests/qemu-iotests/tests/image-fleecing.out | 84 + 2 files changed, 108 insertions(+), 8 del

[PATCH v3 12/19] block/block-copy: add write-unchanged mode

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
We are going to implement push backup with fleecing scheme. This means that backup job will be a fleecing user and therefore will not need separate copy-before-write filter. Instead it will consider source as constant unchanged drive. Of course backup will want to unshare writes on source for this

[PATCH v3 18/19] qapi: backup: add immutable-source parameter

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
We are on the way to implement internal-backup with fleecing scheme, which includes backup job copying from fleecing block driver node (which is target of copy-before-write filter) to final target of backup. This job doesn't need own filter, as fleecing block driver node is a kind of snapshot, it's

[PATCH v3 14/19] iotests/image-fleecing: add test-case for fleecing format node

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 67 -- tests/qemu-iotests/tests/image-fleecing.out | 76 - 2 files changed, 122 insertions(+), 21 deletions(-) diff --git a/tests/qemu-iotests/tests/image-fleecing b

Re: [PATCH 2/2] qapi/block: Restrict vhost-user-blk to CONFIG_VHOST_USER_BLK_SERVER

2021-12-22 Thread Philippe Mathieu-Daudé
On 12/22/21 08:05, Markus Armbruster wrote: > Philippe Mathieu-Daudé writes: > >> Do not list vhost-user-blk in BlockExportType >> when CONFIG_VHOST_USER_BLK_SERVER is disabled. >> >> Fixes: 90fc91d50b7 ("convert vhost-user-blk server to block export API") > > My immediate reaction was "what exa

[PULL 0/7] NBD patches

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
The following changes since commit 2bf40d0841b942e7ba12953d515e62a436f0af84: Merge tag 'pull-user-20211220' of https://gitlab.com/rth7680/qemu into staging (2021-12-20 13:20:07 -0800) are available in the Git repository at: https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-nbd-2021-

[PULL 2/7] nbd/client-connection: nbd_co_establish_connection(): return real error

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
The only caller of nbd_do_establish_connection() that uses errp is nbd_open(). The only way to cancel this call is through open_timer timeout. And for this case, user will be more interested in description of last failed connect rather than in "Connection attempt cancelled by other operation". So,

[PULL 3/7] nbd/client-connection: improve error message of cancelled attempt

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- nbd/client-connection.c | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/nbd/client-connection.c b/nbd/client-connection.c index 722998c985..2bda42641d 100644 --- a/nbd/client-connection.c +++

[PULL 1/7] nbd: allow reconnect on open, with corresponding new options

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
It is useful when start of vm and start of nbd server are not simple to sync. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- qapi/block-core.json | 9 - block/nbd.c | 45 +++- 2 files changed, 52 insertions(+), 2

[PULL 4/7] iotests.py: add qemu_tool_popen()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Split qemu_tool_popen() from qemu_tool_pipe_and_status() to be used separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin --- tests/qemu-iotests/iotests.py | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/iotests

[PULL 5/7] iotests.py: add and use qemu_io_wrap_args()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
For qemu_io* functions support --image-opts argument, which conflicts with -f argument from qemu_io_args. For QemuIoInteractive use new wrapper as well, which allows relying on default format. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin --- tests/qemu-iotests/iotest

[PULL 7/7] iotests: add nbd-reconnect-on-open test

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin --- .../qemu-iotests/tests/nbd-reconnect-on-open | 71 +++ .../tests/nbd-reconnect-on-open.out | 11 +++ 2 files changed, 82 insertions(+) create mode 100755 tests/qemu-iotests/tests/nbd-reconnect

[PULL 6/7] iotests.py: add qemu_io_popen()

2021-12-22 Thread Vladimir Sementsov-Ogievskiy
Add qemu-io Popen constructor wrapper. To be used in the following new test commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin --- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests

Re: [PULL 0/3] Block patches

2021-12-22 Thread Richard Henderson
On 12/22/21 8:52 AM, Hanna Reitz wrote: The following changes since commit 8c5f94cd4182753959c8be8de415120dc879d8f0: Merge tag 'pull-loong-20211221-2' of https://gitlab.com/rth7680/qemu into staging (2021-12-21 13:30:35 -0800) are available in the Git repository at: https://gitlab.com/h

Re: [PULL 0/7] NBD patches

2021-12-22 Thread Richard Henderson
On 12/22/21 10:52 AM, Vladimir Sementsov-Ogievskiy wrote: The following changes since commit 2bf40d0841b942e7ba12953d515e62a436f0af84: Merge tag 'pull-user-20211220' of https://gitlab.com/rth7680/qemu into staging (2021-12-20 13:20:07 -0800) are available in the Git repository at: https